Choosing between Cures: What do vaccine efficacy numbers mean?

Last week, Johnson & Johnson began delivering millions of doses of its coronavirus vaccine across the United States after receiving an emergency use authorisation from the Food and Drug Administration. Central to getting the green light was a trial that Johnson & Johnson ran to measure the vaccine’s efficacy. Efficacy is a crucial concept in vaccine trials, but it’s also a tricky one. If a vaccine has an efficacy of, say, 95 percent, that doesn’t mean that 5 percent of people who receive that vaccine will get COVID-19. And just because one vaccine ends up with a higher efficacy estimate than another in trials doesn’t necessarily mean it’s superior. Here’s why. 

For statisticians, efficacy is a measurement of how much a vaccine lowers the risk of an outcome. For example, Johnson & Johnson observed how many people who received a vaccine nevertheless got COVID-19. Then they compared that to how many people contracted COVID-19 after receiving a placebo. The difference in risk can be calculated as a percentage. Zero percent means that vaccinated people are at as much risk as people who got the placebo. A hundred percent means that the risk was entirely eliminated by the vaccine. In the United States trial site, Johnson & Johnson determined that the efficacy is 72 percent. 

Efficacy depends on the details of a trial, such as where it took place. Johnson & Johnson ran trials at three sites: in the United States, Latin America and South Africa. The overall efficacy was lower than that in the United States alone. One reason for that appears to be that the South Africa trial took place after a new variant had swept across that country. Called B.1.351, the variant has mutations that enable it to evade some of the antibodies produced by vaccination. The variant didn’t make the vaccine useless, however. Far from it: In South Africa, Johnson & Johnson’s efficacy was 64 percent. 

Efficacy can also change when scientists look at different outcomes. Johnson & Johnson’s vaccine had an 85 percent efficacy rate against severe cases of COVID-19, for example. That’s important to know, because it means that the vaccine will prevent a lot of hospitalisations and deaths. 

When scientists say that a vaccine has an efficacy of, say, 72 percent, that’s what’s known as a point estimate. It’s not a precise prediction for the general public, because trials can only look at a limited number of people — in the case of Johnson & Johnson’s trial, about 45,000 volunteers. 

The uncertainty around a point estimate can be small or large. Scientists represent this uncertainty by calculating a range of possibilities, which they call a confidence interval. One way of thinking of a confidence interval is that we can be 95 percent confident that the efficacy falls somewhere inside it. If scientists came up with confidence intervals for 100 different samples using this method, the efficacy would fall inside the confidence intervals in 95 of them. 

Confidence intervals are tight for trials in which a lot of people get sick and there’s a sharp difference between the outcomes in the vaccinated and placebo groups. If few people get sick and the differences are minor, then the confidence intervals can explode. 

Last year, the F.D.A. set a goal for coronavirus vaccine trials. Each manufacturer would need to demonstrate that a vaccine had an efficacy of at least 50 percent. The confidence interval would have to reach down no lower than 30 percent. A vaccine that met that standard would offer the kind of protection found in flu vaccines — and would therefore save many lives. 

So far, three vaccines — made by Pfizer and BioNTech, Moderna and Johnson & Johnson — have all been authorised in the United States after their trials demonstrated they surpassed the F.D.A.’s threshold. AstraZeneca and Novavax, which have ongoing U.S. trials, have published efficacy results from studies in other countries. Meanwhile, the makers of the Sputnik V vaccine have published results based on their trial in Russia. 

For a number of reasons, it’s not possible to make a precise comparison between these vaccines. One vaccine may have a higher point estimate than another, but their confidence intervals may overlap. That effectively makes their results indistinguishable. 

Making matters more complicated, the vaccines were tested on different groups of people at different stages in the pandemic. In addition, their efficacy was measured in different ways. Johnson & Johnson’s efficacy was measured 28 days after a single dose, for example, while Moderna’s was measured 14 days after a second dose. 

What’s clear is that all three vaccines authorised in the United States — made by Johnson & Johnson, Moderna, and Pfizer and BioNTech — greatly reduce the risk of getting COVID-19. 

What’s more, all the vaccines look as if they have a high efficacy against more serious outcomes like hospitalisation and death. For example, no one who got Johnson & Johnson’s vaccine had to go to the hospital for a COVID-19 infection 28 days or more after getting an injection. Sixteen people who got the placebo did. That translates to 100 percent efficacy, with a confidence interval of 74.3 percent to 100 percent. 

A clinical trial is just the start of the research on any vaccine. Once it goes into widespread use, researchers follow its performance. Instead of efficacy, these scientists now measure effectiveness: how much the vaccine reduces the risk of a disease out in the real world, in millions of people rather than thousands. Early studies on the effectiveness of coronavirus vaccines are confirming that they provide strong protection. 

In the months to come, researchers will keep an eye on this data to see if they become less effective — either because the immunity from the vaccine wanes or because a new variant arises. In either case, new vaccines will be created, and manufacturers will provide new measures of their efficacy.