What is a p-value in medical research? It is the probability, under a specified statistical model, of a result as extreme as observed. It does not say the null hypothesis is true, and it does not say how large or important the effect is.
A trial report says a new treatment “worked” because p was below 0.05, and a headline follows. The sections below show why that jump is shaky and how to read the p-value, the effect estimate and the confidence interval together. For more explainers like this, see medical news today.
What a p-value measures
A p-value starts from a null hypothesis, the assumption that nothing is going on. In a two-treatment comparison, a biostatistics teaching article gives the example that there is no difference between the two treatments with respect to their effect. The p-value then asks how surprising the observed data would be if that assumption held.
Fisher’s original wording, as quoted in a discussion of the American Statistical Association (ASA) report on p-values, is the probability of the observed result plus more extreme results, if the null hypothesis were true. The same teaching article says this probability reflects the measure of evidence against the null hypothesis, and that small p-values correspond to strong evidence.
The ASA report defines a p-value as a probability under a specified statistical model. That phrase matters: the number is only as meaningful as the model and the study behind it. The ASA’s own statement is not a consensus statement, and the same discussion notes considerable disagreement among expert statisticians on the question. A reader who finds the idea slippery is in good company.
Common misreadings of p less than 0.05
The 0.05 cut-off is a convention. A significance level of 0.05 (5%) is often chosen, according to the teaching article, but the ASA discussion calls treating p=0.05 as the point for deciding statistical significance arbitrary. Under that convention a p-value of 0.055 counts as not statistically significant, even though it is hardly different from 0.05 in the evidence it represents.
The misreadings below are the ones that most often turn a p-value into a verdict.
| Common belief | What the evidence supports | Planning use |
|---|---|---|
| p below 0.05 proves the treatment works | The p-value does not address whether the null hypothesis is true; it is a probability calculated assuming a specified model. | Treat it as one piece of evidence against “no difference”, not a verdict. |
| A smaller p-value means a bigger effect | The p-value does not reflect effect size, and alone gives no direct statement about direction or size of a difference. | Look for the effect estimate and its interval before judging importance. |
| p above 0.05 means no effect | 0.055 is considered not statistically significant under the convention, but the cut-off is arbitrary. | Read the interval to see which effects remain plausible. |
| Significance is a licence for claims | The ASA report criticises widespread use of p<0.05 as a licence, and says study design, measurement accuracy and assumptions also need consideration. | Ask how the study was designed and what was measured. |
The first, third and fourth rows draw on the ASA report discussion, and the second also draws on the teaching article on p-values and confidence intervals.
Why the first misreading is so common
The authors of the ASA discussion surveyed researchers. Most said the p-value was a test of whether the null hypothesis was true. Some teaching texts also phrase p<0.05 loosely as a chance of under 5% that a finding is a fluke. Both shortcuts blur the actual definition, which conditions on the null hypothesis rather than assessing it.
What to do instead
Report and read the p-value as a graded measure of evidence, not a pass mark. The ASA discussion adds that researchers should report all statistical analyses done and all hypotheses tested. A lone p-value from many tests is harder to judge than one from a single prespecified comparison.
What a 95 percent confidence interval says
A confidence interval is a range of values calculated by statistical methods that is expected to contain the true value. The teaching article explains that a 95% level is the usual choice, meaning the interval covers the true value in 95 of 100 studies performed. That describes the method repeated across studies, so a single interval is not a 95% promise about itself.
This is the answer to what a 95 percent confidence interval is, and it helps with reading in practice. Intervals give a range in which the true value lies, along with the direction and strength of the effect. They also reflect results in the units of the data, such as mm Hg for blood pressure, so a reader can judge whether the plausible range matters clinically.
What makes an interval wide or narrow
Width depends on the sample size and the standard deviation of the study groups. A wide interval signals an imprecise estimate, often from a small or highly variable sample. Confidence level also matters: a 99% confidence interval is wider than a 95% interval from the same data. Compare intervals only when they use the same level.
Why intervals add to p-values
P-values alone do not permit any direct statement about the direction or size of a difference. The teaching article notes that it is often useful to report both measures. Two trials can share a p-value yet have intervals that point to very different practical conclusions.
Significant but unimportant results
A result can be statistically significant and still matter little. The ASA discussion gives an example: with a very large sample, a 5 percent decrease in the incidence of acute graft-versus-host disease could be associated with p<0.05. Sample size drives the p-value, while the size of the benefit is a separate matter.
This is why effect size matters. A significant p-value says the data are hard to square with no difference, not that the difference is worth acting on. Judge importance from the effect estimate and its interval, using clinical sense about what size of change would change care.
The reverse also holds. Because the 0.05 line is arbitrary, a result just above it is not evidence of no effect. The ASA discussion recommends that estimated clinically important effects be reported transparently with confidence intervals and p-values, so readers do not have to rely on one threshold.
Reading a results row together
Results tables usually pair an effect estimate, such as a hazard ratio in a survival trial, with its confidence interval and a p-value. Use them in sequence rather than hunting for the one that says “significant”.
- Find the effect estimate. It shows the direction and size of the difference in the measure the study used.
- Check the interval. Note its width, which reflects sample size and variability, and whether the whole range would matter clinically.
- Use the p-value last. Treat it as a measure of evidence against the null hypothesis, not a measure of effect size or a proof.
- Consider the design. Study design, measurement accuracy and assumptions affect how far any of these numbers can be trusted.
Design decides what even a clean result can mean. Read correlation vs causation in medical studies for why a small p-value does not establish a cause, and randomized controlled trial vs observational study for how study type changes the weight of the same numbers.
Frequently asked questions
| Question | Answer |
|---|---|
| What does p less than 0.05 actually mean? | It means the observed result, or one more extreme, would be uncommon if the null hypothesis and the specified model were true. The 0.05 level is a common convention, but the ASA discussion calls it arbitrary, and it does not show the null hypothesis is false. |
| Does a non-significant p-value mean the treatment does not work? | No. A p-value of 0.055 counts as not statistically significant under the convention, but the cut-off is arbitrary. The confidence interval shows which effect sizes remain plausible. |
| Is a 99% confidence interval better than a 95% one? | It is more conservative, not more informative. A 99% interval is wider than a 95% interval from the same data, so compare intervals only at the same confidence level. |
| Can a result be significant but unimportant? | Yes. The p-value does not reflect effect size, and with a very large sample a small effect can give p below 0.05. Judge importance from the effect estimate and its interval. |
| Should studies report p-values or confidence intervals? | Both are useful together. The ASA discussion recommends reporting clinically important effects with confidence intervals and p-values transparently. |
