P Values vs Confidence Intervals: EDAIC Statistics

A P value describes how incompatible the observed result is with a specified statistical model, often one containing a null hypothesis of no effect. It does not give the probability that the null hypothesis is true. A confidence interval presents an estimated range using a procedure with stated long-run coverage, under the relevant assumptions.

For EDAIC, keep the effect estimate, its uncertainty and a hypothesis test separate. A small P value does not measure the clinical importance of an effect. The American Statistical Association's statement explicitly cautions against treating a threshold as a substitute for scientific reasoning.

Begin with the question the study asked

Before interpreting a number, identify the outcome, comparison and scale. A difference in mean recovery time is measured in minutes. A risk ratio is dimensionless. Their no-effect values differ: zero for a difference, one for a ratio.

If you skip that step, “the confidence interval includes zero” can become an automatic rule applied to the wrong measure. Write the null value next to the estimate before deciding whether an interval crosses it.

Also ask whether the interval and P value come from the same model and test. Familiar correspondences between them require matching methods; they are not a licence to combine unrelated analyses.

Original worked example: an estimated mean difference

Imagine a hypothetical large-sample study estimating that intervention A increases a measured recovery outcome by 4 minutes relative to B. Assume an approximately normal sampling distribution and an estimated standard error of 1.5 minutes. No recommendation about either intervention is intended.

Using a normal-approximation 95% interval:

Estimate ± 1.96 × standard error.

The margin is 1.96 × 1.5 = 2.94 minutes, giving an interval from 1.06 to 6.94 minutes. Rounded appropriately, that is 1.1 to 6.9 minutes.

Zero is outside this interval. Under the corresponding two-sided normal test of a zero difference, that is consistent with a P value below 0.05. The result still leaves a range of compatible effect sizes, not a uniquely established four-minute change.

NIST's confidence-limit guidance explains estimation using a standard error and an appropriate reference distribution. A small-sample analysis with an estimated variance may require a t distribution rather than simply inserting 1.96.

Change precision without changing the estimate

Keep the four-minute estimate but suppose the standard error is now 3 minutes. The corresponding approximate interval is 4 ± 5.88, or −1.88 to 9.88 minutes.

This interval includes zero, but also includes potentially substantial positive differences. The correct interpretation is not that the interventions have been proven identical. The estimate is less precise, and the data are compatible with a broader range under the model.

Now suppose an otherwise comparable estimate is 0.2 minutes with a standard error of 0.05 minutes. Its approximate interval is 0.102 to 0.298 minutes. This interval excludes zero, yet the effect may be too small to matter for a particular decision.

The examples separate precision from magnitude: changing the standard error can change the inferential result without changing the point estimate, while a very precise small effect can cross a statistical threshold without becoming a large effect.

What “95% confidence” does and does not mean

In the usual frequentist interpretation, the procedure would produce intervals containing the fixed true parameter in 95% of repeated samples under its assumptions. It does not mean that 95% of individual observations lie inside this confidence interval for a mean.

Nor is it the same as an individual prediction interval. Estimating a population mean and predicting a future person's value involve different uncertainty. Greenland and colleagues' guide to misinterpretations explains why informal probability statements about confidence intervals need care.

For a practical reading habit, ask: “An interval for what?” A mean, a mean difference, a proportion and a prediction are not interchangeable objects even if each is displayed using two numerical limits.

Non-significance is not equivalence

A test that fails to reject a zero difference does not establish clinically acceptable similarity. An equivalence question requires an appropriate design, analysis and justified margins. Altman and Bland's discussion addresses the error of treating insufficient evidence of a difference as evidence that none exists.

In the wide-interval example above, imagine the study team had considered changes larger than three minutes important before collecting data. The interval extends far beyond that threshold. Reporting only “not significant” would hide information directly relevant to the question.

Do not select that threshold after seeing the result simply to make the conclusion convenient. The teaching point is to make the decision standard explicit and assess the full interval against it.

EDAIC self-test

1. A P value of 0.03 means a 3% probability that the null hypothesis is true.
2. A confidence interval for a mean usually contains 95% of individual observations.
3. A precise, very small effect can be statistically significant.
4. A non-significant superiority test proves equivalence.

Answers: false, false, true, false. The errors respectively invert a conditional probability, confuse an estimate with a distribution of individuals, and turn absence of a detected difference into a different claim. The true statement separates precision from effect size.

Use the EDAIC statistics guide for the broader map. In AnesCORE practice, write the estimate, units, interval and null value as four separate entries before judging a statistical statement.

Sources

• American Statistical Association: statement on P values
• NIST: Confidence limits for the mean
• Greenland et al: Statistical tests and common misinterpretations
• Altman and Bland: Absence of evidence is not evidence of absence

Prepared with AI assistance for examination revision; this article is not a patient-care protocol or a substitute for current primary guidance. Sources checked 7 October 2026.