“Modest” but “statistically significant”…what does that mean?

by Win Noren

In my last blog, I mentioned that the Physicians Health Study II found a modest but statistically significant reduction in cancer for those physicians who were taking a daily multivitamin. The fact that a difference between two groups could be small and yet be statistically significant can seem contradictory. The fact that a difference could be large and yet not be statistically significant is even more mind-bending if you don’t think through the concepts carefully.

The conclusion regarding the impact on cancer by a daily multivitamin was published in the Journal of the American Medical Association. Specifically it was concluded :

Compared with placebo, men taking a daily multivitamin had a statistically significant reduction in the incidence of total cancer (multivitamin and placebo groups, 17.0 and 18.3 events, respectively, per 1000 person-years; hazard ratio [HR], 0.92; 95% CI, 0.86-0.998; P = .04).

What was found is for every 1,000 male physicians who took a daily multivitamin, there were 17.0 instances of cancer. For every 1000 male physicians who took a placebo there were 18.3 instances of cancer. The question then becomes whether there is enough evidence to say that this was a “real” difference or whether this was just random.

This can only be answered if we know how large the study was as the answer has to do with the sample size. When comparing means, variability would also be considered, but for rates, variability is a function of sample size and rate. Our Electronic Statistics Textbook has a great discussion of sample size and other elementary concepts, and I would encourage you to look at the very informative illustrations provided there to get a more comprehensive explanation.

But in short, because this Physicians Health Study II is so large (with more than 14,000 male physicians taking part), even small differences can be declared “statistically significant.” If the study had been smaller, with fewer participants, then a much bigger difference between the number of cancer incidents would have been necessary for it to be clear that the difference was “real” and not just random.

Gaziano J.M., H.D. Sesso, W.G. Christen, et al. “Multivitamins in the Prevention of Cancer in Men: The Physicians’ Health Study II Randomized Controlled Trial.” JAMA 308, no. 18 (2012): 1871-1880. doi:10.1001/jama.2012.14641.