“Modest” but “statistically significant”…what does that mean?

by Win Noren

In my last blog, I mentioned that the Physicians Health Study II found a modest but statistically significant reduction in cancer for those physicians who were taking a daily multivitamin. The fact that a difference between two groups could be small and yet be statistically significant can seem contradictory. The fact that a difference could be large and yet not be statistically significant is even more mind-bending if you don’t think through the concepts carefully.

The conclusion regarding the impact on cancer by a daily multivitamin was published in the Journal of the American Medical Association. Specifically it was concluded :

Compared with placebo, men taking a daily multivitamin had a statistically significant reduction in the incidence of total cancer (multivitamin and placebo groups, 17.0 and 18.3 events, respectively, per 1000 person-years; hazard ratio [HR], 0.92; 95% CI, 0.86-0.998; P = .04).

What was found is for every 1,000 male physicians who took a daily multivitamin, there were 17.0 instances of cancer. For every 1000 male physicians who took a placebo there were 18.3 instances of cancer. The question then becomes whether there is enough evidence to say that this was a “real” difference or whether this was just random.

This can only be answered if we know how large the study was as the answer has to do with the sample size. When comparing means, variability would also be considered, but for rates, variability is a function of sample size and rate. Our Electronic Statistics Textbook has a great discussion of sample size and other elementary concepts, and I would encourage you to look at the very informative illustrations provided there to get a more comprehensive explanation.

But in short, because this Physicians Health Study II is so large (with more than 14,000 male physicians taking part), even small differences can be declared “statistically significant.” If the study had been smaller, with fewer participants, then a much bigger difference between the number of cancer incidents would have been necessary for it to be clear that the difference was “real” and not just random.

 

Gaziano J.M., H.D. Sesso, W.G. Christen, et al. “Multivitamins in the Prevention of Cancer in Men: The Physicians’ Health Study II Randomized Controlled Trial.” JAMA 308, no. 18 (2012): 1871-1880. doi:10.1001/jama.2012.14641.

Advertisements

About statsoftsa

StatSoft, Inc. was founded in 1984 and is now one of the largest global providers of analytic software worldwide. StatSoft is also the largest manufacturer of enterprise-wide quality control and improvement software systems in the world, and the only company capable of supporting its QC products worldwide, with wholly owned subsidiaries in all major markets (StatSoft has 23 full-service offices, on all continents), and its software is available in more than 10 languages.

Posted on May 2, 2013, in Uncategorized. Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: