How Good is Public Health Research?

Yesterday morning I had one of those experiences journalists dread: I opened up the Sunday New York Times and there on the front page of the Times Magazine was a story I've been kicking around. Titled "Do We Really Know What Makes Us Healthy?", the story by Gary Taubes is a thorough look at how so much scientific research in the name of public health gives us dodgy, even incorrect, results. Told through the vehicle of hormone replacement therapy, Taubes' story is really a discerning look at the purpose and limits of epidemiology - which is our best way of establishing what sort of behaviors and interventions might be good for us, and which might be bad. One thing I was expecting Taubes to mention was the killer study by John Ioannidis in the August 2005 issue of PLoS Medicine: "Why Most Published Research Findings Are False". This erudite bit of statistical analysis pointed out that one-off results are often wrong - in fact, are more likely to be false than true. One of the most downloaded papers PLoS has put out, it's the sort of clear, counter-intuitive and declarative paper that I wish was characteristic of more scientific research. I expect Taubes didn't get into it because it's full of talk of positive-predictive-vaules and power and bias. But still, if I had written the story - which I swear I had a version of on my to-do list - Ioannidis would've been on the source list.