The more you hear the less you know

July 31, 2012

Free Market Foundation

(Highly publicised new therapies often later found to be ineffective)

There are many reasons why health scares and medical treatments come and go. Now we have a scientific study of scientific studies that sheds some light on the subject. An article in the July 13 Journal of the American Medical Association (JAMA) examines how often research that gets a lot of attention in the scientific community and in the wider world is then challenged by later studies that reach opposite or less dramatic conclusions.

According to the researchers:

Of 49 highly cited original clinical research studies, 45 revealed effective medical interventions.

Of these 45, one-third (about 14) were eventually contradicted either totally or in part by other studies, 20 were replicated and 11 remained largely unchallenged.

Here are a few examples:

Hormone therapy, initially said to reduce coronary artery disease events in women, is now believed to increase the risk of such events.

Vitamin E therapy, which was said to be a heart protector for men and women, was found to be ineffective.

The initial positive results of a drug said to slow HIV disease progression, and treatments to prevent strokes, were later found to be much more modest, or to have only short-term effects.

One explanation for the discrepancy is simple: the shorter the study and the smaller the group studied, the more likely it may be that a subsequent, deeper, investigation will contradict or alter the original thesis, says JAMA. Additionally, research that points to possible therapies tends to get more attention than those that show “negative findings.”

Furthermore, the reality of refuted research is not all that shocking. What is, says JAMA, is the speed at which a single study can become fact or acquire power it doesn’t deserve.

Source: Editorial, The More You Hear, the Less You Know, Wall Street Journal, July, 15, 2005; based upon: John P.A. Ioannidis, Contradicted and Initially Stronger Effects in Highly Cited Clinical Research, Journal of the American Medical Association, Vol. 294, No. 2, July 13, 2005.

For text (subscription required): http://online.wsj.com/article/0,,SB112139336919186544,00.html

For JAMA study (subscription required): http://jama.ama-assn.org/cgi/content/full/294/2/218

For more on Therapies and New Technologies: http://www.ncpa.org/iss/hea/

FMF Policy Bulletin/ 26 July 2005