Study Concludes: Most Studies Are Wrong.

The Scientific Method

The Scientific Method (Photo credit: afagen)

As the number of scientific studies exponentially mount, surely we are advancing scientific inquiry at an ever increasing rate.  But perhaps we are simply increasing the scientific “noise.”

Enter the Reproducibility Initiative, which will try to reproduce your findings for you by an independent lab.  Yes, we’ve come to a point where you will need to pay to have the results you think you have confirmed by someone else.

Consider the NewsDaily’s article that “Bayer Healthcare reported that its scientists could not reproduce some 75 percent of published findings in cardiovascular disease, cancer and women’s health.” Or that “Amgen reported that when the company’s scientists tried to replicate 53 prominent studies in basic cancer biology, hoping to build on them for drug discovery, they were able to confirm the results of only six.”

How is that possible?  Don’t we have scientists dedicated to publishing whatever results occur?

Anyone who remembers science class knows the answer.  When you got the results you expected, you didn’t go over the equipment and the method with a fine toothed comb.  You assumed you did the experiment right and turned it in.  Only when you got wildly odd results that didn’t agree with what you were looking for in the slightest did you go back over your method and equipment to find the error.  Even if you had the highest ethics, it would be perfectly possible to miss some error as long as the results fell into a “reasonable” outcome.

Having more people do the same testing can lead to better results, but if you are all testing in the same area and watching one another’s results, chances are good that you started looking for the same results in your experiments.  A friendly classmate might even help you by telling you how to change your equipment to get a desired result.  So more tests do not necessarily lead to more accurate results.

But don’t believe me.  Have a gander at the most read PLoS article ever, entitled: Why Most Published Research Findings Are False.

In this appetizing little mathematical jaunt, the author takes us down the reality that false positives are far more likely than finding the truth.   Even before we add in publication bias, tenure track pressures, and financial incentives, it is just too easy to find the results you’re looking for.  The author Ioannidis states:  “manipulation could be done, for example, with serendipitous inclusion or exclusion of certain patients or controls, post hoc subgroup analyses, investigation of genetic contrasts that were not originally specified, changes in the disease or control definitions, and various combinations of selective or distorted reporting of the results. Commercially available “data mining” packages actually are proud of their ability to yield statistically significant results through data dredging.”

So just how many of us are taking drugs created for an illness, supported by studies created to support that drug’s ability to treat that illness, and prescribed by doctors who believe that the drug will effectively treat our illness despite all of our claims that the drug really isn’t working?  Meanwhile we as patients want to have something that works for our illness, so we spend a lot of time giving the drug “time to work” when it really never does anything to help us.

As someone who works in the alternative healthcare field, I don’t know whether to laugh or cry.  It feels like the rug just got pulled out from under all the work we’ve done to start bringing the field up to the standard of scientific inquiry.  Suddenly what was clinically relevant information is in question, and the standard drugs that we’re trying to compare to the alternatives are also in question.  How do we know what works?

Fortunately, I’ve got an ace up my sleeve.  I’ve been working with ornery, independent minded patients for years who don’t mince words when things don’t work.  So I’ve got an ongoing practice based on what is working in the field, using my patients as my resources.  Maybe it is time for all doctors to use their patients, rather than the drug reps, as their resource for what really works.

Advertisements

, , , , , , ,

  1. #1 by rebecca2000 on August 17, 2012 - 11:30 pm

    I am all about natural care. I was on all sorts of medications before I started alternative healthcare. I swear I take nothing but iron now and that is because I quit seeing my chiropractor when I moved. I need a new one that does more than back crack.

    Ever read the book Henrietta Lacks? It is very informative.

    • #2 by Christopher Maloney, Naturopathic Doctor on August 18, 2012 - 10:00 am

      I haven’t, but I added it to my list. For any readers not familiar with Henrietta Lacks, here’s the wikipedia version of her extraordinary story. http://en.wikipedia.org/wiki/Henrietta_Lacks

      • #3 by rebecca2000 on August 18, 2012 - 12:03 pm

        I am glad you checked it out. They talk about all sorts of horrible things they did to people back then. Doctors, in the name of science. They would ask people with children to move into houses with lead paint. They would inject Ms. Lacks cancer into people to see what happened and so on. Scary.

        I am following you. I would love if you checked out my blog. It isn’t a health one, but humor. I really enjoy making people laugh.

  2. #4 by Graham Coghill on August 18, 2012 - 2:33 am

    When you say ” So I’ve got an ongoing practice based on what is working in the field, using my patients as my resources. Maybe it is time for all doctors to use their patients, rather than the drug reps, as their resource for what really works”, does this mean you are using anecdotal evidence to back up your ideas? My feeling is that if randomized controlled trials are as prone to error as you describe in your post, then the success rate of anecdotal evidence would have to be practically negligible.

    • #5 by Christopher Maloney, Naturopathic Doctor on August 18, 2012 - 9:55 am

      No, actually, it’s about using the evidence we do have and helping each patient find what works for them. Individualizing the dosages, the protocols, and the results. The success rate for that approach is phenomenal, because it takes into account long-term compliance and tailors the treatment (whether drugs, physical therapy, surgery, or alternatives like nutrition and lifestyle) to the patient’s needs.

      Take a simple example. After two years a patient had not completely healed from a fracture. In a quick examination of the brace used, it was clear that she was partially tourniqueting her leg and preventing blood flow to the area. After questioning it was clear that the doctor never saw her with the brace on and her physical therapist had voiced concerns but had never made an official request for a larger brace. Only because I took the time to take a complete history, examined the leg fully, and then took the time to watch and see how she attached the brace was I able to see the issue.

      The problem with this approach is that it is incredibly time-intensive. Within our insurance model of payment, doctors cannot justify spending the kind of time they need to go through, explain, and work with a patient to tweak treatment so that it is most effective. As a result the vast majority of patients are on medications and treatment protocols that have never been truly individualized to their needs. When they complain about a side-effect or want more results the answer tends to be to change medications or to add an addition medication to the mix. That is a direct path to polypharmacy, which muddies the picture of the patient’s pathology and leads to “just living with it” rather than finding what will resolve the issues.

      So yes, the doctors do need to start listening to the patients, but that involves changing the framework in which the doctors work so that they can work directly for the patients.

      • #6 by Graham Coghill on August 18, 2012 - 6:14 pm

        But in your post, you say “When you got the results you expected, you didn’t go over the equipment and the method with a fine toothed comb. You assumed you did the experiment right and turned it in. Only when you got wildly odd results that didn’t agree with what you were looking for in the slightest did you go back over your method and equipment to find the error. Even if you had the highest ethics, it would be perfectly possible to miss some error as long as the results fell into a “reasonable” outcome.”

        How can you be sure that you don’t fall prey to this effect every day. Isn’t this why science uses randomised controlled trails (double-blind if possible) to gather evidence?

      • #7 by Christopher Maloney, Naturopathic Doctor on August 21, 2012 - 10:49 pm

        The point of the post was that even large, multinational RCT trials are giving us false information, so your faith in the RCT process may be misdirected.

        Your concern is that I’m engaging in what is known as the practice of naturopathic medicine, which involves working on solving a patient’s concerns.

        Unlike a laboratory experiment, these are suffering human beings who have not signed up for participation in an RCT. My ethical obligation is to use all the information available to help them get well. With the proliferation of false positive RCTs, it is ever more necessary to focus on the doctor/patient interaction.

        The issue I think you hope to raise is how do we know what we do works? Simply because the chronically ill patients have improved to a point where they no longer feel ill.

        It is possible that after several years of illness my patients spontaneously improve under my care. What a wonderful thought, but unfortunately not reliable. Instead, it is the changes they make in their lives through hard work and dedication that lead to slow improvement.

        My role in that process is to act as guide and coach, to provide information and encouragement. Again, could the patient do it without me? They have not, so I seem to be a necessary part of the equation. I suppose a basketball team could win a championship without a coach, it just isn’t very likely.

        I would recommend watching any of the dozens of videos by naturopathic doctors to get a sense of the sorts of treatments we know work. I posted a nice discussion of the scientific method applied to icing boo-boos which I think illustrates just how complicated modern RCT science can make our lives. https://humanbodyengineer.wordpress.com/2012/08/18/should-you-ice-your-boo-boo-more-studies-necessary-says-meta-analysis/

  3. #8 by Graham Coghill on August 21, 2012 - 11:26 pm

    “The issue I think you hope to raise is how do we know what we do works? Simply because the chronically ill patients have improved to a point where they no longer feel ill.”

    But that’s my point exactly! I don’t dispute the fact that a proportion of RCTs are wrong. What I’m saying is that if RCTs, purposely designed to eliminate errors, with double blinding etc can be wrong in claiming that a certain treatment produces a certain outcome, then how can a single person, operating with no blinding at all and no statistical analysis, claim a better success rate in identifying that a certain outcome resulted from a specific treatment? I’m sure that in these circumstances, the error rate would be even higher.

    • #9 by Christopher Maloney, Naturopathic Doctor on August 22, 2012 - 11:43 pm

      As I point out in the last comment, it could be that my chronically ill patients spontaneously get well simply by entering my presence. It could also be the work of my lawn gnome statue, who’s positive influence may be so powerful he creates a healing field around my entire office. But neither is terribly likely.

      It could even be your psychic influence as you pray for my patients, which I thank you for and I will continue to treat as if you were not praying to maximize your benevolent influence and to cover all my bases.

      In real life, unlike an experimental model in a lab, we are dealing with dozens of factors that affect a person’s health simultaneously. I can say conclusively that my patients have not done well for a period of years before they come see me. By incorporating my treatment plans into their lives, they improve and sometimes cease having symptoms. If they stop seeing me, often while continuing conventional care, they often get worse. The conclusion they come to is that I am a worthwhile part of their healthcare plan.

      Are they mistaken and are we somehow engaging in an error-filled experiment? No. I would not take credit for their improvement. I am merely a catalyst. Rather it is the patient who is engaging in ongoing experiments with his or her own body, finding out how to be healthy. I help provide them with the guidance on how to conduct that personal experiment. Like any laboratory researcher, they go through a series of trials, and we incorporate successful trials into further testing.

      So the analogy to a tiny, non-RCT trial is a false one, because I am not in charge of an experiment. Rather I am helping to motivate and guide the researching patient in the process of just going into the laboratory every day. They are the researchers, and we need each one of them to engage in the ongoing process of finding what will bring them to health.

      It is the shift of power, and of focus, that differentiates the model of medicine that we need to move toward.

  1. Should You Ice Your Boo-Boo? More Studies Necessary, Says Meta-Analysis. « Human Body Engineer

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: