NIH CLINICAL CENTER GRAND ROUNDS
Episode 2009-001
Time: 1:08:39
CONTEMPORARY CLINICAL MEDICINE: GREAT TEACHERS
Evidence Based Medicine: A Story of Science, Policy and Politics
Presented by Dr. David F. Ransohoff, Professor of Medicine and Clinical Professor of Epidemiology, and director of the Clinical Research Curriculum at the University of North Carolina School of Medicine
ANNOUNCER: Discussing Outstanding Science of the Past, Present and Future – this is NIH Clinical Center Grand Rounds.
(Music establishes, goes under VO)
ANNOUNCER: Greetings and welcome to NIH Clinical Center Grand Rounds. For our premiere podcast, we bring you a lecture from our monthly series on Contemporary Clinical Medicine: Great Teachers. Dr. David F. Ransohoff, Professor of Medicine and Clinical Professor of Epidemiology, and director of the Clinical Research Curriculum at the University of North Carolina School of Medicine will discuss “Evidence-Based Medicine: A Story of Science, Policy and Politics.” If you would like to see a close-captioned videocast of this lecture, we invite you to log on to http://videocast.nih.gov and click the “Past Events” link. We take you to the Lipsett Ampitheater at the NIH Clinical Center in Bethesda, Maryland, where Dr. Barry Kramer, associate director for disease prevention at NIH, will introduce today’s speaker.
(Music fades)
KRAMER: Thank you. So, you're about to see why the choice of David Ransohoff to deliver the great teacher lecture this year was so easy for us on the committee when choosing speakers and also why it's my pleasure to introduce him. The title of the talk as you know is Evidence-based Medicine, Science, Policy and Politics. And I can almost guarantee you even though I haven't heard this particular lecture, I have heard a number of David's lectures and they are always very wonderful and the topic, obviously, is very stimulating. David is professor of medicine at the University of North Carolina, Lineburger Comprehensive Cancer Center, and director of the clinical research curriculum at the medical school. He went to college at Harvard in Cambridge for his undergraduate degree and attended medical school at The Case Western Reserve University School of Medicine. He then did his internship and residency in medicine at the Dartmouth Hitchcock Medical Center and did a fellowship in clinical epidemiology at the Robert Wood Johnson Clinical Scholars Program of Yale University and then in gastroenterology at the University of Chicago. He published over 100 scholarly medical publications and on the editorial board of numerous journals, including the Journal of the National Cancer Institute, which is the journal that I edit, and I’m very pleased to have him on the editorial board as an associate editor. David's primary research interest is in improved methods of colon cancer screening. He’s published extensively on the use of colonoscopy screening and surveillance, blood testing, sigmoidoscopy and virtual colonoscopy. His most recent work is stool DNA testing, serum proteomics and the development and assessment of other new omics methods to screen for cancer. All are obviously at the interface of science, policy and politics. Working with the National Cancer Institute's Early Detection Research Network and others, he is leading a multi-center study about serum proteomics to diagnose colon cancer. At the University of North Carolina he directed the k30 faculty development program since its inception in 1999 helping junior faculty build careers in clinical and translational research. And then finally on a personal note, David is particularly widely read. He is quite protean person and scholar and one of my favorite activities is actually talking with him about the latest books he has been reading. And in addition, I really value David as a friend. And so I think you're going to be very pleased to hear this lecture. David.
RANSOHOFF: Thank you for that nice introduction. I'd like a copy for my mom and my wife. Thank you very much. I want to thank the organizers for this invitation. I am really honored to be speaking at this meeting and in this auditorium. Second, by way of informed consent, much of this talk is new material and I’m not 100% sure how long it will be. So if you see me making a few mid-course corrections near the middle or end, that's why. But I’m expecting and I’m hoping that we have a lot of time for discussion at the end. More than normal, that's the plan. Barry and I have agreed on this. So please get your questions, your cards and your letters ready. I'd like to have some interaction and your thoughts on this.
This is the title, Evidence-based Medicine, Science, Policy and Politics. There is a disclaimer. And this is the goals of the talk which is to illustrate the principles and process with evidence-based medicine which we'll define in a minute. And to show the role of evidence, that is what gets generated at the NIH and elsewhere. Evidence obviously is necessary but it's not sufficient. Evidence doesn't compel anything, for example, evidence in an intervention work doesn't mean that people will use it. Conversely, people use things for which there is no evidence that it works. Prostate cancer screening would be one. We hope it works but we don't know that it works. And policy, which means -- literally -- wise government. These two things, evidence and policy, are over seen by our professions of science and medicine. Politics is going to refer to things outside evidence and policy. The overall goal is to encourage a rational approach to policy and implementation and to not let politics or other things distort policy too much. That's what we call the ideal world. The method will be to illustrate these principles and ideas by focusing on colon cancer screening as an example to understand the relationships of evidence, policy and politics. And I pick colon cancer screening, because as a clinician and gastroenterologist, and as a clinical epidemiologist, I watched the field of colon cancer screening evolve in all 3 areas, evidence, policy and politics. So I can describe these evolutions and perhaps some lessons.
First, we need to discuss briefly what evidence-based medicine is. And where did it come from? What is it supposed to accomplish? This definition comes from a widely-cited article by Dave Sackett, published in the BMJ. David is a clinical epidemiologist at McMaster who got recruited to Oxford to run evidence-based medicine there. Evidence-based medicine is the conscientious explicit and judicious use of current best evidence in making decisions about an individual patient. It uses the best available external clinical evidence from systematic research. Systematic means there is some framework to assemble and assess evidence. Research is often from basic sciences but especially patient-centered clinical research, meaning about accuracy and precision of diagnostic tests, power of markers, efficacy and safety and therapeutic and preventative regimens.
So these questions at the bottom, they are not basic. They are not ideology or mechanism -- the main things that get studied at NIH. The science of clinical research is different about diagnosis, prognosis and therapy. It's in a different category. Okay. Why was this field developed in the first place? And the answer, part of the story is that part of history is that preventive medicine was popular in the 1950s and 60s before it became popular again recently.
Things go in cycles if you live and watch long enough. And back in the 50s and 60s, it was assumed to be good. Like screening for cancer. The yearly physical exam. Yearly screening for diabetes, yearly chest X-rays and so on. But that assumption, that prevention was inherently good, was challenged by clinicians and clinical epidemiologists who wondered “how do we decide whether preventive intervention is appropriate to do?” Could prevention sometimes be bad? What if it doesn't work?
We think it works, we hope it works, but what if it doesn't? What if there are side effects from drugs to treat hypertension or being labeled as having hypertension?
David and his group showed in an article in the New England Journal, if you label people with hypertension, steel workers in Canada, they miss work. It's not from the disease, it's from labeling. What if we label people and we can't do them any good? These clinicians said we need evidence about benefits and harms to guide decisions about what is appropriate to do? And they set up a framework it that became evidence-based medicine to gather and weigh evidence, a conceptual framework best illustrated in the U.S. Preventive Services Task Force formulation of four key questions to decide, is it appropriate to screen? That can actually be translated to, “is it appropriate to do any intervention?”
The task force asked first, is the burden of disease high? Is it making a major effort to figure this out?
Second, does the disease left untreated, hypertension, lead to a bad outcome? Third, does screening or treatment reduce that bad outcome? Do blood pressure medicines work to prevent an important outcome? And then four, what is the balance of benefit-verses-harm?
The improvement of outcome-versus-harm for medications or invasive procedures – mastectomy, prostatectomy -- whatever it is, the weighing of benefit-versus-harm is formal, explicit, quantitative, sometimes using mathematical modeling where appropriate evidence is available. And then the task force applied these questions. This is its first task, to preventive measures starting at the annual physical exam, which was routine and popular in that era. And they deconstructed the yearly physical exam and looked at everything that doctors do and the result was that most parts of the annual physical exam were no longer supported by the task force, the American College of Physicians, or the AMA. Policy, the recommendations of major professional organizations changed besides systematic review of evidence about benefits and harms.
Interestingly, patient demand and insurance coverage perpetuated routine physicals making the annual physical an example of where evidence may be ignored in deference to habit or popularity. It looks good so it must be good, illustrating what may be called politics, forces and actions outside of evidence and professional policy.
Okay. Forget politics for a minute. The main idea is that what the task force did was to establish a process that was used to understand preventive and therapeutic decisions. This process, it didn’t spring out of Zeus’s head. This process was developed by human beings. And it was adopted and then adapted by some other organizations. But the U.S. Preventative Services Task Force has provided kind of a gold standard for how this thing is done.
The process of doing evidence-based medicine is detailed and time consuming. It's expensive. It may take well over a year, maybe several years to formulate the questions to assemble the evidence, do systematic reviews and meta-analyses and write reports to write evidence reports of benefits and harms, to develop recommendations, to get external review at several steps. In this process, not all kinds of evidence are going to count the same. This is important when we consider colon cancer screening as an example we are about to move on to. A clinical trial in general counts the most if you're trying to learn about benefit and also about harms.
It's a little bit like the FDA where it's required that you have one or two clinical trials of a new therapy. The task force isn't quite that strict but it leans in that direction. Any way, the task force based on evidence develops recommendations and they rank in a hierarchy and I want to show you what that is. Preventive services task force grades its recommendations according to one of five classifications reflecting the strength of evidence, clinical trials and other kinds of things. What is the magnitude of net benefit? Benefits minus harms? And the main thing I want you to see is there is a hierarchy.
A. And B are recommended. C is not recommend for or against. D is recommended against. This is the product of evidence-based medicine. This is policy -- the bottom line after you assemble and weigh the evidence and payors pay a lot of attention to this. The task force process is perhaps the ideal process or at least it's an important reference point when we are thinking about how other organizations handle evidence.
Okay, so that's the background about what evidence-based medicine is. Let's look and see how does this work in the real world by looking at the clinical problem of colon cancer screening.
There once was a time, if you can imagine long ago, when we didn't do colon cancer screening. I’m not going to ask how many people remember that era because it won't be that many people. From that time, evidence, policy and politics evolved. And we're going to examine selected topics, screening of average risk persons here with the standard tests, FOBT is fecal blood testing, sigmoidoscopy, colonoscopy – this is where most of the evidence and activity has been. We are going to look at new screening tests briefly, and then the surveillance of high risk people after they have had a polyp is called postpolypectomy surveillance. Each is a bread and butter issue in gastroenterology, clinical medicine, colon cancer screening with lots of significance for day-to-day care in clinical practice.
A very brief background about colon cancer, the magnitude of the problem. There are roughly 130,000 cases per year, 56,000 deaths. It's the number 2 cancer killer. Adenomatous colon polyps are precursors to colon cancer and they are very common. Did you know that 30-50% of people over age 50 have one or more adenomatous polyps? Most are very small. It's a disease or a problem of old age. Only 10% are over a centimeter and that's important because we think, we have some data at least, that they become cancer at a rate of 1% per year. The point is, if our job in the United States is to remove all polyps, that's a really big job. I’m not saying whether it should or shouldn't be, but this is a really big job. If we could, we would like to know which adenomas themselves will become colon cancer and should be targets to be removed.
Do we tell by size or histology? You have to take them out to learn histology and so forth. Keep this question in the back of your mind. This is going to be more important. Which adenomas, after you remove them, indicate future risk of colon cancer that is high because those people might need surveillance? We will come back to that.
Risk factors for colon cancer, age is important as it is for all cancers. Risk doubles with each decade. Family history is important. These are syndromes that are uncommon but they’ve taught us an enormous amount about physiology -- familial polyposis, APC – adenomatous polyposis of the colon – lives on chromosome five. Autosomal dominant, it taught us a lot about common colon cancer. HNPCC, hereditary non-polyposis colon cancer, involves chromosome two. A fascinating new mechanism of carcinogenesis. We learned a lot about carcinogenesis from these syndromes, but they are not common. Sporadic cancer is the most common and we don't know which chromosomes it lives on. We do know the first-degree relative is colon cancer and the risk is 2-3 fold and if that person is younger, it's increased by that much.
So that's the background.
Let's talk about screening for average risk people. In the beginning, meaning in the 1980s and the 1990s, before 1996, the guidelines for screening average risk persons were varied and they were heated guidelines from multiple organizations. And these were called blood testing alone, sigmoidoscopy alone and the combination and colonoscopy as well. Some organizations did recommend screening including the American Cancer Society and at one time the NCI recommended screening. It was in the guidelines business. It has since gotten out of that business. The recommendations, however, were generally not heeded because the task force, which was the dominant recommending organization, the task force didn't recommend and payors weren’t interested. There was no strong evidence. No strong popular political support. Doctors might do an office rectal exam and smear a guiac card, and that would be screening. But then important evidence began to evolve and let’s look at that evolution for each of the routine tests, occult blood testing, sigmoidoscopy and colonoscopy.
The principle of occult blood testing is to detect blood in stool in a guaiac-based test like Hemoccult-2 commonly used in the United States, detects the peroxidase-like activity of hemoglobin. It's not a direct test for hemoglobin. It’s the peroxidase-like activity. The method is give a patient 3 cards to look at three stools on different days. Two windows on each card. Develop a guaiac-based solution and if any window is positive, the test is considered positive. New occult blood tests detect human hemoglobin. A limitation of fecal occult blood testing is that some colon cancers may not bleed and if they don't bleed, you can't detect them. That's false negatives. The false negative rate for colon cancer is 70% or more. Most cancers of any one application at any one application of the test don't bleed. Or some people without colon cancer, have normal blood loss, making false positives. Everybody bleeds a little bit each day. Or for the guiaic false positives could be caused by vegetable peroxidase or myoglobin and so forth.
So stool blood a surrogate marker is pretty darn indirect and it's imperfect. But even with these limitations, important evidence of efficacy was obtained from clinical trials showing that fecal occult blood test screening reduces colon cancer mortalities by 33%. Mortality reduction of colon cancer using every year rehydrate.you rehydralyte the slide to increase the sensitivity. In the Minnesota study sponsored by the NIH and NCI, published in the Journal in 93, and by 15-18% in the every-other-year nonhydrated occult blood testing in the two major studies in Europe. In each clinical trial, if the FOBT was positive, then a colonoscopy was done to remove polyps. The role in these studies was work up, not as a primary screening test. These clinical trials were enormous big news in the field of colon cancer screening because they had been going on for 20 years and they were really long awaited.
There is a lesson here. This clinical trial evidence was so difficult to get, we are talking 20 years, 250,000 subjects in 3 clinical trials. This is the number 2 cancer killer. And it's so hard to get evidence for this. That is a serious problem. Who in the world is going to do this research? What junior faculty member is going to say, this is my investment. It will take decades for this trial to be done. Who is going to fund it? And if is there another way to get strong evidence? This last question is the Holy Grail question in clinical epidemiology.
Can we learn about cause and affect without doing an experiment at clinical trial? The short answer is no. Clinical trials in general are the gold standard for learning cause and effect. The Women's Health Initiative, we learned about estrogen and heart disease in a clinical trial that contradicted the what we thought we knew from the Nurses Health Study – it contradicted results that we knew from observational nonclinical trial data. Any way, there is no way to get strong evidence. This is a really important question.
In 2009, here is what is outstanding about occult blood testing. New occult blood tests immunochemical perform better. They have sensitivity of about 60% compared to Hemoccult at the same specificity. What role may they have? These have been endorsed by major organizations. Fecal occult blood testing is still alive as a colon cancer screening option. Can the cost and effort of false positives be reduced can compliance rates be improved? It’s kind of a cumbersome thing to do. Can it compete with other screening modalities for convenience and cost? The point is, it's still here and it's the only thing that right now we have clinical trial data about.
Let's look now at evidence about sigmoidoscopy. Clinical trial evidence is not available but there are two major clinical trials ongoing. There is the National Cancer Institute's PLCO, prostate, lung, colon, ovary trial of four different screening modalities – a clinical trial expected to produce results in the next years. And then there is UK trial, randomized controlled trial of once-in-a-lifetime sig. Case control evidence is available but it's generally considered less strong than a clinical trial. How many people know what a case-control study is? Okay. That's pretty good. If you don't know, I’m not going to be able to explain it in a sentence. But in a case-control study, it's not a clinical trial. It's a kind of study that you do backwards. You start with people who have the outcome, cancer or not cancer. Cancer mortality verses not. And you look backwards to see whether they were exposed to the thing you think might be protective. You may not understand that sentence but go read in a basic clinical epidemiology text about it. It's backwards reasoning -- the numerators and denominators are reversed compared to what you do in a clinical trial. It's a study that got major problems of design if you're getting the idea, which is why we don't trust case-control studies. But there is a case-control study that was strong enough, in this case, to be important. And here is that study. This is a study that shows that sigmoidoscopy screening reduces by 60% cancer deaths within reach of the scope. This is a straight sigmoidoscope for a case control study. And what was done from Joe Selby did from Kaiser Permanente in Northern California in a wonderful article in the New England Journal -- April 30, 1992 -- Joe looked at people who died of colon cancer, yes, 261, and asked how many -- and this is cancer within reach of the scope. How many had previous sigmoidoscopy. The answer was 9%. Among 868 people who had not died of colon cancer within reach of the scope, he asked how many people had sigmoidoscopy and the answer was 23%. A bigger number. A bigger rate than this. And this translates through odds, ratios and other sort of math to a 60% reduction for lesions within the reach of the scope.
Okay. Why was this case-control study -- because these are second class citizens and for good reasons. Why was this considered strong enough? The answer is he had some unusual strengths. First it was what was called a nested case control study. It's nested within a cohort. This is important because in case-control studies, if the control group is different from where the case group came from, that can affect results. This was a nested case-control study.
Second thing, the reason for exposure, why did they get the sig? Was it for screening or symptoms – in a case-control study, you often can't tell what the reason for exposure was. It was known in this study. At Kaiser Permanente, they were doing a clinical trial of sigs and they had a separate screening facility where the screenings sigs were done so they knew when somebody got a sig for screening. And then last there was an unusual internal control. There are so many biases that can affect case control studies. And what Selby and Noel Weiss, who was the research methodologist on that study said, there is some crazy biases here producing all of the efficacy of sigmoidoscopy, then it's going to work on the right side too and we're going to see protection on the right side. They didn't find protection on the right side. And that kind of internal control group you can do with this disease, very clever, that made this a strong case-control study -- one that in general we think we can believe in. And these strengths provided the basis for the U.S. Preventive Services Task Force in 1996 to make a recommendation, which was a very, very big deal. The task force in 1996 said, we are going to make this a Grade B recommendation. You can do sigmoidoscopy screening. The task force never recommended sig before. In 1989 there had been insufficient evidence. And this may not look like a big event to you. But in the guidelines-making business, and in the field of clinical epidemiology, what the task force did, it's like watching the Supreme Court make a decision. What are they going to decide? And what are they going to tell us about why they made that decision? This was a gigantic decision to accept case control evidence and in fact, the head of the U.S. Preventive Services Task Force at that time, who is now the editor of Internal Medicine said, the 1996 task force will be known how we handle sigmoidoscopy because it wasn't studied in a clinical trial.
As we watch the evolution of evidence and policy, we are seeing several kinds of evolution here. First evidence is evolving, we have clinical trials of occult blood testing that work. And in case-control studies, methods are evolving. There is a nested case-control studies improving our understanding of exposure and the policy making process is evolving. Case-control study here is being considered strong enough to base a recommendation. So in 1996, here is how things stood. Occult blood testing, yes, FOBT and sig – no. Hasn’t been tested in a clinical trial or case-control study Colonoscopy hasn't been tested. That's where things stood.
Let's move now to colonoscopy. We are going to spend a little bit more time on colonoscopy for several reasons. It’s clearly important. It's how in the clinical trials, the occult blood testing colon cancer mortality got reduced and no we know it's popular as a screening test. There is no clinical trial. And there is no case control study. No strong case-control study saying it works. So how did it get to be recommended and popular and what does that teach us?
This is a story that involves evidence, policy and politics. The concept of screening colonoscopy evolved dramatically over 15 years. In 1992, screening colonoscopy was a lunatic fringe idea. If you were an officer of the American Society for Gastrointestinal Endoscopy, you could talk about that in the privacy of your own home but didn't talk about screening colonoscopy in polite company. It was way out there. In 2009, screening colonoscopy is a Medicare benefit. How did this evolution occur and what are the lessons for evidence, policy and politics?
Here is the story. And I think this is instructive. Before 1992, we discussed no controlled studies that support any screening. In 1992, the were the sigmoidoscopy case-control study from Joe Selby and then the clinical trials of FOBT that provided general but indirect support for colonoscopy. And in 1993, the Journal published results in the National Polyp Study which will play a role in this NIH and NCI-supported study. And the title of the article here was, Randomized Comparison of Surveillance Intervals After Colonoscopic Removal of Newly Diagnosed Abnormal Polyps. This is the first study that the National Polyp Study published in the Journal that year. And its goal was to learn among people who had an adenomatous polyp removed, what should be the timing of surveillance colonoscopy? Because they were thought to have a very high risk of adenoma and cancer and you had to do a colonoscopy within a year or two. The study design was, this was a clinical trial, 1,400 patients at 7 centers. Polyps were removed and people were followed to the development for new polyps and cancers and this was the design. Two arms here, Arm A, was the intense intervention. So Arm A got a colonoscopy year one after the adenoma was removed and then in year 3 and 5, Arm B was the same except they didn't get intervention at year one.
That was the clinical trial question that was being tested. If we don't do a colonoscopy at year one, will you get cancer or will you live or die? That was the question. And what they found was that there was almost no cancer in either arm. There was like two in each arm. Very, very early cancers. A very different result than they expected. And the percentage with adenomas with advanced pathology was the same. This isn't a very high percent. We didn't know how ominous this was.
The natural history after you had a polyp removed was much less ominous than we feared. This is an unexpected result. People thought if you had adenomatous polyps in the 90s, don't buy any magazine subscriptions or long playing records. You're not going to live long enough. It was a terrible thing. This is what got learned. This is important new knowledge to know the natural history is so benign. Then, the National Polyp Study decided to do a very different analysis using the study data. This was their second article in the New England Journal that year and this is one of the most cited articles in gastroenterology, The Prevention of Colorectal Cancer by Colonoscopic Polypectomy at the end of 1993. The purpose here was to learn does polypectomy reduce colon cancer incidents? This does relate to screening. It's a different question than they set out to answer in their clinical trial.
The design of this analysis is not a clinical trial. It was a cohort study. In which the intervention group was made up of the combined arms of the NPS, because they both had the same result. Cancer didn't happen. And then the comparison group was historical controls of three cohorts from other settings. This is called a historical control study. As evidence, this generally doesn't count real strong compared to a clinical trial. And here is the result. These are accumulative incidents of colon cancer. And the National Polyp Study is in black at the bottom. A low incidence of colorectal cancer that was much less than that from the Mayo Clinic in a cohort study from Saint Marks in England and then from surveillance epidemiology and the end result of federal database that showed results of cancer in people in comparison were much higher. And from this study, the National Polyp Study, from this result, the study concluded 76-90% reduction in colon cancer incidents compared to groups that didn't get colonoscopy but here is the potential limitation using controls as the comparison group. In the National Polyp group, here everybody had colonoscopy before they were followed because to get into the National Polyp Study, you had to have a colonoscopy to find the polyp and people who had cancer were excluded.
In contrast, in these other groups, nobody got a colonoscopy at baseline so people with asymptomatic cancer could be included and they would be counted as new-incidence cancer later, making the cancer incidents appear to be higher in these 3 historical control groups. Now the National Polyp Study, they were aware of this problem and they in their analysis, tried to adjust for this by making assumptions about how long colon cancer might dwell and be asymptomatic, and they excluded some cancers from the comparison groups. But the problem is you can never be sure in a historical control design if the assumptions you're making are appropriate. That's what you're trying to test and learn. You can never be sure if the assumptions are appropriate. Which is why we are cautious about how a historical control study should be interpreted. But this study was very influential. And while the National Polyp Study investigators were cautious in their interpretation, others seemed to pick the 90% number. Here is a typical 90% statement from a trade journal from last year, 2008. More than 150,000 people diagnosed. 90% could have been saved if they had gone for screening colonoscopy. All figures are available. The point is, there is no evidence or strong evidence to support this based on evidence we have now. The efficacy of colonoscopy is closer to 60-70% which would be great for a screening test.
Let's go back then to the guidelines. This is 1996. The task force was not persuaded by the National Polyp Study. The National Polyp Study was published 3 years before but the task force is still saying insufficient evidence to recommend colonoscopy and that is where things stood in 1996. And then things sort of began to accelerate and now it's going to get real interesting to figure out why did colon cancer screening get so popular? I don't think this is a fasten your seatbelt moment but this is the next parts of the story are kind of dramatic.
What happened in 1997 was that there was a new guidelines-making group that was formed called the Consortium. It was initiated by the subspecialty GI societies, the American Gastroenterology Association, and the ASC and so forth and supported by the ASPCR, the predecessor of AHRQ. The consortium was led by gastroenterologists and generalists, Bob Fletcher, methodologists, and the BMJ. I got approached early on to be the co-chair of this and I don't think I would have been a strong candidate but I immediately said no, I won't serve as chairman of this. And you shouldn't have GI organizations doing this. This should be done by a totally neutral party, generalists and methodologists like the U.S. Preventive Services Task Force. And I think there are issues in conflict of interest even within the profession that we have to manage. Anyway, this consortium decided that colonoscopy was an option, in part because they used somewhat different and less stringent rules of evidence than the task force illustrated an interesting phenomenon. In the United States, there are over 300 guidelines-making organizations. According to the National Guideline Clearing House, which is on a government website, and many guidelines are on the same disease and overlap. So it there is difference in recommendations, there is no way to adjudicate between them like in the court of law, the Supreme Court and the appeals courts help us resolve differences. This is an exaggeration but it can happen, but if you don't like a guideline for whatever reason, you can make your own guideline-makings organization. There are over 300, and set your own rules. I saw one subspecialty group making some guidelines. I was in a meeting and they said, there are no clinical trials in our field so we'll accept case-control studies and observational data as Grade 1 evidence because we don't have clinical trials to work with. It was just crazy. But the point is, some guidelines groups are better than others. Some are clearly higher than others on the totem pole, for example how they are regarded by Blue Cross, Blue Shield or CMS Medicare.
There is a lot of guideline-making organizations and not a clear way to adjudicate among them. So, here we are back at the evolution with the National Polyp Study in 1993. The Consortium in 1997, and then the biggest events in the business. In the year 2000 there were two major events. First was Colon Cancer Awareness Month, March 2000, with Katie Couric on the cover of Time Magazine and her colonoscopy broadcast on the Today Show. After her husband died at age 42 of colon cancer, she became the most visible advocate. And then, an even bigger event, the July 20, 2000 issue of the New England Journal that had two articles reporting colonoscopy finds right sided adenomas that are not found by sigmoidoscopy. One was David Lieberman's VA study Use of Colonoscopy to Screen Asymptomatic Results using colonoscopy and the other was Tom Imperiali’s study of the Eli Lilly Group where people got colonoscopy as a benefit for employment -- The Risk of Advanced Proximal Neoplasms According to Distal Colorectal Findings. Both had similar results and the findings were widely expected. They were in the New England Journal because they hadn’t been documented before. My teacher said, if the obvious hasn't been documented, document the obvious. That's what is happening here. But these were not unexpected results. They turned out to be a big event because of how the results were interpreted and I was an author on the second study and it helped train Tom when he was at Yale. The articles supported the colonoscopy screening in an average risk person was 1% and 5-10% advanced adenomas based on size and histology. It's a big number but nobody knows natural history. The reason this lesion got invented and is used, I'll tell you a secret, the main reason it's used is because clinical researchers wanted an outcome more common than cancer. This lesion had been to be invented so we could have outcomes to count and studies but we don't know the natural history of it, but it looks like a big number. We don't know the natural history. And this is a very big issue right now.
The article showed the sigmoidoscopy would miss most proximal lesions depending on the threshold you use, what kind of lesion in the sigmoid is considered a sentinel lesion to indicate doing a full work up after a sigmoidoscopy. All was expected by everybody who is in the field. But then here was the big thing. There was an editorial that interpreted the data very strongly and said, it's time to go the distance and that colonoscopy is true colon cancer screening. The edtorialist was not a clinical epidemiologist. He was a basic scientist and head of gastroenterology at NGH. This editorial went way beyond the data, in my view, as we would have interpreted it in our study, for reasons I will explain shortly. In the meantime, this story got reported by the New York Times, on page 1. And when I saw this story in the New York Times, it was below the fold and I picked it up and I turned the paper over and saw this on the bottom left hand corner. I said, how in the world is this getting on the front page of the New York Times? Because it's not news. It's what everybody expected. Like the task force in making recommendations. It's documenting the obvious. So what’s the hook or the point that would make this front page news? And this is important because this is going to affect how we see colonoscopy in the United States. Here is what it said.
The test most commonly recommended to screen healthy adults in healthy adults, should be replaced by a more expensive procedure that examines the entire colon. We were relying on sig being able to detect sentinel lesions on the left and if you have rioted right sided lesions you’ll have sentinel lesions – so if you find anything on the left, you work up and find something on the right. That's what this story is saying these studies proved was wrong. And the story said sigmoidoscopy which is cheaper and easier to perform has been used on the optimistic theory that if no abnormalities were found in the colon, none would be found higher up. On an optimistic theory, none were likely to be found? Did people think that sigmoidoscopy detects right-sided cancer because of sentinel lesions? The answer is no.
Previous modeling, cost effectiveness and recommendations for sigmoidoscopy like the taskforce hadn't relied on the ability to detect right sided lesions, nor did modeling or recommendations after the year 2000. So does evidence suggest as reported in the Times, because this is going to be important, the test most commonly recommended should be replaced by a more extensive procedure. The answer is no. We will talk in a minute about why. I know it doesn't make sense to you but we will talk about it in a minute. The editorial and subsequent reports had enormous influence. One lesson is that editorials are in general maybe the weakest part of most journals which I say as somebody who has written them and edited them. They don't get peer review. They are under a tight production process, especially if you're a weekly, editorials can be weak. Any way, here is what the guidelines said following these articles.
Colonoscopy was an option here. The only group that says, preferred, is the American College of Gastroenterology which is all gastroenterologists. At least there is some generalists and methodologists in each of these groups. The point is that the studies published in 2000 didn't persuade the most authoritative groups that colonoscopy was the best test. So is colonoscopy the best test? If not, why not? And to address this, the task force in the Institute of Medicine did analysis of four cost effectiveness analyses. This is policy making at some of the most explicit levels and it's hard to do and it's expensive to do. This is the kind of thing that we need to do. And what they found was, and Mike Pignone, a trainee in one of our programs at UNC was a key author on both of these, the head of general medicine at UNC now. What these studies found was, at any one application, colonoscopy is the most thorough. That's obvious to start. But colonoscopy in a program of repeated testing is neither the most effective or cost effective option. It's not even the most effective.
Why not? And the answer is in biology. This is what they learned by looking at models. Long screening intervals with missed cancers with fast growth rates and models that they looked at used different assumptions about growth rates and dwell times. How long does it take for a cancer to appear from a polyp to grow to kill somebody? Is there a spectrum of growth rates? We don't know. And what you assume about dwell times makes a huge difference on the outcome. And a less sensitive test applied more frequently might in a program of repeated testing, might be more effective than colonoscopy. It can detect fast-growing lesions better. That's why every 10 years, it may not be the most effective. What these analyses found was a program of sig plus FOBT was as effective or more effective than a program of colonoscopy it's not just because it's less expensive. Program of sig plus occult blood testing might be more effective regardless of cost.
This kind of analysis is counter intuitive. The concept I hope is pretty clear but you don't get to this point until you have done the quantitative rigorous analysis of the evidence. This is very counter intuitive for most clinicians and most patients. I liken this to pilots who, when they fly, and when it's confusing, when it's night time or in clouds, you have to trust the instruments. You can't trust what feels intuitively right. It's very hard to discuss this with many clinicians, even clinical researchers. And certainly patients -- it doesn't make sense this. This is kind of hard. We as policymakers and guardians of this body of knowledge and so forth, we have to develop the routes and trust the instruments. It's hard to do. This makes a big tension between evidence, policy and politics.
And I am going to make some mid course adjustments and I’m going to stop in a couple of minutes. We are going to talk briefly about new tests and then skip surveillance, even though it's a wonderful important topic. New tests include immunochemical FOBT that is more sensitive in guiaic-basde tests. It's not widely available but evidence in the last two years shows that this is much more sensitive than Hemoccult. It may be 30% sensitive for cancer. And immunochemical occult tests 60% sensitive. That's a big deal if fecal blood testing using Hemoccult was cost effective before. The new tests are better, not widely available in the U.S. This may change. Recently been endorsed and supported by the U.S. Preventive Services Task Force. Virtual colonoscopy, CT screening, sensitive and specific, will it be reimbursed? CMS is in the process ever deciding that. Stool DNA markers showed in an earlier study I was an author on that the sensitivity was 60% but the cost is high. The group that is working on it is trying to figure out how to make the overall process more sensitive.
And I’m going skip surveillance even though it's wonderful but I'll put the slides on the web in case anybody wants to hear them. I want to save some time for questions. This slide shows what the conclusion about surveillance was going to be. If colon cancer screening business is an iceberg, screening is on the top and surveillance, what you do with people who have been found to have polyps, that's the main part of the iceberg. If you do enough exams on people, we'll find adenomas in a huge number of people and enroll people in surveillance. This is where a lot of effort and expense and risk is involved in something that we, the profession, need to manage. So the last couple of slides are, and then I'd like to move to your cards and letters and questions, please so we can have some discussion about this.
The purpose was to illustrate the principles of evidence-based medicine and to show the roles of evidence, policy and politics and to encourage a rational approach to policy. The last slide is that evidence-based medicine provides a framework to answer. Should we do some intervention? Whether it's prevention or whether it's therapy. We ask several questions about the natural history of disease, left untreated, does intervention improve that? Is there a cost or harm? That's the conceptual framework which when done well is common sense but making it quantitative and explicit is a tough thing to do and real important as clinicians and policymakers. Evidence is obviously critical to the side policy but it doesn't compel anything by itself. Policy can be made by different organizations and can vary and there is no way to adjudicate among organizations. I don't know what we can do about that. But this is a serious problem. Do you remember the ACG point about colonoscopy as the preferred option? I worry that in the clinical behavior business, the heart beats to the fastest pacemaker. If somebody says, do it, maybe that is what clinicians will do. I know many people in practice who cite the American College of Gastroenterology. It's a professional organization. It's got this opinion. It's in a guideline. How do we adjudicate or control? There are no formal way. I can't envision is that happening. But this is a problem when we have all these guideline-making organizations, they aren't all the same.
Last, politics. The things outside of the science and what we ought to be doing, this needs to believe managed in order to serve policy.
And I'd like to end here, thank you very much for your attention and I hope you have your cards and letters and questions ready. Thank you.
[applause]
KRAMER: That was wonderful. Are there any questions?
QUESTION: While people are coming to the microphone. As you mentioned, there was a tectonic plate shift in the mid-1990s as a result of case control studies that Joe Selby did and you mentioned the internal controls, that is the assumption was made they had only done colonoscopy, they would have picked up the cancers on the right side and would have lowered the mortality rate on the right side. Since they didn't, sigmoidoscopy had its own internal control and was proven. Now more recently, a case controlled study came out of colonoscopy and showed that the benefit, at least protected through the case control study, from colonoscopy was restricted to the left side of the colon, which is exactly where sigmoidoscopy picks up the lesions. And I wanted to know how you put that in the context of how Joe Selby’s original study was judged.
RANSOHOFF: So your question is the Baxter study which appeared in the Annals of Internal Medicine a little while ago. That invalidates the thinking, I think of the Selby study is what you're asking. It's a good question. And it's hard to think about whether it does or not. I haven't seen anything written about that. It's definitely worth thinking about this. This is now challenging what has become dogma in the GI world and it's a fair question. I’m really not sure how to answer it. What happened in the Baxter study in the Annals a couple of weeks ago, case control study of colonoscopy, found protection of 60% on the left. No protection on the right. 0%. And this was colonoscopy. And this was a gigantic shock to the GI world and to the clinical reach world. Among the possible explanations are the people who were doing -- there is a bunch of explanations. The people doing the procedure were gastroenterologists and 70% weren't in Canada, where it was done. Was it not a complete exam? Did it not do a thorough inspection? Did it not get to the cecum. All of those things need to be adjudicated. I think people may come back to your question, but it will be one that is very uncomfortable for people to look at.
QUESTION: Two questions. Number 1, if you have a tough one that showed benefits, for example, sigmoidoscopy and now you come up with a more modern test, say your stool DNA battery, how much testing do you need to be confident or have a guideline on the new test.
RANSOHOFF: This is a real important question. Dr. Greenwald is asking if you have got evidence like from a clinical trial that shows occult blood testing Hemoccult works, which 2 does, we have that from clinical trials. If you dependent a new stool DNA test or some other test to detect polyps or early cancers, do you have to do another clinical trial at the outcome like the initial clinical trials that were done? And the general thought in the field is, this has been questioned by some people. If you have a clinical trial which shows that mortality gets reduced, and you have got a pretty clear idea of what is the intermediary metabolism, discovering early cancers or large polyps, whatever it is. Then if you do a study or if you can show a diagnostic test detects those things, that may be enough to approve it without going on to a clinical trial of outcome. Does that answer your question?
QUESTION: Second question. A little off the topic. My impression of it, I don't have data. Are there a lot more upper GI endoscopies because of obesity and reflux disease. Are there any data to support doing the esophgogastroscopy?
RANSOHOFF: if we have four hours -- let's do that off line. That's a really important topic and with some articles, this will be more important in the near future. Yes, sir.
QUESTION: Colonoscopy is number 1?
RANSOHOFF: Yes.
QUESTION: So like it's still pretty invasive so you don't get a lot of patients getting their colonoscopies.
RANSOHOFF: Gastroenterologists don't think it's invasive. I’m kidding.
QUESTION: I did research at comprehensive cancer center and they found a protein a couple years ago like 70-fold higher in the blood. And I –
RANSOHOFF: They found?
QUESTION: Some protein they found –
RANSOHOFF: The CCM1.
QUESTION: Yes. Yes. CCSP1 and 2. So there are any protein blood tests coming out?
RANSOHOFF: No, there are people trying to study. I had done a study with the early detection research network. There is nothing – a noninvasive test for colon cancer, this is the Holy Grail in the cancer screening business. A lot of promise. Nothing that passed rigorous testing in the right kinds of patients. We can be hopeful. I think the work is terrific. Very clever stuff but finding the right specimens in populations to test, that kind of analyte is very hard. Hopefully that will happen. You have a question sir, and then you.
QUESTION: Thank you. You mentioned in passing that you had concerns about conflict of interests in participating groups. I think it's important to draw attention to the fact that the screening procedure which is the most expensive one, was recommended by the organization that is comprised of people who do that screening procedure.
RANSOHOFF: I don't get it. What are you driving at? I’m kidding.
[laughter]
QUESTION: Don't you think that it would be appropriate for the science community if they want to continue to be seen as impartial scientists to turn over the assessment and analysis of their data to groups that have no direct or indirect conflict of interest with the outcome?
RANSOHOFF: Yes. And that's the reason that when I was approached to consider being co-chair of the consortium, I said no. This is important. Whenever I review an article and they ask about conflict of interest and it's about colon cancer screening, I have done this. I put on the bottom, I’m a gastroenterologist. I get paid in part because of the colonoscopies that I do in my division. This is an important kind of conflict of interest. The professional organizations in part have become trade organizations. There is mixed roles in them. And this is something that I believe -- I think we need to handle this, we as a profession, need to handle this in a better way. That is very important. And that, in my view, is why I think an organization like the task force which really is neutral, they have methodologists and generalists and they rigorously manage conflict of interests. They need to be paid attention to most. Thank you for bringing that up. It's delicate but I think very important. Question here and then here.
QUESTION: Yes, thank you. So I understand that good randomized clinical trials trump a good case control study. One thing I wonder, in the development of guidelines, does a good case control study trump a bad RCT?
RANSOHOFF: I’m sorry. What?
QUESTION: Which is better, I guess. A bad RCT or a good case-control study?
RANSOHOFF: Bad RCTs can be pretty bad. A good case-control study, they can be far and few between, that's better than a bad RCT. I mean, there really are bad RCTs. We have ways of measuring quality. But yes, in general, an RCT doesn't automatically trump. RCTs have different degrees of quality and there are many instances in medicine where RCTs conflict. It's not as if all RCTs give the same answer and it's the truth. We have to look at the quality as we do with case control studies.
QUESTION: I’m Shannon Brownlee.
RANSOHOFF: Hi.
QUESTION: Just to let you know who I am. I have two questions. Why have we not launched a large randomized control trial of colonoscopy, and 2, is it a reasonable decision for an individual to decide not to get screened with colonoscopy?
RANSOHOFF: For the second question, absolutely. The recommendations are people in general ought to get screened according to the guidelines recommending organizations. But yes, colonoscopy shouldn't be considered the preferred. If you look at the recommendations on the books, there are some twists to this. But basically any of several things is okay. The business about why not do an RCT of colonoscopy is a tough one. And NCI had a conference about this in 2001 in which I participated in which they were trying to figure out whether to fund it. The short answer is such a trial would be very expensive, especially in the United States because there would be so much contamination of the control group and I think we, as a country, need to consider whether evidence obtained in other countries where it's easier logistically to do such studies would be appropriate. For example, Norway is planning a European study, clinical trial, based in several countries and it will be much less expensive than what we can do here in the United States. And given the Baxter study and its recent results, the need for having a clinical trial is more urgent than it was in the past but logistically, it would be very hard to do. Party -- Peter do you have thoughts about anything to add about clinical trials?
[low audio]
[low audio]
RANSOHOFF: Yes, sir.
QUESTION: do you know anything about the status of the -- [inaudible]
RANSOHOFF: Carcinoembyronic antigen? It's not a good marker for early cancer. Carcinoembyronic antigen how I started my career at yale doing an autopsy on why carcinoembyronic antigen, claimed by Phil Gold to be a blood-based test for colon cancer, why that crashed and burned. Why the initial results looked so positive and why over time the results got so discouraging and the article that we wrote when I was a fellow doing an autopsy on why it didn't work, became an article in the New England Journal which established the rules of evidence for evaluating diagnostic tests. And I was at the 25TH anniversary of the Montreal Cancer Center that Phil Gold developed the assay and we have a wonderful discussion about this. It is an example of a molecule that refuses to die. It wasn't good for colon cancer screening, it was initially promoted for. And it then got tried to be used, tell us about reoccurrence of colon cancer and the yes is -- answer is yes, but by the time it’s positive, now you’ve got metastatic disease can't do anything about it. It was a very interesting idea. It started off a whole field. But in my view, it doesn't have a use in the diagnosis or from what I know, about the management of colon cancer.
QUESTION: so, as a Medicare beneficiary you mentioned that the new guidelines…
RANSOHOFF: CMS supports colonoscopy. They are trying to think about whether to support virtual colonoscopy. They do support colonoscopy.
QUESTION: Every 5 years? 10 years.
RANSOHOFF: I think it's every 10 years, and if you’ve had an adenoma – any adenoma -- even a tiny one. CMS will pay for every two years for the rest of your life. Now consider that up to 50% of people that age may have adenomas. If our job in the United States is to make sure that everybody has no adenomas, gastroenterology will have a lot of business. Even my GI friends say, I think the CMS guidelines are too aggressive. They are not really guidelines. But CMS pays for screening and then with an adenoma they pay for surveillance, the thing we didn't have a chance to talk about.
QUESTION: Taking into account the rate of complications during colonoscopy, in comparison to sigmoidoscopy or other less-invasive methods, are these rates taken into account when you do evidence-based guidelines?
RANSOHOFF: Yes. This is very important. It's the harms of screening. GI doctors, we don't think that colonoscopy is that invasive. But it is. People bleed. They perforate at a rate of .2%. these are the kinds of things that the task force considers. Other organizations may not consider them. The recent ACS guidelines, for example, the American Cancer Society, and the consortium guidelines that came out last year, the consortium now teamed up with the ACS, they did not consider harms at all. That's a serious deficit in the making of guidelines. Some countries that say colonoscopy, people will get hurt. People who won't benefit at all and this is one they don't adopt it at all. This is a really – colonoscopy is in general safe, but when we're talking about screening, this is an invasive procedure and yes, harms, absolutely has to be considered and in the appropriate evidence-based considerations it is taken into account.
KRAMER: We have come to the end of the time but feel free to try to ask additional questions of David in the lobby. There is food up there.
[applause]
(Music fades in, under VO)
ANNOUNCER: You’ve been listening to a lecture on the subject, “Evidence-Based Medicine: A Story of Science, Policy and Politics,” presented as part of the NIH Clinical Center’s “Great Teachers” lecture series, recorded at the Lipsett Ampitheater at the NIH Clinical Center in Bethesda, Maryland, on January 14, 2009. Our speaker was Dr. David F. Ransohoff, Professor of Medicine and Clinical Professor of Epidemiology, and Director of the Clinical Research Curriculum at the University of North Carolina School of Medicine. A close-captioned videocast of this lecture is available online at http://videocast.nih.gov. Join us next time when NIH Clinical Center Grand Rounds brings you two speakers. Dr. Howard Markel is the George E. Wantz Distinguished Professor and Director of the Center for the History of Medicine and Professor of Pediatrics and Communicable Diseases at the University of Michigan Medical School in Ann Arbor. His topic will be, “Designing the Historical Atlas of the 1918-1919 Influenza Pandemic in the United States.” Our second speaker will be Dr. Jeffrey Cohen, chief of the Medical Virology Section and Senior Investigator in the Laboratory of Clinical Infectious Diseases as the National Institute of Allergy and Infectious Diseases at the NIH. His topic will be “Smallpox Vaccination at the NIH: Unexpected Findings.” NIH Clinical Grand Rounds podcasts are a production of the NIH Clinical Center, Office of Communications and Patient Recruitment. For more information about clinical research going on every day at the NIH Clinical Center, log on to http://clinicalcenter.nih.gov. From America’s Clinical Research Hospital, this has been NIH CLINICAL CENTER GRAND ROUNDS. In Bethesda, Maryland, I’m Bill Schmalfeldt at the National Institutes of Health, an agency of the United States Department of Health and Human Services.
