NIH CLINICAL CENTER GRAND ROUNDS
Episode 2009-006
Time: 1:04:35
Recorded Feb. 18, 2009
THE WINNER'S CURSE: THE MARKET FOR EXCHANGE OF SCIENCE
Dr. Neal Young
Chief, Hematology Branch, NHLBI
ETHICAL CHALLENGES FOR MEDICAL JOURNALS
Dr. Christine Laine
Senior Deputy Editor, Annals of Internal Medicine
ANNOUNCER: Discussing Outstanding Science of the Past, Present and Future – this is NIH Clinical Center Grand Rounds.
(Music establishes, goes under VO)
ANNOUNCER: Greetings and welcome to NIH Clinical Center Grand Rounds. We have two speakers for you on today's podcast. First, Dr. Neal Young, chief of the Hematology Branch at the National Heart, Lung and Blood Institute at the NIH will speak on the subject, "The Winner's Curse: The Market for Exchange of Science." He'll be followed by Dr. Christine Laine, deputy editor of The Annals of Internal Medicine, who will speak about "Ethical Challenges for Medical Journals." If you would like to see a close-captioned videocast of today's subject, log on to http://videocast.nih.gov and click the "Past Events" link. We take you to the Lipsett Ampitheater in the NIH Clinical Center in Bethesda, Maryland, where Dr. Frederick Ognibene, Director of the NIH Clinical Center's Office of Clinical Research Training and Medical Education, and Director of the Clinical Center's Clinical Research Training Program, will introduce today's first speaker.
(Music fades)
OGNIBENE: So our grand rounds speakers today will examine a timely topic, namely how scientific data are judged and disseminated. Dr. Neal Young, chief of the Hematology Branch of the National Heart Lung and Blood Institute will talk about the “The Winner's Curse, The Market for Exchange of Science.” He'll be followed by Dr. Christine Laine, senior deputy editor of the Annals of Medicine, who will talk about “Ethical Challenges for Medical Journals.” In addition to his position at NHLBI, Dr. Young has directed the trans-NIH center for human immunology, auto-immunity and inflammation since 2007. He is Harvard graduate and received M.D. degree from Johns Hopkins School of Medicine. He interned and completed his residency training in internal medicine at the Massachusetts General Hospital. His research interests include human hematopoiesis, aplastic anemia, and the pathogenesis and treatment of bone marrow failure, PAVO viruses, hepatitis viruses and the immune response to viral infections. He is also involved in research on apoptosis, the role of cytotoxic lymphocytes and disease, interferon, cytokines and other interleukins and the mechanisms of genomic instability. He is a prolific writer and his memberships include the American Society for Clinical Investigation, the American Association of Physicians, the American Federation for Clinical Research, the American Society of Hematology and the International Society for Experimental Hematology. He is also a master of the American College of Physicians. The paper Dr. Young will discuss today was published in PLoS, the Public Library of Science And Medicine, last October. Neal.
YOUNG: So thank you. This is actually an experiment for me in many respects. I think it's the first time first that I have ever volunteered to give Grand Rounds, having been here for 30 years I usually don't do that. Second, I’m not talking about the scientific work with -- that my laboratory or the clinic has performed. That's easier because you tend to practice those talks ahead of time. This is something that's outside of my usual official work duties. And although I was very tempted to try to do this talk entirely without Power Point slides, I’m not going to be able to get away with that. So I’ll give you a couple of slides as an introduction, putting myself and you at ease that we’re all in good hands. Then I’m going to read some segments of this paper published many -- this is the long version of the paper, I doubt many of you have actually read it and I hope at the end you'll be inspired to look at that.
So I have to put in this potential conflict of interest. I do edit a journal and I’m talking about publications so I need to put that down. We also have to have CME goals. So the first one is to understand, I thought the first one was maybe I can have you understand the importance of a subscription to Seminars In Hematology but it's actually to understand at least from a different perspective a system of publication from an economics point of view. The second is to read, again I would recommend that you read Seminars in Hematology – but for this, it’s to read for substance, not simply for the venue. Enough of that joke. The third is to experiment with publication venues.
So, this particular project had its origin in some reading of my own, but it developed from this original idea to several other accessory thoughts. These are the subjects that I do want to cover very quickly. I want to tell you what “the Winner’s Curse” is in economic terms and why I got interested in it, how it relates to scientific publication, the issues of the curse as it affects impact factor and the mismatch in the market for information, some aspects of scientific uncertainty, path determination and herding, and some of the consequences of this analysis.
I read a paper on genetics about two years ago that got me thinking about “the Winner’s Curse” and I spent a very long winter holiday in my wife’s house in the Apennines. It was extremely foggy and we couldn’t go anywhere. I read a lot of economics and I had discussions with a young physicist who was in attendance and with a burgeoning economist -- he doesn't look very much like Chicago intellectual in that photo. There's other ways I was inspired to complete a manuscript but needed adequate co-authors. I was lucky to find Professor Omar O’Bailey, a professor at George Mason University, but at the time graduated at the department of economics at University of Chicago. As a process of convergent evolution a very valuable collaborator who has done the empiric work on the validity of the medical scientific data that we rely on in publications. He published this inflammatory medicine some years ago which I recommend you as other publications as well.
The result of this is we did publish this paper in PLoS Medicine. Though I was accused in the blogosphere after this blog came out of getting sour grapes, not getting too many rejection notices and I certainly have gotten too many in my opinion. This paper actually attracted a lot of attention. I was astounded that we can get it published. There's a story to that, not to hard to infer. But I was called within a week of the paper coming out by colleagues here in Europe and the United States that I was on cover of the Economist and there indeed was the winner's curse. I want to go back. Can you get that running for me?
Here is the economist article. Off its website, this is the lead of their science articles, a very nice write-up, they don't get it all entirely right. I had never gotten that notice before for any of the scientific work that I had done. Several weeks went by and one of my children called me and said now you're on Slate. So Slate also just instituted a program of video clips on science, it's very entertaining. This is a first, we're all going to dance as this comes up, I think this is a first for Grand Rounds. Here is the initial Slate presentation. Just give you the beginning of that. I’m very grateful to Terrence for this excellent audio-visual, I would not have wanted to conquer this on my own.
VIDEO: No, don't do that. That's not the only way we're going to learn.
Hey, so you know that break through headline you just read on Digg? The one about how scientists can totally zap memories out of your brain? According to these researchers the very fact –
YOUNG: I was pretty delighted to see this.
VIDEO: The argument is that prestigious science journal, ivy league school, don't have enough room for the over achieving point detectioners apply. It might envision the scientists -- (overlapping speakers)
YOUNG: Right there. That's good. Then just the last slide I want to show you, a couple of weeks ago I also receive third degree from colleague in a German economist newspaper and you can see the translation that start, it was a particular thrill to me myself, this Brock young was so young in this German article which is a very good compilation of what I say.
That's the background. Now I do want to discuss some of the content of this article. So that's the end of the Power Point. So how did this start? With he my reading a paper in a review paper in genetics. Actually, I’m happy to say I can still occasionally read papers with content. This paper described the fact that one has to be cautious in looking at the first linkage analysis that's published because of “the Winner's Curse.” The overstatement of a relationship between a genotype and phenotype. And like most of my colleagues with the exception of those few who are serious geneticists I have no idea what the winner's curse was. So I have asked lots of people and I haven't had anybody come up with a good definition.
It's a very familiar principle from auction theory in economics. What it fundamentally says is that we -- there's a tendency in an auction to overvalue for the winner to overvalue, to bid too high for whatever is being auctioned. I’m going to describe that in greater length in a moment. But the functionally what this means, I think the easiest way to think about this, if you think about also of people doing a linkage analysis, trying to relate a gene to a phenotype. I’ll use an example right here. So I have Omar, my co-author here, he's a very lucky fellow. He gets a result, an excellent laud score, sends it off to the Annals of Internal Medicine, it looks like a relationship between some annotated gene and phenotype. And it’s published. That's great. Meantime, Mark and Fred are not quite so lucky, they're doing the same sampling and they're getting a result that isn't nearly as striking from a statistical point of view. And first of all they lost the race to Omar so it will take a while convince another that publish not so striking results. We're pessimistic people, we’re also sampling. We don't get any relationship. We don’t see a relationship between the gene and phenotype. And Elaine says well, we'll keep working but it maybe years before we have a paper and we may not be able to publish that anyway. And worse is that I’m looking at my resource. I say I don't have positive results. Omar has published in the Annals. There's a couple of other people who have seemingly good results. I’m going to quit. Our version of what comes out, the version that we see as readers comes out is distorted result of this process.
I tried to think to myself whether in the clinical realm, clinical medicine I try to think of examples in which the subsequent papers to the initial pilot trial, for example the Annals of the New England journal or elsewhere, the subsequent clinical trials were better than the initial trial. I’m sure they exist but they're certainly not plentiful. Almost always the experience of clinicians is as more and more work is done, as sample size increased, as other groups begin the report, the results begin to retreat from that initial spectacular finding. So that is basically the application of the winner's curse. And requires more discussion but that's the simple way and certainly the way I first began to think of it.
As we say in this manuscript, an underlying assumption of this view of the publication process is that it is susceptible to an economics analysis. That is not entirely a -- on firm ground. We don't have a price. Economists as I have learned to my dismay do care about prices. We have values but we don't have good values to place on especially the information we produce. But in a gross way, in a grand way we can sort of have some view from the economic perspective of this market that we're dealing with in information.
Of course as scientists we do like to think of ourselves doing things that are absolutely unique, that don't have to do with the rest of the world at all. We are exploring nature, doing what Newton and Galileo and all the people that we admire do. That is just different from all this exchange of goods and materials that distinguish us, especially here at NIH where we may not have goods and materials to exchange. I think that we are -- we do make not only an assumption that we can be looked at from an economic point of view but also not so terribly different from the rest of the world in terms of way markets may actually operate.
We certainly do know that despite our idealism, what we publish has enormous value. Now, when you think of that value we think in many different ways. It's part of what increases the complexity of this analysis. The value obviously for example for a new drug in terms of a pharmaceutical development or a biotech start-up is enormous. Publication has enormous monetary value. Articles that are published have value to the journals. The journals obviously can profit in many ways from excellent articles or articles that collect a lot of attention in all the ways that commercial enterprises they're required to make money. We also know our publication is absolutely critical to our own careers. I’m going to discuss that at the end. Publication is in fact the single perhaps most important result that we are measured by as scientists or as clinical researchers.
Now, I told you a bit about the winner's curse. I want to tell you how it originated to give you some vivid sense the way it operates and the way it may operate in publication. When it was discovered and named in the oil industry. And the phenomenon that was being -- that was uncovered by oil industry statisticians, economists and others was that in their bidding for drilling rights, all current information, in their bidding for drilling rights for things of Alaska oil and the Gulf Coast, they over the course of years believed that they were in fact overpaying for the drilling rights. Systematically overpaying. Losing money. We think about drilling in a drill, you get oil, you just want it but most holes you put in the ground are not going to yield oil. You don't know that. What you do know is everybody values oil the same. So the companies all are interested in oil as a commodity but they don't know what the actual value is. Each of them have their geologists and their experts in terms of where the oil may be but they're making decisions based on very inadequate information.
The fundamental idea on the basis the oil companies developed, statisticians developed is that the problem was that the best estimate of the value of the particular oil site was likely the average of the bids. So when you won you really lost. You were paying more than all the information that have been gathered by the oil companies the geologists and other experts accumulated. That was reflected in the market value as a market value being the average. So when you pay more than the average you were paying more the market value. You might get lucky and do well occasionally but most of the time these are dead holes with no positive results. Turns out when you hear about the gremlins, Friday evening business news when you hear about the gremlins doing the calculation there's a lot of gremlins on Wall Street trying to figure out or they were trying to figure out how to beat the winner's curse. How to underbid by the winner's curse. In fact, being expert does help in terms of avoiding the winner's curse I’ll describe in a few moments.
How does this apply in terms of the market for scientific information? Most of us as authors don't feel ourselves as really in fact we have valuable information we want to get to market. That market is others who work in our field, it's other scientists who may not work in our field but we want to let them know about our data. It's the lay press that may as in this instance pick up on something we published. Pharmaceutical companies, biotech, the public at large which in our institution is mainly responsible for funding. So we know that getting our information out has enormous value. So we in fact are the -- we are the oil wells in this instance. The bidding is peculiar, this is where the analogy is only roughly approximate. The bidders are the public. It's the people who want to know the scientific information. But their employing loosely speaking, the journals as they're intermediates. I think this has some interesting consequences because this is a little bit like having oil countries contracting out without much down side to their mistake which is is actually part of the company.
We're aware of the winner's curse especially as we get old. It's familiar to senior scientists in this audience that we make jokes about, not the Annals but the New England Journal or Science, maybe they get it right 50% of the time or published in Nature so it's probably going to be wrong. We have these jokes that we don't really -- at least I myself didn't really understand particularly why I had such a cynical view but I think as you'll hear my argument you'll appreciate why that cynicism may have a structural basis.
Now there, is a phenomena that editors and clinicians are worried about for many years which is the problem of publication bias. Publication bias strictly speaking is the favoring of articles with better statistics. So again as an original example of the laud score or P value that maybe attached to a clinical trial; we have recognized especially in clinical research for a long time the fact that journals like papers with definitive answers statistically. This is a more general statement of that based on not just a statistical value but all the other components that go into making a paper attractive to a journal. I’m not going to describe because I don't have the time but Johnny Inedes have published a series of interesting papers which you can argue about his conclusions but one of the few addressed empirically in the clinic and lesser extent in basic science how valid the literature is. How reproducible the literature is and what patterns of publication actual actually are.
I want to mention a couple of references. One is that we think about science, we have great faith. I use that word advisedly, we have great faith that we will self-correct. This system is an ideal system, science self-corrects. Obviously to some extent we can make that argument. We have seen bad things thrown out and good things replace them. As John pointed out, this is not a necessarily a fishing process. It takes years before the negative papers to correct positive papers when they do appear. It's not instantaneous, and lays on the order for the major high impact type publication. Let me order two years. The second is there's often something called John dubbed the proteus effect, hearsay the bouncing -- this is the bouncing back and forth of positive and negative data around narrow statistical significance so 20 or 30 years later we're not sure whether anti-coagulation should be used in oral -- the third, this is relevant to the second speaker, we depend as the Annals does on meta analyses and meta analyses as John argued inflate already inflated results. There's an example recently published New England Journal concerning anti-depressant drugs. This is almost an experiment in which an individual that worked at the FDA and had access to unpublished literature or the negative results from antidepressant trials, combined that information with the published results and many of the apparent beneficial effects of antidepressants disappeared when all the data were available for analysis. Don't go off antidepressants, but it suggested -- that would have been picked up in a meta-analysis of published work was in fact inflated by absence of important negative data.
John and also investigators who use text mining have described cascades of information so that for example Andre at the University of Chicago published a lovely paper showing that in drosophila genetics, not an area particularly subject to hyperbole, once a positive paper appear there's a cascade not to be expected based on modeling or statistics, a cascade of further positive papers confirming this original result that doesn't fit any model other than the fact that once published, once a positive paper is out there's a desire to only publish further confirming papers.
I think as an aside there's an interesting contrast between clinical and basic researchers. Though I have feet in both camps there's a tendency of basic scientist, you have to sit on our NIH Central Tenure Committee, there's a bias on the part of basic scientists that clinical people really done know how to do serious research. But there's more attention to this issue of negative data and the winner's curse in general in the clinical than the basic science realm. There are most reputable journals now do -- are interested in negative data to a greater extent than they have historically. There's a sense we know we have to register negative studies. We have to register our clinical trials. We have to report negative data in some form, we're ethically obligated to do that. The apriori rather than aposteriori are well executed and adequately interpreted as opposed to is the result dramatic enough? That's an important principle in performing and publishing clinical research.
I think that's in contrast to basic science which is my co-author pointed out there's not very much statistics within molecular biology, there's not much in terms of standards for reproducibility. We don't know how valid much of the molecular and biochemical literature is that is actually published. If anything, we celebrate positive results that are unexpected. If you get a positive result in a clinical trial that's not part of your primary end point, good luck trying to publish that. A reputable editor will tell you that you need to do another study looking at that. The problem is looking -- is significance. Something very familiar to people doing clinical research. You can find something significant if you look at sufficient variables.
In contrast, the basic world we know that we read in Cell and Nature and Science and we love to see those unexpectedly surprisingly, those are the things that alert us the paper is going to be exciting and interesting. That's in contrast to basic science in the clinical realm.
Now, we like to think, at least my sense, I’m being a relatively conservative person, we like to think the system we have now, the system we have grown up with is really the right system. I think many, especially people with gray hair like mine, who have published and I have got papers in the New England journals and happy. I’m happy with my career, we like to think the system is best of all worlds, it's evolved organically. Isn't it a good thing? We don’t know. The only thing we need to look at the business section of the newspapers to know that markets make mistakes. Markets require correction. One advantage, one of the few advantages of getting old, you can see the difference between the way science operated when you were younger and the way it operated when you get older. And that's both good and bad. Just different.
So I -- there's a huge difference in the amount of data that are now available to be published in the biomedical world and the number of venues compared to when for example I started in Alan Sheckner’s lab several decades ago. What was it like to do an experiment back in old days? I had to purify and make antibodies to hemoglobin, characterize them with the appropriate assays. And at the end I would feel pleased, six months or a year later to publish my paper in the Journal of Biological Chemistry. That’s not a journal that has a tremendously high impact factor anymore, but nobody knew about impact factors. It was sufficient to get that paper out what would be considered middle level journal. How many data points in that paper? Three or four. Some as aspects of the binding affinity and on off constants. Features of the antibody, three or four data points. Now I see people in my laboratory, we use kits, we'll contract out, do sequencing and we'll generate thousands, thousands of pieces of data. Of course we know that they're not equally important but they're data.
Now we look at output. I think we have had this experience if you run a laboratory: People come from foreign countries and will have maybe two full sentences in English they can speak when they come. One sentence will include publication of a paper and a journal of sufficiently high impact factor. The number of outlets that we have for this enormous volume of data is if anything much more restrictive. As John put it, we have tetra bytes of information generated in laboratories throughout the world. It's not we are more sufficient, we have far more medicine, the doubling of NIH budget, the spread of efficient techniques to place us all over the world, the ability to contract tons of information, very limited places to put it that people read it. As John pointed out, not only are we concerned about impact factors but the journals have arranged themselves in a hierarchy in which most of the data that gets cited for many reasons we can discuss in a few minute, that the journals put themselves in a hierarchy so that the most important journals, distinguished by impact factor are ones that accumulate most of the citations. They're not evenly distributed.
My time is short. I want to finish with two aspects of this that are worth thinking about. One the function of branding, the other is phenomena of herding. These are interesting terms that come from economics. Branding is that we have really I think as a scientific community, this is not an issue having to do with that. This is an issue having to do with evidence, this has to do with scientists. We less at NIH but certainly the university, given over to other parties our branding as scientists. A brief vignette, a young physician scientist comes to visit at NIH, he's looking for a job. He published a paper in Cell. I joked as if he walked around with a halo over his head. It was amazing. I need this space, I need that, the paper in Cell glided him through his NIH interview. We were thinking thousand accumulate the resources until we read the paper. It was good. It wasn't great. We had to look at the substance and say do we want to work on this at the NIH. The initial approach was he must be magic, he's got a paper in Cell. So we have given that over in terms of promotion, funding and so on.
The second, issue of herding. I want to finish with this. This is a very apropos of our current circumstance. Herding means we operate on other peoples’ information. In an ideal market, we are all accumulating information from our various sources. That’s what makes markets powerful – is that everybody in this room has different experiences, different intelligence, different ways of looking at things. When we combine that, we hope that we’re going to come up with the best possible solution as opposed to my telling you what to do. We don’t want “top down.” We want people contributing coming up with the best approach. Herding occurs when we obtain information sequentially and begin to operate, not on our private information, but on public information. And there are excellent economic models that indicate it doesn’t require a lot. It requires Warren Buffet to say “buy” or “sell” and the next person who doesn’t know whether to do one of the other is biased to do what Warren Buffet said.
The phenomena we can see in science – and fortunately, most of the time, we forget the negative data. Let’s look back at Gene Therapy. You wanted to make a career you published in Gene Therapy. Science, Nature, everyone is picking up Gene Therapy. Few people are cured by gene therapy how many decades later. That's a lot of questions but no question that herding was occurring based on a few people getting into the field. Are we going see that with mouse immunology? Cell published an editorial about how the mouse isn't maybe the best model for anybody that that studies human disease. Will the next generation of hematologists look and say do people believe that you can change a stem cell into a heart and it’s occupied 30% of heart muscle as published in the New England Journal? There's this cascade that can occur in terms of operating on inadequate information.
So my time is up, I know there obviously will be questions and attacks but we finish by indicating that -- I think this is -- to the extent I can be sagacious for younger people. Science is hard work. It's really hard work. I think it can be discouraging working in laboratories especially as a young person, first of all you start to worry which field do I go into. Do I have a possibility of publishing a major journal if I work in this interesting but not popular field. Second, why is that other guy getting his paper published in such a famous place, I’m slogging along doing the best I can. It's hard work. We want to make sure people don't have to win an incredibly difficult lottery to feel satisfaction at the end. I think that is what we're faced with, with current publication system. I’m going finish. I don't know if there's a few minutes for talks for discussion.
[Applause]
OGNIBENE: We don't have lot of time but I guess a quick question for Neal.
QUESTION: First of all that was a very provocative talk. This maybe more a point of clarification than question. Part of the point I think you're making is that there just are not enough venues for publishing. You seem to be particularly picking on the better journals and strikes me as someone who at least in clinical reviewing for better journals, it's clear that you have a very different standard. Many of us review quite a wide spectrum of journals and you review it differently for those journals than you do for others involving many to the best if you can just the sort of the thing that make for what generally over the years are larger samples, better statistics now in many genetics papers you require at least one if not several replication samples and so on. So some of what you said seem to be particularly, quote, picking on the elite journals. I rarely take the position of standing up for the elite, that's not my default bias. But seems to me that maybe unfair because I think there is a lot of built-in in all of us who review, know how differently we do review.
YOUNG: So I think that's a very complicated question in many respects. The elite journal, Nature and Science, they pride themselves on their selectivity. They pride themselves on selectivity. It's explicit in Nature and Science that whatever the reviewers, whatever the reviewers comments the editors reserve the right for whatever gets published. So the -- there is an enormous bias to have a very low rate, very hard to get into those journals. I don't know that it's correct that those journals are better reviewed. In fact there's a paper published recently in one of the Cell spin-offs indicating that the rate of retraction is higher in the high impact factor journals. That maybe because those articles have more scrutiny. We don't know the answer to that. But I don't know that there's any objective evidence that the review process of the scientific information we all produce is actually superior in those journals than others. And I think that you -- my experience has been that in hematology I think articles get better reviews in Blood because Blood knows who to go to for reviewers. We see articles appear in the New England Journal of Medicine all the time, we can't understand how the article got in. They don't know who the real experts are. They get the big picture but don't have the details. I think the impression I think -- I don't believe that the articles are better. I think what they are is more selective. So there are lots of biases that have to do with aposteriori considerations which is what I’m talking about. Is this article seem exciting to me, which reasonable if you're running a magazine but not sure it reflects on the quality of science. It's absolutely worth debating but I think it is worth also examining whether that's really a true statement. So I think I move to give our guest some time.
OGNIBENE: Hopefully we'll have some chance for interaction at the end when we finish at 1 o'clock.
Our second speaker Dr. Christine language received M.D. from the State University of New York stony brook. Following a residency in internal medicine, New York Hospital Cornell University. Dr. Laine has an MPH degree with concentration in quantitative methods in clinical epidemiology in Harvard. She's division of internal medicine at Jefferson Medical College in Philadelphia and her research interests include doctor/patient communication and healthcare delivery for chronic conditions. Dr. Laine is secretary of International Journal Editors, member of the ethics committee of the World Association of Medical Journal Editors and a member of the policy committee and vice president of the Council of Science Editors. Now Dr. Laine will speak on the ethical issues in medical journalism from a journal editor's perspective. Christine.
LAINE: Good afternoon. Pleasure to be here. So disclosures. When I made the slide I didn't think I had any but I am the editor of a medical journal, as Dr. Young did I’ll disclose that. I also want to disclose I have published and declined to publish some of Dr. Young's papers. [laughter] I’m not mentioning any off label uses of drugs or devices and the objectives are to define medical challenges in journalism and describe strategies journal editors take to address some of these challenges.
So there's an unfortunate situation that Dr. Young alluded to. The public has lost trust in the medical device industries in medical journals because we publish a lot of the work from those industries and really in biomedical research itself. Why is this happened? There's several factors that contribute to the state of mistrust. First, there's deceptive authorship practices, mishandling of conflicts of interest, delay of suppression of research results, manipulation and even outright fabrication of data, and sensational reporting, particularly of clinical results.
So I’m going to talk about these issues and describe what journals do. These are my children Matthew and Amelia. Matthew is two years older than Amelia. Suppose Amelia had to write paper in school and Matthew had been in the same school, had to write the same paper. Suppose she said Matthew, will you write this for me? And she gave him $10 to write it and he wrote it and then she put her name on his paper. I think everybody including my children would say that's wrong. But we have come to a situation that school age children would clearly say is wrong has become commonplace in clinical publication.
So the fundamentals of authorship are that all persons designated as authors should qualify for authorship. I’ll talk more about that. All those who qualify should be acknowledged and each author should have participated in the work to be willing to take public responsibility for it. These are some types of author contributions that Annals and other clinical jury rooms have started to ask people to disclose. The International Committee of Medical Journal Editors offers these criteria for authorship. I think the committee recognizes it's the decision of the investigators to decide who deserves credit, who should be acknowledged for this work. These are criteria that certainly define people who should be on the byline. Anybody who made a contribution to conception and design of the study to the acquisition of data or analysis and interpretation of data who is also helped to draft the article or revise it critically from important intellectual content, not just grammar. All authors have to approve the final version to be published. Not the final version submitted to the journal but also the final version to be published.
Contributions that -- that journal editors would typically advise people don't alone justify authorship or somebody who is just drafted the manuscript but had no substantive contribution to the work itself, somebody who is just helped acquired funding. Somebody who's only collected data, somebody who only referred patients or study samples to study or somebody whose only role was general provision of the will be or clinical department where the research was conducted. These contributions are important but they probably don't qualify somebody for authorship. They should be acknowledged but probably not warrant a place on the byline.
So we have a situation with ghosts and guest authors. This is a situation like I described with the example of my children. So there's a nobody, a writer, the ghost, who writes an article then somebody, the guest, agrees to put his or her name on the byline. So what's wrong with ghost writing? First, it obscures contributions and credit. You don't really know who did the work. It hides conflicts of interest. If the guest may not have any conflicts of interest but perhaps the writer does. It blurs the lines of accountability. If something about the quality of that science comes into question after publication if the post peer review process, which of those individuals is going to be able to take accountability for the work? So I call this the spooky ghost and unwelcome guest, a cautionary tale.
So several years ago in 2003 Annals published paper comparing Vioxx to an NSAID. The author of this paper signed a statement defining all his contributions and attesting that he met the criteria for authorship. Two years later the New York Times questioned the appropriate reporting of cardiac events in that 2003 paper. At that time the author said he never really saw the data, he only made suggestions on an already written paper that the sponsor had written. And then as a sequel to this ghost story, this year JAMA published sort of an expose about a lot of guest and ghosts related to the work on rofacoxib.
How do we exorcise these ghosts and guests? I think that one thing we see that authors -- that academic industry collaboration is actually probably very good for science but people academic author versus to insist on early and active involvement and really have to resist opportunities to sign off on papers that are already written. Authors have to insist on sufficient access to the data and actually roll up their sleeves and review it so they can be accountable for the validity of the work. And authors can’t attest the papers reflect their interpretation of the data they shouldn't be authors. You need to fully disclose the contributions and the potential conflicts of all who help prepare the article. Ultimately I think the biggest guide for people is to say, would I be willing and able to stand by the content of this paper if someone like New York Times reporter questions the integrity? And if you would be able to stand by the data and explain it then you probably deserve to be an author. If you look like the gentleman on the right of the slide you probably have no business being an author.
The journals have responsibilities for deceptive authorship practices. I think that for a long time journals made it seem that if there was a professional writer on a paper, that was bad. And we weren't going to publish the work. But there are good reasons to have professional writer, it certainly makes the job of the journal easier when the paper is written in good English. So we need to make it clear that medical writers can be legitimate contributors and their roles and affiliations should be acknowledged, not hidden. The other thing journals have to do is when we do detect ghost written manuscripts or guests on paper we should respond in ways that involve not only the sponsors but also the authors and their institutions and not just let it go so people continue to get away with these ethically questionable practices.
Several actions that we take and that are possible as we publish a notice to let our readers know that this actually wasn't written by Dr. Smith, it was written by Ms. Jones who worked for this company, we alert the authors academic institution to suggest that they investigate the matter and we provide specific names if contacted by the popular media or government organizations about who these practices involved.
Another thing deceptive authorship practice is redundant reporting or is a “Salami Science.” It's deceptive, a form of self-plagiarism and makes investigators appear more productive than they are. This reflects a little bit back on what Dr. Young was talking about, it makes less evidence seem like more evidence. If you have a paper that has a positive attractive result and then you publish another slice of that same study you're augmenting the favorable results in the literature. And people don't read things so carefully so it -- two analyses from the same database are often interpreted by various stakeholders as two separate studies. So redundant publication is not good. It's deceptive.
What can journalists do? There's been a lot of hype about plagiarism identification software, Deja Vu is one of them. We have played around with it but it really uncovers more noise than true redone redundancy or plagiarism when you apply clinical research so we haven't found it a very effective mechanism. We can insist authors inform editors of all published and in-process related publications from the same data set. We can insist, authors don't always do that, so we started to routinely conduct literature reviews related to papers under consideration. At least once or twice a week at Annals we identify something that really is redundant and we can stop the review process and decline to publish it at that point. Journals need to resist the urge to publish very thin slices of studies and then also take action when redundancy is identified and let the accountable institutions know about this redundancy.
Now, the second issue that caused the public to lose trust is conflicts of interest. Having a potential conflicts of interest is not a manifestation of improper behavior or scientific misconduct by failure to disclose is because that's deception. I think for this audience I probably don't have to go over this but it's important to note that conflict of interest is relationships on the part of the investigators or their institutions so that they have a vested interest in the outcome of the study. These relationships are one potential source of bias. They can be financial or non-financial but those that involve money are ones the public worries about the most.
Conflict of interest didn't seem a problem in the ‘70s and '80s. This graph was taken from an article, that looked at the article's index in Medline where the main topic was conflict of interest. There were hardly any until the beginning of the '90s which is a time there's more industry/academia collaboration. The numbers skyrocketed. This stops in 2002 but if somebody continues the study through 2009 the red bar would continue to go up. In 1995 a study of about 2000 like science faculty in the top 50 NIH studies revealed that almost 20% of the respondents reported that they had delayed publication of articles for more than six months to serve proprietary needs -- either to protect a scientific lead to slow dissemination of undesired results, to resolve intellectual property or ownership disputes or allow for the opportunity to apply for a patent or other reasons. So conflicts are out there and we need to -- we can't eradicate them but we need to manage them sensibly.
The first thing we have to do is recognize that they exist and they could influence the design, the conduct and reporting of the clinical research and the person with the conflict is least equipped to decide whether or not it's actually having those influences. Then we need collaboration between the public researchers, physicians, academic medical centers, biomedical journals and industry to disclose conflicts. I think that we're moving rapidly towards a time where there will be one big conflict database where everybody who has gotten any money for doing anything in medical science will have the nature of that relationship and the amounts on some website and we can avoid situations where we run into it all at the same time. We'll publish an article, a researcher hasn't identified a conflict. Some reader pulls a different paper from the New England Journal where that same person identified a conflict and there's a lot of confusion about what the truth is.
The next thing that we can do to try to regain some trust is to increase the transparency in medical research. Some mechanisms for this, I’m going to talk about, include clinical trial registration, increasing the rigor of review processes, demanding increased levels of transparency and creating a culture where we foster reproducible research. I’ll talk a bit about what I mean by that concept.
So clinical trial registration. Back in 2004 there was a lot in the lay press about that came out that the work on antidepressants in children and adolescents, a lot of studies that showed worrisome adverse effects on suicide, being among them, had failed to be published. What reached the literature were the positive studies that showed benefit. This is an example of a problem that I’ll call selective publication. Dr. Young talked about this. It's a little bit of the publication bias. Trials with results that didn't favor the sponsor's products were not being submitted for publication. There are scores of examples of this.
Another problem is illustrated by this example. In 2001, Cathy DeAngeles, the editor of JAMA found out -- JAMA published results of a six month study of a popular arthritis drug, the Cox-2 inhibitors again, that showed that the drug caused fewer gastrointestinal problems in comparable medications. During the review process all four reviewers and editors said these are interesting results but six months is a short study for a drug that people need to take for long periods of time as they had -- if they had inflammatory arthritis and things like that. So JAMA asked for -- if they had longer-term results and the investigator said no, we don’t. You only have the six months studies. We didn't plan the study to gather any more data. What happened when somebody fished around with the data that was on file in the FDA, that they did have 12 month results. 12 month results were the primary outcomes signed in the protocol but there was no benefit of the drug at 12 months so they chose to present the six months. This is a problem that I call protocol drift. So the outcomes reported in the paper migrate from what was specified in the initial protocol.
So back in 2004 a group of medical editors decided what could we do to at least try to help this selective publication? And we decided that there had been opportunity to register clinical trials before and there had been some mandates for particular trials that needed to be -- needed to be registered but all in all registration was voluntary and not very many clinical trials were actually registered. And a group of journals decided that we wouldn't publish clinical trial results if they weren't -- if the methods weren't registered before the first patient was enrolled. That had an enormous response with I think greatly increasing the work at clinicaltrials.gov and other registries around the world and now it's accepted and investigators doing the clinical trials need to register them.
The pros of trials registration is first doctors and patients can find trials to enroll in more easily and it limits the ability to suppress unfavorable results because people know the trial exists and approximately when it will end. We can go to a trial registry and look at what the primary outcome was supposed to be and if they're not reporting that primary outcome we can in an informed way ask for reporting of the primary outcome. And then it also allows authors of reviews and meta-analyses and guidelines to more easily locate relevant trials, even if they have not been published.
Now, the problems with potential problems with trial registrations is that investigators worried a lot about intellectual property theft and sponsors worry about divulging the proprietary information. Journals worry if registries expand to include results we would be put out of business. Why would anybody need to publish in a journal? Scientists were worries about the release of non-peer reviewed results because some scientists think the peer review process does something. Now, we have gotten a point where there is, with the FDA Amendments act registration is now required. The potential benefits are it speeds the release of results and eliminates the need for publication to disseminate these results and the disadvantages are it speeds the release of results because some of these results are getting out to the publish before they've been properly vetted. And it virtually eliminates the need for peer review. There's also a no context of commentary if the results are published in a registry. There's not the opportunity ahead of editorials to have letters to the editor and have all of that post publication peer review.
So what are safeguards that as results registration comes into play? Journals have talked about allowing a lag time between trial completion and results posting to enable attempts for peer review and publication. I think we have about a year which isn't a lot of time, to have the registries clearly labeled unpublished studies. We suggested that maybe clinicaltrials.gov -- there needs to be indicator for studies never submitted to a journal, studies that it submitted to X number of journals and never published. And replacing the registry with the peer-reviewed publication. That hasn't happened either but I think what's going to happen -- what's going to happen is we will begin to see circumstances where the results in the registry are different than the results this get published in the New England Journal or Lancet or Annals or JAMA or Blood and we're going to need a mechanism to rectify those discrepancies.
Now, the other problem that has caused people to lose mistrust is manipulation and fabrication of data. And how do journals identify data manipulation and fabrication? It's very hard. If somebody wants -- we weren't there when the research is done, people are very smart, technology is improving to help people create convincing data that appear real. But we do have processes in place. I think this after working in the medical journal for ten years, nothing gets put in print the same way that it came in. And there are numerous examples of things, we obviously make mistakes and publish material that ends up not being true, some of it ends up being ethically suspect and things like that but there are a lot of cases where we do stop things before that happens. And the processes we have in place are very rigorous peer review where we try to match up reviewers with expertise in that particular topic. Formal statistical review journals handle this will in other ways, at Annals we have a team of about eight statisticians who, work for the journal who review things and they unpack the black box. They will ask for protocols, statistical code and even data to facilitate review. Other journals require independent review by somebody other than the sponsor. And we have a very high level of scrutiny for results that are too good to be true. So results that really look astounding or remarkable, we really are very suspicious about. Those will be the cases where we go to the point where we're asking for data and we're reanalyzing it in-house.
There's also software to detect image manipulation. This would have been particularly helpful in some of the stem cell research fraud and some of the software was developed prior to those episodes and journals have published certain types of science, more basic science than the clinical research have started to use this software to try to detect images that have been manipulated.
Talk a little bit about reproducible research, which is a policy that we published an article on in 2007 in Annals and we initiated a little experiment, pretty much in the early stages of this experiment. What I mean by reproducible research is the scientific community arrives at the truth by independently verifying new observations. Again, relevant to Dr. Young's talk, the first study might be very positive but after a while things don't look maybe so good after a while. So that replication process is really important. The highest level of confirmation is when you can replicate the same findings by independent scientists in independent settings but this happens infrequently, particularly in clinical medicine where you're talking about enrolling people in trials and they're adverse events. After the first positive study comes out there are issues about whether it's ethical to continue to enroll people in trials.
The second best thing is reproduction which is using the same data set, can other scientists replicate -- reproduce the results? But in order to do that people need to be willing to share their protocols, their statistical codes and their data. And most people were not willing to do that.
So we started with a very small baby step in Annals about two years ago. We started to require authors to include a statement in their articles about the availability of the protocol, the statistical code and the data. So they don't have to make these things available but they need to publicly state in the work whether or not they would make it available to others who wanted to do the study. We let them put in the condition so they can say the data are available as people buy them for $50,000 or they're available but you have to have one of your initial original investigators collaborate on that project and we're starting to see different permutations of the conditions under which people make available. Still the most common thing is that people are not making the data available but at least we hope that this is a little step towards changing the culture and making it -- have people be proud when they see they'll share these materials and encourage efforts by others to reproduce their results.
The other thing that has caused a loss of public trust is sensational reporting. And journals sometimes themselves are reporting results sensationally. We try not to do that, to, you know, temper the enthusiasm that some investigators have for their own findings but the problem is we publish this stuff and it goes out into the lay press. And the lay press, we have little control over what they say. Usually things that are published as cures or horrible safety concerns, it's more nuanced what the actual study says. So the -- and journals like to be mentioned on the front page of the Washington Post, and the New York Times so journals have to temper their enthusiasm for public recognition and be more responsible in the way that they communicate these results to the reporters and lay media to minimize unwarranted enthusiasm or fear and rash clinical decisions based on the work that we publish.
Some of the things that we have done at Annals is to be sure in our tip sheets and press releases that we include absolute event rates when results are reported as relative risks that we disclose potential conflicts of interest in the materials that go to media so that reporters have an opportunity not to uncover those things after the fact but they know about them when they're working about the results and generally emphasize the translation of research results into practice involves a very delicate balancing of benefits and harms and there's few studies that clinical journals publish, even the ones at the very top of the food chain that should change clinical practice immediately.
So again, just to summarize some of the steps that journals are taking to try to regain public trust is to avoid deceptive authorship practices, more fully disclose conflicts of interest, to register trials and their results to ensure complete and timely reporting, to increasingly improve our peer review process to include formal statistical review to ask for data and protocols and statistical code when it's necessary. To develop a culture that encourages reproducible research and discourage sensational exaggerated reporting to the public. All these are just small steps to try to prevent -- I think this is what much of the public thinks about clinical research, it looks like a game. Thank you.
[Applause]
OGNIBENE: We have some time if people are interested in staying for questions for Dr. Laine or Dr. Young. If not, thank you and they'll be here for a while in the front. Thank you very much. [Applause]
(Music fades in, under VO)
ANNOUNCER: Today we've been pleased to bring you two speakers -- Dr. Neal Young, chief of the Hematology Branch at the National Heart, Lung and Blood Institute at the NIH who spoke on the subject, "The Winner's Curse: The Market for Exchange of Science." He was followed by Dr. Christine Laine, deputy editor of The Annals of Internal Medicine, who spoke about "Ethical Challenges for Medical Journals." Once again, you can see a closed-captioned videocast of this lecture by logging onto http://videocast.nih.gov -- click the "Past Events" link. The NIH CLINICAL CENTER GRAND ROUNDS podcast is a presentation of the NIH Clinical Center, Office of Communications, Patient Recruitment and Public Liaison. For more information about clinical research going on every day at the NIH Clinical Center, log on to http://clinicalcenter.nih.gov. From America’s Clinical Research Hospital, this has been NIH CLINICAL CENTER GRAND ROUNDS. In Bethesda, Maryland, I’m Bill Schmalfeldt at the National Institutes of Health, an agency of the United States Department of Health and Human Services.
