DAVID GREENE, HOST:
There are a lot of exciting ideas in cancer research out there, but they never pan out. And one reason is initial studies don't hold up to scrutiny. Well, a research group is trying to figure out how big a problem this is. And the first results suggest it's just not easy for scientists to reproduce the work of others. Here's NPR's Richard Harris.
RICHARD HARRIS, BYLINE: A few years ago, Brian Nosek stunned his field of psychology when he and his colleagues determined that many of those experiments couldn't be repeated successfully. Next, he turned his attention to laboratory cancer research.
BRIAN NOSEK: Reproducibility is a central feature of how science is supposed to be, and it's not clear to what extent it is happening in practice.
HARRIS: In 2011, scientists at the companies Bayer and Amgen each tried to redo dozens of promising cancer experiments to see if they could get the same results. The vast majority of those attempts failed.
NOSEK: The community responded very strongly to these reports of challenges to reproduce some of these core findings.
HARRIS: But Bayer and Amgen wouldn't say which experiments they examined. So their work raised questions but left no way for scientists to follow up.
NOSEK: And so the reproducibility project in cancer biology was an attempt to sort of advance that discussion with an open project.
HARRIS: Nosek, at the Center for Open Science and the University of Virginia, made sure this project was transparent about how it picked the studies, transparent about their methods and their study plans. They got grants to replicate key experiments from up to 50 high-profile studies in collaboration with a company called Science Exchange. They are now publishing the results of their first five attempts in the journal eLife.
Timothy Errington is a biologist at the Center for Open Science.
TIMOTHY ERRINGTON: Three of the five show very, very striking differences from the original. And then there's two of them that, I think, you'll get a lot of different opinions on whether they, quote, unquote, "replicate" it or not.
HARRIS: Did these initial results surprise you?
ERRINGTON: Yeah. Oh, yeah.
HARRIS: Some of the original labs cooperated while others didn't. In one case, the original scientists went the extra mile to help the follow-up labs reduce potential sources of error.
ERRINGTON: The lab gave us the same drug. This is wonderful - right? - 'cause, like, that could have been a sticking point. Oh, wow, they gave us the same tumor cells that they used.
HARRIS: But the replicating lab didn't end up with the same results. The inevitable question then is whether the original science was wrong or whether the scientists who tried to repeat that work tripped up. In two cases, the original scientists are so confident in their findings that they've been able to draw millions of dollars in investment to start developing new drugs. Sean Morrison, an editor at eLife and a prominent cancer biologist, notes that it can take months or years to perfect the laboratory techniques used in any given experiment.
SEAN MORRISON: And one of the difficulties of the reproducibility project is that they have limited time and resources to spend on any one study. And as a result, they can't go back and do these things over and over again when the first attempts turn out to be uninterpretable.
HARRIS: Errington agrees that their results leave that very important question hanging.
ERRINGTON: And as exciting as that is and important as that is - and hopefully, somebody does follow up on it - we're a bit more curious on - well, what does that look like when we do it across many, many, many studies?
HARRIS: They're looking for patterns across cancer research and also trying to identify common reasons that labs might have trouble reproducing one another's work. Are the directions offered in a paper too sketchy? Or maybe an experiment only works under certain unusual conditions. Morrison at eLife says the entire reproducibility project is itself one big experiment.
MORRISON: I think it's too early for us to know whether this approach is the right approach or the best approach for testing the reproducibility of cancer biology. But it'll be a data point, and it'll start the conversation.
HARRIS: The conversation is important because the vast majority of treatment ideas that come from this basic science fail when they're tried in people. Cathy Tralau-Stewart at UC San Francisco says scientists often don't know why those failures occur.
CATHY TRALAU-STEWART: And so that's why I think these sort of studies are really, really important.
HARRIS: The point isn't who's right and wrong but all about the why.
Richard Harris, NPR News.
(SOUNDBITE OF RUSPO SONG, "FILOMENA")