Reproducibility: Eat, Sleep, Lab, Repeat

For a while now, or for at least my currently short spanning experience of research, reproducibility has been a hot topic in scientific research, and particularly biomedical research. However reproducibility in the lab is increasingly gaining traction in the press, and last week an article reported by BBC News stated that science may be facing a reproducibility crisis. The BBC reported a study highlighting that ‘more than two thirds of researchers have tried and failed to reproduce another scientist’s experiments’. Lets delve a bit deeper into this…

So what is reproducibility? For those of you unfamiliar with this term in the context of lab research, reproducibility is the ability of a study, project, or single experiment to be repeated and duplicated by another person or group, to give the same result or general conclusions.

And why is it important? Reproducibility is extremely important in scientific research methods as it allows results and observations to be confirmed, the conclusions drawn to be more thorough, provides the study with credibility, and can determine what future work or decisions are made. In layman’s terms, it also proves that results or events are not simply ‘single occurrences’, a ‘fluke’ or even ‘beginners luck’.

What does this mean for science? Let’s think about this with an example. Suppose a new drug was being investigated in breast cancer that could have some amazing effects at reducing tumour growth, but may also be linked with some nasty side effects. The team investigating this needs to ensure that their experiments are robust, they have been repeated multiple times with separate samples, and they have been thorough in their methodology and data analysis. They may want to even repeat these experiments using different operators (members of the team) machines, reagents, and even different techniques to ensure that they have drawn the right conclusions. They can then be sure that their drug of interest is worth being explored in greater detail, and to eventually progress to the clinical trial stages.

And if all these steps were not done? A potentially harmful drug, with no proven cancer zapping abilities may progress through to patient administration at trial stage. This not only wastes time, puts patients at risk, but is a huge financial waste too.

What about reproducibility in the news? This is a tough one, because as an advocate of accurate science communication I believe it is important that readers, and the general public, are clued up about scientific research across all fields. I feel it is important that the public also take what they hear or read, and critically analyse it themselves (don’t believe everything you hear), and understand that not all research is 100% trustworthy (improvements may be needed), immediately ground-breaking, and years or research can often return small and minor results which contribute to a bigger picture. But I feel that articles like these can also cause problems in some cases, and I worry that reporting that ‘most research cannot be replicated’ (ok, I may have paraphrased a little) may lead the public to not trust scientists or new research in the future.

Why are we having a potential crisis? I personally think this ‘reproducibility crisis’ is the result of the immense pressure that scientists are facing today –publish or perish! Now too often it seems that researchers value quick churning out of data, publicity, flamboyant findings, or simply fear being overshadowed or scooped, rather than putting emphasis on carrying out solid, dependable research which may take a number of years before reaching a stage appropriate for publishing.

How could we help the scientific community to improve? I’m a firm believer in learning from your mistakes, something that comes into play more frequently as I progress through my PhD. This is what I think could be lacking from many papers or journals. As a student, I would love to be able to read about the mistakes, experiments or sub-projects that were dropped from a paper, so that I could improve my work or experiments in future, and this could help research around the globe, as well as showing that not all research is as perfectly curated as it seems in a paper; the problem is, there is no incentive for researchers to do this.

Another potential area for improvement is the data analysis process, don’t we all love this? This is integral to the integrity of the data and conclusions drawn as the smallest of mistakes here can lead to very different findings. Often it is not the experimental methods that cannot be reproduced, but the data analysis. Again, accurate and honest (and simple) reporting of these processes in papers could really help!

Let’s wrap this up…  From my perspective, I think that reproducibility will always be an issue that scientists have to overcome, but I think that the very nature of the work lends itself to be difficult to reproduce – biological processes especially are highly context dependent or chaotic, and difficult to replicate in the lab because of this. But, researchers have a responsibility to do everything in their power to ensure robust methodologies and analysis. I feel that if a result can be replicated by multiple experiments, or groups, then it validates the outcome as more likely to be correct, but also authenticates the methods used.

I hope this post hasn’t been too lengthy, I felt like having a discussion here and wanted a change from some of my shorter posts. I'd love for the discussion to carry on in the comments.

Thanks for reading!