A conversation with Chris Chambers about the value of registered reports

TitlePage
The basic idea of study pre-registration. Taken from Open Science Center.

Dr. Chris Chambers is a professor of Cognitive Neuroscience at the School of Psychology in Cardiff University, and one of the leading figures in the registered reports (RR) initiative. I asked him how he thinks RR’s can tackle the reproducibility problem, how he responds to critique towards RR’s and future developments.

Background: The basic idea of registered reports is that the editorial decision to publish a scientific article is not based on the study’s outcome. The journal’s peer-review process takes place before the experiment is even conducted. This also means that researcher decide upon hypothesis, experimental procedures and statistical analysis before they collect data. This idea shifts the focus of the editorial decision away from the results and towards the methodological rigor and value of a research idea. If research question and methodological are considered sound, authors are offered an “in-principle acceptance” of their paper. This means that their study will get published regardless of the obtained result, as long as the authors adhered to the approved methods (Munafò et al., 2017).

[] = signals a comment I made

Q: You are one of the main supporters of registered reports (RRs). What is your motivation for that?

Well, I think primarily it is to try to make science more transparent and reproducible. So registered reports aim to eliminate two big problems in science. The first is a replication bias, where journals selectively report positive results or results that tell a story that is particularly clear, striking or beautiful. The second problems is that that motivates various forms of research bias like p- hacking or HARKing which is hypothesizing after the results are known where the researcher, even unconsciously, changes their hypothesis to fit unexpected results and then presents that hypothesis in the introduction as if they were predicted from the beginning. I think biases combine in quite prodigious ways, to undermine reproducibility of the field, so when we then try to do large scale replications, as we have been doing in psychology for some years, we find that very few studies actually replicate successfully.

Picture1
Taken from Dr. Chamber’s Registered Reports presentation.

So the aim of registered reports is to try to tackle that problem at a ground level, at source. So if you can create a publishing format, where the decision to publish isn’t based on the results, but is based on theoretical significance and methodological rigor, then a lot of these biases are eliminated. Publication bias is eliminated because if you decide what to publish before results exist, then it is not possible for the results to influence your decision, and also a lot of these other biases are eliminate or at least dramatically reduced. So if you have to specify your predictions and your analysis plans in the beginning, then it’s much harder to, for example, p-hack or to change your hypotheses, these behaviors are basically impossible in this framework. So the overarching motivation here is to create something that does not yet exists in the social and life science, which is a format of publication where the main currency is the value and quality of the work and not the outcome of the work.

Q: How much is it the responsibility of scientist to change the flawed publication system?

I think only scientist can change the system. I mean, I am a scientist and a lot of the people that are working on registered reports and related initiatives are scientist, so we talk about science being self-correcting. And this is self-correction in action. This is the scientific community saying we have a problem with the reliability of the research in our area, and we have a problem with the transparency of the research in that area. So we are trying to create solutions. Rather than just kind of picking over the ashes of the problems, the crisis, as it is being referred to. The scientific community is coming together to find answers. I do think this is a responsibility and I think had even a small percentage of scientist focused on actual improving the scientific process in the last 50 years rather than just cranking the handle to support careers, grants, publications and everything else. If we just spend small percentage of our time just maintain the house around us, we wouldn’t be in the situation right now.

Q:  Do you think that Metascience is not sufficiently taught in PhD programs. And do you think that this maybe enhances the problem? 

metaScience
Do PhD students learn enough about meta-science? By Chris Devers.

Indeed, and I think it goes deeper than that. It is actually about the philosophy of science. How many students can really explain the deductive scientific method? How many students really understand what a p-value is, and the context of that framework, what it means and what it doesn’t mean. What it can be used for and what it should not be used for. And because a lot of student lack that fundamental education, when it comes to actually doing research they simply repeat the mistakes that their seniors are making. So you end up with a situation where we are teaching bad incentives, and the incentive structure around us dictates everything. People will do what is rewarded and then avoid what is punished essentially. At the moment the big problem is that because of the lack of education, and the lack of motivation amongst the senior scientific community to deal with this problem, the incentives which motivate and drive me as an individual scientist are very different from what would be best for the community. For me as an individual the most important thing is to produce an awful lot of high-impact papers and attract a lot of grant money. For the scientific community, the most important thing is that I do important research that is published regardless of the outcome. Which is literature that is unbiased as possible which contributes to a larger evidence base. That this community incentive is really different to the individual incentive and when you put those two things in opposition obviously individuals do what it is in their self-interest. This is the tragedy of the commons essentially.

Q: How much of this incentive structure is really changed by registered reports? How much is not solved by it?

I think it is part of the solution, it is not the only solution. For one thing, not all research can or should be preregistered in the first place. So it tackles a problem within a certain sphere of science, which is a big sphere, but only a part of the problem and therefore a part of the solution. That means, some research is purely exploratory and observational [which do not need to be pre-registered], or some research is hypothesis-driven, but can’t be pre-registered because of logistical constraint. So there are always these other cases, where it won’t necessarily provide a solution. But in many cases it does, just pick up a copy of a Psychology journal or Cognitive Neuroscience journal, you will find that every single papers reports hypotheses, and most of the time the hypotheses are supported. Almost all of those papers could be pre-registered and almost all of them could be suitable for registered reports. That doesn’t mean that this should be the only way of publishing, but it should be an option to publish in that format in every journal. As for other solutions, pre-registration is just part of it but there are other important solutions like data sharing. Making sure that all the materials, the digital materials and all of the data and code is made as transparent as possible, regardless of what kind of science we are doing. Within the constraints of ethics and law we should make our research as transparent and accessible as possible.

Q: To what extent should the methodology, data analysis and outcome prediction be pre-registered?

It should be stated in enough detail that another expert in the field could repeat that experiment or that analysis without needing to contact the author to find out what they really did. If you look at method sections at the moment in papers, the general description of analysis in prestigious papers are not reproducible usually, and that creates problems when people try to do replication studies. Because to do replications you either have to guess or you have to contact the original author, if they still alive or interested in talking to you or not. If you run the study otherwise and you have to guess and you get different results to them, the original author can always turn around and say “You didn’t do that, you didn’t do this”. So it creates this, this kind of poor process really. It’s a weak scientific process when you have to guess what previous researchers did based on their papers, all that information should be in the paper, and the more detail the better. For example, some journals launching registered reports now, ask authors to conduct simulation, so when they generate their analysis plan, they ask them to generate simulated data, figures and everything to see how it looks. And then all you have to do is run your experiment and replace your simulated data with your real data, and there is your output. I think the more information you can present the better.

Q: Does study pre-registration really hinder people to fabricate their data?

No it doesn’t. It’s really important to emphasize that it is not an anti-fraud device. If somebody wants to fake it, they will always fake. You can find a way, you can beat any system. Science is basically an honesty system. Although you could well argue that, as the emphasis on replication increases, it becomes harder to get away with fraud. If in order for a discovery to be announced, in any significant way, an independent group has to replicate that result, that would eliminate most fraud over night. This is why you don’t see any fraud in disciplines like physics for example, they discover a particle in one accelerator, they go and replicate it in a completely different one. It impossible for fraud to get hold in that community. Because replication is such a big part of what they do. Now in our area, where replication is not a big part of what we do, it is relatively easy for somebody to go away with making up a data-set, as for instance Diederik Stapel did many times. And then to present that and get a career out of it. And there is probably a lot of people like that in science who got away with that for a long time.

Pre-registration doesn’t eliminate the risk of fraud, but I suspect it changes the incentives to make fraud less attractive. So fraud comes in various shapes from outright fabrication, which is kind of lost-cause behavior, to people massaging data in certain ways and not declaring it. The nice thing about the registered report format is, because it takes away the risk of your paper being rejected on the basis of the results, there is less incentive for an author to massage the data in the first place. They can simply do science. So maybe it will stop people going down the road of fraud. Because people always start down that road you know. They start by changing one number here and one number there. Maybe if you create an environment where there is less pressure to get great results, we would end up reducing fraud from that point of view. Of course it won’t stop the determined fraud stuff. But it raises the bar, it makes it harder to do, because if you go through the review process before you do your experiments, there is simply more transparency. The more transparency there is, the more difficult fraud is to get away with. So what I think what will happen is that people who were determined to commit fraud, will be less likely to do that through some kind of pre-registered format. They will just send their papers to journals with standard formats, because there is less scrutiny.

Q: How much do registered reports take into account that science is an organic process? For example, if a pre-study shows that one’s methodology is not optimal and one wants to change it.

That is fine. There is no problem doing that, you can simply run an experiment and adjust and run again, and most registered report formats include an incremental registration option, where you can simply add experiments in sequences, where you can adjust as you go along. There have been very impressive proposals, one by Rob Leech and colleagues, that integrates Bayesian Optimization into the pre-registration process particularly in brain imaging research, where you could actually use a kind of real time data analysis, build in a kind of online adjustment to your protocol, which is still pre-registered. There are interesting kind of models developing around that, but fundamentally pre-registration is not about stopping people from being flexible in what they do, it’s about being transparent about that flexibility. So there are legitimate reasons why a researcher would deviate from a protocol. That is absolutely fine, the main thing is that it is transparent.

Q: Do you think open science and pre-registered reports could be combined? For example, one’s methodology could be already made public in the review stage?

Certainly, there is nothing stopping researcher from posting their protocols as pre-prints or publishing them in a journal like BMC protocols, or various other protocol journals. And indeed Royal Society Open Science had it very first registered report. The authors submitted the protocols to the journal and they also posted it on the open science framework, so that would enable other to see it and if necessary have a discussion about it. A practice that doesn’t happen because academics don’t really have a lot of time to read other people’s protocols, but they do have time to review them. So the registered reports model works in that sense. I can certainly see a future where this process is completely open. So protocols are accepted and we can track the entire revolution of the process, from the very first version to the accepted protocol and the final paper. And that is what eLife is doing, so if you look at the cancer reproducibility initiative on eLife. They publish all the stage of protocols and then they publish the replication studies next to them. So you can see the protocol and then you can see the outcome. Typically with a registered report that doesn’t usually happen, usually its only published at the end, as one document.

Q: How popular are registered reports at the moment?

It varies from journal to journal. And varies from one discipline to the next, we have got 51 journals today and there is more in the pipeline. It has been going on now for 4 years, we have got a steady uptake in journals, there are around 100 registered reports that have been published in various forms, across different journals. I am very pleased with it, it is always going to be a gradual uptake, because it’s a very different way of publishing, it is a very different way of thinking about the scientific process. Pre-registrations require a different mind-set for scientists in terms of planning. Rather than rushing into an experiment and collect data as quickly as possible, and spending ages on the analysis, because you are going to figure out what you are going to do in the first place. And try to sort of reverse engineer your experiment, which is typically what happen to a lot of research in Psychology and Cognitive Neuroscience. Instead of doing that, you invest that time into the front-end, into verification, building you hypotheses, and your analysis plan at that point. And then the analysis is relatively quick and easy usually. So there is that different mind-set, because of that change in work practice, for some time, it will be a gradual revolution. It is not that everybody suddenly starts submitting registered reports.

Q: So one of the main factors that hinders the adoption of a news system is the research culture? That people are not used to it?

Academia is quite conservative, things are done in a certain way. Not necessarily because that is the most sensible way, but because it’s the way things are done. At least, scientific method or academic practice fall into that category. There are a lot of conventions, which actually, when you look at them, don’t make a lot of sense. Peer-review itself is something that is loaded and celebrated so much, but actually it is kind of nonsense to think that three random reviewers on a paper could provide you a reliable estimate of the work. I mean, peer review itself has problems, it’s better than the alternatives is what some people argue, but still there are lot of convention and there is not a lot of evidence on the line. So whenever people come up with a new idea you pushing against a kind of status quo which is very heavy. So any kind of significant developments are worth being happy about. In the context of registered reports, we have seen some great developments. We have seen it taken up by the Nature Publishing Group, which is really significant, because it means that this is not a format that people see as being a kind of gutter for low impact studies. This is a place for publishing research where the question is so important and the methods are so rigorous that the results don’t matter. The results will simply be the results. And that is something that is attractive to high-impact journals. Just a couple of days ago BMC biology launched them as well, which is the first time a general biology journal has taken on this format indefinitely and across such a broad range of areas in such a prominent journal. So, really it is quite exciting times watching all these different strands develop, and we will see to what extent the format has traction in all those different areas in the years to come.

Q: That leads me to my final question. What does the future bring for registered reports?

It is proving a different way to do science. It’s going to take time. One of the exciting developments we are seeing is the so-called “pre-registered report funding model”, where only with a registered report, the protocol is reviewed by the journal after the authors have funding and after they have their ethics approval. You can actually take that process and conduct it one step earlier, so you can have the protocol, or a series of protocols reviewed by the funding agency, the ethics committee and the journal all at the same time. Therefore, on the day you get your funding, you also get your papers accepted and your ethics approved, it all comes at once. So we have got the first one of these registered report funding models being approved and under the way. Marcus Munafò from Bristol is leading that. Between the journal Nicotine and Tobacco research and the fund cancer research UK, they formed a partnership and there are a many more coming. So you can imagine when those funds and models start to generate, then we are changing the incentive structure. So now, as a researcher, I could pursue grants from registered reports and then, as more journals start offering them, I could start publishing registered reports in more places. I think it is not too crazy to think that researchers in the future could build careers that are strongly based around pre-registration, around this kind of transparent, rigorous approach. We just don’t know, we have to wait and see how it evolves how the format changes in the years to come. What kind of papers they are going to produce. How impactful, how important will these papers be. We don’t know yet, it is all kind of running an experiment.

Thank you very much!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s