How the Center for Open Science tackles the reproducibility problem

Taken from Center for Open Science website

Dr. David Mellor is responsible for the community sector in the Center for Open Science (COS). The COS aims to increase openness, integrity, and reproducibility of research. To do so, they provide entirely free and open-source products and services. I talked with him about the COS, open science and the future of publication.

Background: As seen in an earlier post, the credibility of a scientific claim can be enhanced by direct replications. The idea of open science is to increase transparency of science by providing open access towards research articles, data and methodology. This in turn, allows reproduction and critical review of findings. Currently, science often lacks openness, because many published articles, data, materials and codes are not publicly accessible (Munafo et al., 2017).

[] = indicates a comment I made

Q: What does the Open Science Centre do?

We are a non-profit Organisation (NGO) and our overall mission is to increase the transparency and reproducibility of research. We do that through three main activities: meta-science, community and infrastructure. We conduct meta-science research to understand the baseline and identify barriers to reproducibility. So the reproducibility project in Psychology came out 2015 which replicated about 100 Psychology studies. We have an ongoing reproducibility project in Cancer Biology, which includes about 30 studies, and potential others in the future. The purpose of those reproducibility projects is to get a baseline estimate of how reproducible published research is, and to identify barriers to reproducibility.

The community part of our team develops policies and incentive programs and training in order to address the problems that out reproducibility projects identify. Most polices are aligned with the transparency promotion guidelines, which cover journal and founder practices for data sharing, material sharing, reporting standards, and action such as pre-registration and replication standards.

Badges used by COS

So part of our meta-science projects are badging programs  in journals, which encourage open science behaviours. Badging programs lead to a tremendous increase of data-sharing just by allowing researcher to brag a little bit, and show that they are conducting these best practices.

Good evidence.PNG
Example for Effectiveness of badging. Percentage of articles reporting open data by half year and by journal. The darker line indicates Psychological Science, the dotted red line indicates when badges were introduced in Psychological Science. Taken from Kidwell et al. (2016).

We also have the pre-registration challenge, which is an education campaign that encourages people to try out pre-registration themself and initiate it as a habit. The first 1,000 researcher will be given 1,000 dollar prices, and that has been founded by our primary founder the Laura and John Arnold Foundation.

That is our meta-science and community team, most people who work here are developers, and they are building the Open Science Framework (OSF) that enables all the actions that we advocate for.  It is a platform for sharing data, for pre-registration and essentially for connecting different parts of the research work-flow. So that is our structure, and the reason for that three part structure is to identify problems, search for solutions and then build solution.

Q: How are publishing and doing science interrelated?

Traditionally, publication has been a fairly superficial summary of everything that goes on in the research lab. There tends to be not enough information in the published literature to really understand everything that went on. So our vision of what gets shared and disseminated is much, much greater: the underlying data, the underlying code, all the individuals steps, everything that was planned ahead of time, everything that was discovered along the way and the learned lessons during the experimental process. This is a much more difficult and messy story to tell, but it is all important information for understanding the body of evidence that exists for any given claim. So publishing is basically the dissemination of a story and we want to make sure that that dissemination occurs in a very honest way. In order to do that we make sure to reward people for sharing the honest story, because right now people tend to be rewarded for sharing the best possible scenario and the cleanest possible story and that comes to the expense of the more credible, messier reality that went into it. We are building tools and policies that can help to reward that messier, but more accurate story.

Q: If I want to start a research career, I have to get published in the most prestigious journals, how does open science change that incentive structure?

A lot of our work focuses on dealing with and creating policies for decision makers, such as journal editors, publication committees, societies, funding agencies and tenure review committees. Those are decision makers who hold the key rewards, so you get rewarded by getting published, you get rewarded by getting funded, hired or promoted. In all those key areas, there are ways to reward stories that are more accurate and less clean, but it takes a shift in how you conduct peer review and how you fund research.

So the TOP [transparency and openness promotion] guidelines were created to give publishers and funders a way to publish or fund more accurate but less clean stories. The best example of that is the pre-registered report, where peer review occurs before results are known. Usually, the back and forth of the peer review process focuses on improving the proposed study. They ask the questions: does this research question really deserve to get answered, is it important enough? And through that process you usually improve sample size, ensure it is highly powered, and critique the experimental design at a point in time where you can actually change what has been proposed. And put in quality checks to make sure that the work has been done in a rigorous way. Right now, if you send a journal a “messy story” that has a lot of negative results in it, it is very difficult to see whether the study has been done in a rigorous way or whether the study turns out like that because it has been done in a sloppy way. But if you build those pre-specified checks in ahead of time, then that gives a way that circumvents the bias that is impossible to avoid once you see the results.

So you need to make sure that you build that credible story before you see the results, that credible work-flow. So registered reports is really the chemical of the practices that we advocate for, making sure people are rewarded through publication or grants for work that is highly credible but maybe less clean and sexy.

Q: So would you say that promoting open science lays in the hand of those key decision makers?

It takes individual researchers to have these values and to work towards open science. But we can’t ask individual researchers to be more susceptible to open science, to be more open to critique, if everybody is not willing to do that, and if everybody is not getting rewarded for doing that. So we find ourselves in a large collective action problem, where everybody is sort of waiting, “you go first”, “I will be open if you are open”. And in order for that [Open Science] to work the rewards have to be in place in order to make sure that that is not clear suicide, to make sure that is something the individual can do, and gets hired for doing it, even if is not making work that makes news headlines. If it is credible, rigorous, important and interesting work it should get more rewarded.

Q: How does the Open Science Framework create incentives?

The OSF can be used along the whole research process.

The Open Science Framework (OSF) enables all the actions. It enables to make a persistent record of your code, of the analysis plan before you start creating the study and is a place to persistently host the data. It has built mini-incentives to make work more open. So everything is private by default, but once you make it open you get a page on which you see who is looking at your work, who and how many people downloaded your project. Those are rewards and impacts individual researcher care about, because they can be used as evidence for better impacts in a scholarly eco-system.

It has build-in work flows for pre-registration, forms you can fill out before you start collecting data. Anything that you throw into the Open Science Framework, any file turns into a blueprint, so that is a way to share and disseminate information more quickly than a traditional peer-review model allows. So when you see people viewing that page, downloading the preprint or sharing it, that is another way to cite evidence of your impact. So those are the types of rewards that people then can use on their CV, job description, or anywhere else they want to. Open Science Framework itself is not a decision maker, obviously, but it allows you to collect information about your broader impact.

Q: Is it possible to publish on the Open Science Framework (OSF)?

Right now the only thing that is close to publishing on the OSF is a preprint, we are PsyArcXivadding services to that preprint community, so that for example in PsyArcXiv [a pre-print service for Psychology] it will be possible to set policies and  peer-review standards and moderate what goes into it. So right now you can post anything and it shows up as a preprint and the unbranded OSF preprint server will continue to be the “wild-west” , anybody can post anything they want. But the individual community might want, for example, the PsyArcXiv to put more checks, moderation and more peer-review into that pipeline, before something gets surfaced in their preprint server. For example, the SocArXiv [a pre-print service for Sociology] might want a different set of policies for what gets discovered and surfaced in their preprint server. So we expect to see a diverse range of opinions and practices in all those different services. Some of them will rival and will look pretty much like traditional publications, some of them will continue to be very rough, not being peer-reviewed much. But we embrace that diversity because we know that there will then be competition to see what the best model is. And we expect eventually it will become more evident what is the best model and that competition will have allowed individual researcher to evaluate and go to the services that they think offers the best experience. You can’t determine what the best model is until you have a lot of different players, making a lot of different decisions. So our vision is to enable a place for that to occur.

Q: A common criticism is that Open Science promotes low-quality Science. For example, you do not peer-review preprints.

So the question of how rigorous or how much quality any given research project has is a really hard question to answer. Some would say the peer-review system isn’t able to evaluate that, or what it evaluates isn’t correlated with rigor. On the other hand, and this is kind of prevalent opinion, peer-review isn’t great but it is the best system we have to evaluate quality and rigor.

So the key with dissemination of information is really transparency in terms of show-casing and making very clear how a given study was evaluated. What credibility steps it has passed, and how widely that information has been disseminated. So a pre-print that has not been peer-reviewed by anybody might be very rigorous or might be crap, and until I take a very close look at it, it is hard to make that assessment. So the more information behind that pre-print the more data, code, methods and reporting of exactly what was conducted, the more detail behind that, the more I am able to evaluate its credibility.

Assuming I don’t have time to do that for any given preprint, it is only possible if that information exist and lot of time all this information is lost after a year after the publication. So it has to be preserved and it has to be accessible freely and openly for any given person. And you have to have either the time to look yourself or you have to have the trust that comes from seeing who else evaluated it. So if you see that two or three respected peers, whatever that means, evaluated and gave it some credibility than you can take it a little more at face-value. But if you later want to go into more depth with any given assertion, as long as that data, and all those materials are available, you known that you will be able to do the evaluation by yourself if you need to. It comes down to trust but verify. Give some credibility by working through some transparent mechanism, but then have the ability to verify everything that went into that work.

Q: Would you say that Open Science could replace traditional publication systems?

I don’t know. I think an increasingly diverse set of standards from what gets put out there will continue, I think a lot of work will get surfaced in a preprint and won’t get submitted for publication. I hope and I think the threshold and standard for what gets published will increase so maybe more preprint, fewer formal publications, but the standards and expectations of these final publications might be higher. And I would give more weight to the difference between exploratory and confirmatory studies, so right now I would argue and the evidence from reproducibility projects shows that a lot of the published research can’t be replicated. I would like to see a future where a lot more of these ideas are presented in a more honest fashion, as exploratory findings that deserve follow-up, and the threshold for publication really gets to the registered-reports format, where it is only published if it is much more credible and has gone through a much more rigorous peer-review system than what currently happens. Our ideal vision of the future is that more ideas get out more quickly through things like preprints, and when it gets formally published it’s level of trust is high.

Q: Is Open Science practice currently considered in academic hiring and performance evaluation?

There are several universities and departments who are putting these principles in their job calls, so a typical job calls will say “We are looking for a Psychologist with a training and experience in these areas, and we value x,y and z”. And those value statements in the job calls typically haven’t, up until recently, included instances of transparent research practices. We are seeing and collecting more and more snippets of open science values in job calls. What it really takes is making sure that that then gets internalized within the individual and the hire committee really have to internalize the value that is on a job call, because they could put that into a job call but then ignore it. So it is harder to measure, to verify that it [Open Science Values] is being considered in job evaluations. But it is a problem that requires different ways of tracking or evaluating. So there is a meeting coming up here in Virginia in about two months with the Society for the Improvement of Psychological Science, and hiring and promotion is one of the working groups in that meeting. We are hoping to collect job calls with open science practices, and evaluate candidates based on these transparent research practices just to make sure that it is done in a fairly rigorous, unbiased way.

[I asked the following question in a subsequent e-mail]

Q: A critical point is that direct replication studies are not funded, nor valued in the scientific community. Is, and how is, the Open Science Center working on creating incentives for researcher to conduct direct replications?

You are absolutely right, that replications are both critical for science but worthless for scientists. Not only are they viewed as unoriginal, but the outcomes are always problematic: successful replications are deemed most boring of all, confirming what we already know, which of course is silly, because without replication we don’t have much evidence for what we though we knew, and failed replications are subject to post-hoc methodological scrutiny and questioning the researcher’s competence. We advocate for policies to both funders and to publishers to make replication studies a valued contribution to the work that they support. The TOP Guidelines cover replication policies with 3 different levels of possible policy: (1) the organization can encourage replication studies, (2) the organization can accept calls for replication studies and review them blinded to results, or (3) the organization can conduct peer review of the replication studies before being conducted. On a related note, we definitely have our work cut out for us, only about 3% of psych journals state that they accept replications and I only know of one funder who does so.

Thank you very much!



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s