Diamonds in the rough: Publishing your negative and duplicated results

During my Master’s degree, I spent nearly the whole first year testing a new experimental set-up for growing sterile fungal cultures. After an enormous amount of work, the new apparatus we had dreamed up—which seemed brilliant on paper—didn’t work. Hundreds of hours of effort were scrapped, and the project shifted focus. We did not put an article out into the scientific world detailing what we did and why exactly it didn’t work as expected. Had we, it may have given the next ambitious young mycologist the information they needed to create an innovative new way of studying these organisms.

Every time we test a hypothesis, we know that there’s a chance the result will not be as expected—there will be no effect observed, or the hypothesis will not be supported in some other way. The experiment may also produce marginally significant results, suggesting an effect, but not demonstrating it unequivocally. Collectively, these are referred to as NNI results, standing for negative, null, or inconclusive.

The usual response is to get up, dust ourselves off, toss the NNI results, and start over with a different hypothesis or methodology. NNI results, though informative, will rarely see the light of day in publication. There are various reasons for this: there aren’t a huge number of journals that review and publish negative results, some researchers don’t consider the time it takes to write up and submit negative results to be worth an article that may not get published, and some consider that publicizing negative results amounts to admitting failure and isn’t good for their career. But how many others will end up testing that same hypothesis, not knowing it had already been tested and disproven?

A scientific culture in which negative results aren’t valued not only means there is the misperception of wasted effort but also a lack of encouragement for corroboration of results through formal reproduction. Knowing what didn’t work as expected, or what worked robustly enough to be readily reproducible, is good for science. The importance of negative results was even enshrined last year in the new European Code of Conduct for Research Integrity, put out by a network of high profile academic organizations known as All European Academies.

The code of conduct urges researchers and research publications to treat negative experimental results as valuable and equally worthy of publication. What’s more, the European College of Neuropsychopharmacology and the Organization for Human Brain Mapping have begun offering prizes for negative results and well-executed replication studies, respectively.

A related flavour of academic frustration is getting “scooped.” This is when another research group is working on the same set of experiments or with the same hypothesis that you are, but beats you to the punch and publishes first. It’s heartbreaking to feel that years of hard work were for naught because someone else gets their results out there while your work is still underway.

Again, these results are mourned and then sometimes scrapped, never seeing the light of day. This is not productive, given that these duplicated results have the potential to corroborate the other group’s finding, an important consideration given the current crisis of reproducibility that scientific research finds itself in.

In fact, both duplicated and negative results add to the body of scientific knowledge and should be made available to the research community whenever possible. With that in mind, here is a brief rundown of options for publishing or disseminating your NNI and scooped results.

Peer-Reviewed Journals

Several journals have been launched in the past decade with the goal of providing a venue dedicated to publishing NNI results. These include Negative Results, Data Journal, and The Journal of Negative Results in Biomedicine. All have since ceased publication. According to the latter journal’s website, its purpose had been to provide a platform for results that would otherwise have gone unpublished, a goal it considers met now that a number of other (non-dedicated) journals accept this type of results.

Indeed, dedicated NNI journals may have been unsustainable once more broad spectrum, multidisciplinary journals opened up to publishing these findings. The following is a non-exhaustive list of scholarly journals—all open access—that currently consider negative or duplicated results.

• FACETS is Canadian Science Publishing’s own multidisciplinary journal and will consider negative results and replication studies for work with demonstrated value to their fields.
• PLOS ONE maintains a collection called Missing Pieces that showcases work that was inconclusive, produced negative results, or failed to replicate the findings in other published work. In general, this journal focusses on sound science above impact (in the sense of highly novel research that attracts a lot of attention), making it a good option for negative results.
• PeerJ is another multidisciplinary journal that will consider NNI results. The journal focuses on sound methodology over novelty and also maintains its own preprint server, described below.
• PLOS Biology recently formalized their policy of considering duplicated results for publication. Researchers may now submit duplicated findings within six months of being scooped and they will be reviewed for publication.
• F1000 Research is a multidisciplinary open-access journal that’s not peer reviewed in the traditional sense, but undergoes an informal, open commentary and review by readers. Although a number of potential flaws in this format have been pointed out, it is currently indexed on PubMed.

Other Venues

Another option for disseminating NNI results is uploading to a preprint server or other public database. Although these do not constitute peer-reviewed publications, some are citable and the format allows for open commentary and informal review from others in the field. (See here for a discussion of some of the yet-to-be-resolved issues around preprint servers.)

• PeerJ and bioRxiv are two of the major players in biology preprints, although there are many, and the number is growing. ArXiv, has been the name of the game in physical sciences preprints for many years.
• Figshare is an online data repository where researchers may freely post data, code, figures, and images, in practically any file format. It serves as the supplementary materials repository for some publications, but any researcher can store data there. This may be a good option for those who don’t want to write up their negative results, but still wish to share the data that produced them.
• Organizations such as the Centre for Open Science are encouraging researchers to pre-register their research initiatives, with the goal of ensuring that all outcomes, positive or negative, are then made public.
• At least one researcher (in the humanities in this case) has taken to self-publishing her research in the form of a website after being unable to place the controversial research in traditional journals, a strategy that she discusses here.

The current movement toward open-access publishing and sharing of data via online repositories is improving the visibility of NNI results. Disseminating such useful findings can be economically difficult with the closed-access model of scientific publishing, because they are inherently less novel than positive results and less likely to be covered in readily accessible media. Open-access author fees, on the other hand, can still present a barrier or disincentive for some.

This is why there is a cultural shift that needs to happen as well. Researchers, as well as society as a whole in their capacity as funders and users of scientific knowledge, must come to see NNI results as valuable contributions to the body of science, rather than evidence of failure or a waste of time and money.