A review with a difference: Tips for peer reviewing data papers

Introduction. Methods. Results. Discussion.

The typical structure of an original research paper. Known as the IMRaD format, these papers start with an Introduction section, explain the Methods, present the Results, and finish with a Discussion.

Chances are if you’ve peer-reviewed a paper, it followed IMRaD. You assessed the paper, making sure the methods and study design are appropriate, that the analysis is sound, and that the conclusions are sound. But not all papers follow IMRaD.

We’d like to introduce you to data papers.

Data papers are scientific publications that focus on describing one or more datasets.  They highlight the data and make it easily accessible to the wider scientific community and other stakeholders. They do not include any data analysis or scientific investigation.

And they do not follow IMRaD.

How do you review a data paper?

Like other works submitted to journals, data papers go through peer review. Because they’re not traditional IMRaD papers, reviewing them comes with different responsibilities.

If a request to review a data paper arrives in your inbox, here are five things to consider to assess the paper.

1. Refer to the journal’s guidelines on how data papers are structured

When Dr. Ellen Macdonald, professor of Forest Ecology and Chair of the Department of Renewable Resources at the University of Alberta and co-Editor-in-Chief for the Canadian Journal of Forest Research, submitted a data paper to a journal, she found the review process “a pretty positive experience.”

There was, however, one issue that cropped up. “The only slightly annoying thing was one reviewer suggesting a major reorganisation of the paper that conflicted with the structure and organisation prescribed by the journal,” Macdonald explains. Since data papers are less common than than original research papers, misunderstandings are only to be expected.

Journals may differ in how they want data papers to be structured. As the peer reviewer, referring to those author guidelines provides details on how the data paper is structured and sections to look for when reviewing.

2. If you’re unsure, reach out to the journal editor

As a general rule, “data papers cannot be reviewed with the same criteria as research articles since they have a different structure and fulfill different purposes,” says Dr. Sam Bashevkin, Senior Environmental Scientist at the Delta Stewardship Council.

For the most part, data paper reviewers are assessing data papers for the very first time.  “The major challenge was knowing how to judge [the paper],” Macdonald reflects on her first experience as a data paper reviewer.

For novice data paper reviewers in particular, specific guidance can help ensure that the review is fair, objective, and on point. “If the journal does not have different guidance for data papers than for research articles, ask the editor for this guidance,” says Bashevkin.

3. Ensure that the key components of a data paper are there

Although data papers do not follow the standard IMRaD format, they should contain several key elements.

“The basic objectives for data papers are to describe the data set such that all users understand how it was produced and used, including a summary of the metadata (details on measurements, how errors and missing data were handled, etc.), and ensure it is freely available for use in a registered repository,“ explains Dr. Greg Henry, professor of Arctic tundra ecosystems, at University of British Columbia, and consulting and founding editor of Arctic Science.

While data papers do not have to be created for a specific purpose, it is crucial that they “also show the potential utility of the data set for other types of questions,” says Henry.

There are several ways authors can signal the data’s usefulness. One of the most powerful ways, Henry says, is to use examples. “[This means] citing papers that have used all or parts of the dataset,” he explains.

4. Be prepared to look carefully at the data

Data is the backbone of a data paper. For Macdonald, a thorough, careful peer review means going beyond the actual data itself. “If you review [a data paper] carefully—including actually looking at the data files—it can take a lot of time. There is a lot of detail to go through,” she explains.

The datasets described by data papers can vary considerably in length. Arctic Science’s inaugural data paper describes a dataset on tundra phenology spanning two decades and containing over 150,000 observations.

Observations in the tundra phenology dataset taken at study areas with (A) <6000 and (B) >6000 observations each | https://doi.org/10.1139/as-2020-0041 

Peer reviewers will be pleased to know that they are not expected to assess every observation but rather the overall quality of the dataset (e.g., Is the data organized consistently? Are the appropriate standards and notation used?).

One of the challenges reviewers face is deciding if the data described is sufficient for a data paper. This question goes beyond the quality of the observations collected in the database.

“It’s also hard to know how much data constitutes a data paper. Is publishing a single data point a good idea? Probably not. But is five years of data enough? Ten years? There is no easy answer,” says Dr. Heather Lynch, professor of ecology and evolution at Stony Brook University.

5. Check that standards are met, but allow for some flexibility

Good data papers have a strong metadata component—a description of the data or datasets, the period they cover and their spatial extent, and data collection protocols.

Metadata standards are important, but not to the extent, cautions Lynch, that it stifles the datasets. “I would ask that reviewers not impose a one-size-fits-all set of requirements on metadata. As long as the description of the data and its format is standard for that community, there is no need to ask data paper authors to adopt some cumbersome global standard.”

Giving data papers and their associated datasets room to breathe can mean including “messy descriptions” that still provide valuable information for new users of the data set. “In my own field, the written narrative surrounding a data point is the most helpful part,” explains Lynch, who studies changes in the abundance and distribution of Antarctic seabirds.

For Lynch, “messy descriptions” may look like: There were some number of penguins around the giant rock at the southern tip of the island, though their species was not identified. “[N]ot everything fits neatly into a spreadsheet,” says Lynch.