Writing replicable and reproducible methods II: Statistical analyses and data

“Back in my day…” This quintessential phrase parents use to describe their comparatively more analog childhood to their children also pops into my head when I read statistical analyses in historical papers. Take J.R. Brett’s 1964 paper in the Canadian Journal of Fisheries and Aquatic Sciences. The study compared oxygen consumption of salmon swimming in different temperatures of water. The statistical analyses, “correlation tests” and “t-tests”, seemed rudimentary in comparison to today’s digital computing.  

I wondered if, “back in the day,” these tests were calculated by hand. Bruce Patten started working at the Pacific Biological Station, the same research facility Brett worked at, in 1983. Now retired, he suspects the tests were calculated by mechanical or electromechanical calculating machines, as electronic calculators were rare in the 1960s. 

Regardless of the hardware used, Brett’s transparent statistical reporting is what makes repeating the study a possibility. Access to the data would make computational reproduction possibile (i.e., reaching the same results using the same data). Bringing both elements together for a methods section, including before you start writing it, is what ultimately ensures a study can be replicated or reproduced.

How to prepare data and a statistical analyses section for replication and reproduction

1. Cover the basics 

The basics of statistical reporting require attention as they are necessary not only for replication and reproducibility, but also for evidence synthesis.  

“It is incredibly frustrating to have to remove studies [from syntheses] that are otherwise done well because they missed something as fundamental as reporting their sample size,” says Dr. Trina Rytwinski of the Canadian Centre for Evidence-Informed Conservation.  

Fundamentals to report include: sample size (the overall number of samples and the sample size used for statistical analyses), variance, error values and type, equations (e.g., how effect sizes were calculated), statistical software (e.g., version of R and R packages). 

2. Describe data validation, transformation, and processing  

In addition to reporting what validation checks were used on raw data (e.g., linearity), provide the results of the checks. Transforming data? Include what variables were transformed, how, and why. Dealt with outliers or missing data? Include explanations for how these data were handled. If constrained by word limits, you can provide this information as supplemental files. 

3. Be specific about sampling 

State whether samples are considered independent, pseudoreplicated, or nested, and identify the units of replication. State your sampling strategy (e.g., random, opportunistic). If experimental treatments are involved, state whether randomization was used to assign the treatment and, if so, what type. Note any spatial (e.g., transect) or temporal structure such as repeated measures on the same subjects (e.g., longitudinal).  

For example: “To test how water temperature affects fish diversity, we opportunistically collected 20 water samples from 10 rivers in both 2024 and 2025 to extract eDNA; the same rivers were sampled each year (longitudinal design), and river was treated as the unit of replication in a repeated-measures analysis (n = 10).” 

4. Name the statistical test and the data used 

With reproducibility in mind, the more detail the better, even for seemingly obvious things. Pair the name of each statistical test with the data used and, as necessary, which software and version were used for which tests. It’s also helpful to indicate how the test addressed the study’s research question. For a list of essential statistical details to report, read the Canadian Science Publishing Author Guidelines.   

5. Establish data management and sharing plans before you start the research  

Decide early in the research process how data will be stored and shared. If data or code are being shared publicly, visit fairsharing.org to find general and community-specific repositories. Funders and institutions, including Canada’s federal agencies, increasingly require or encourage data management planning. 

 6. Include an informative README file 

Writing early and throughout a study is good practice not only for your methods but also for preparing a README file—the instruction manual researchers will use if they download your data and code to reproduce your work.  

README files include metadata, or “data about the data,” and other information needed for reproduction:  

• Data variables: explanations of column headers, abbreviations (e.g., M for Male and F for Female), and codes for missing data (e.g., NA) used in the .csv file  

• File inventory: description of what each file or folder contains  

• Context: the date range of collection, geographic coordinates of a sampling site 

• Executable workflows: a step-by-step section detailing the software and exact commands a researcher needs to type to re-run your analyses.  

7. Ensure humans (and machines) can find shared data and know how to use it 

If you are sharing your data, consult the FAIR Guiding Principles. To make data findable, it will need a unique and persistent identifier, such as a Digital Object Identifier (DOI).  When you upload your data to a repository like Dryad, it will be assigned a DOI (e.g., a unique string of characters). The DOI helps search engines, and the humans using them, discover your data. To make data reusable, upload documentation alongside the data that clearly explains how it can be reused (e.g., a data usage license indicating restrictions and how to attribute credit).

Supporting reproducibility through open data and open code practices remains a challenge that is both cultural and incentive-based, says Dr. Marija Purgar, President of the Society for Open, Reliable, and Transparent Ecology and Evolutionary Biology (SORTEE).

“Researchers often receive limited recognition for sharing data and code, while facing real or perceived costs and risks, such as being scooped, misinterpreted, or not receiving appropriate credit. These challenges cannot be addressed by researchers alone,” says Purgar, adding that “improving reproducibility is a shared responsibility, and journals, funders, and institutions all have an important role in setting clear expectations, providing training, and rewarding open research practices.”  

Compared to the scientific reporting landscape in J.R. Brett’s day, the considerations around replication and reproduction have grown complex. This post and our companion post provide practical tips to help ready your research for whatever future landscape exists. 

Continue building your knowledge. Read Part I to discover the foundations of replicable and reproducible methods.

Thank you to all community partners, subject-experts, authors, and CSP staff whose knowledge and feedback helped build this blog post, especially Dr. Trina Rytwinski and staff at the Canadian Centre for Evidence-Informed Conservation, Dr. Marija Purgar, Saurabh Tiwari, Dr. Jacob Thundathil, Dr. Chris Rooper, Dr. Becky Furlong, Hilary Belleville, Bruce Patten, and Rebecca Michaels-Walker.