Proposed Replication Guidelines

Overview

These guidelies provide a description of the types of information that authors should record in their report of a replication. These guidelines have been excerpted from the following paper:

Carver, J., "Towards Reporting Guidelines for Experimental Replications: A Proposal." Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) [Held during ICSE 2010]. May 4, 2010. Cape Town, South Africa.

Information to Report

Based on the review of published replications and the understanding of the goals of an experimental replication, I propose that following items should be included in any report describing an experimental replication.

Information about the Original Study

To help the reader understand the replication, a replication paper needs to discuss some information about the original study. Authors should provide enough information about the original study to allow the reader to properly interpret and understand the replication without providing so much detail that the reader is distracted from the main goal of the replication paper. A replication paper report should provide the following information about the original study (at a minimum):

1. Research Questions - a description of the research question(s) that were the basis for the study design.
2. Participants - the number of participants and any relevant characteristics of those participants.
3. Design - a graphical (or textual) description of the study design.
4. Artifacts - a description of and/or links to the artifacts used in the study.
5. Context Variables - any important context variables that affected the design of the study or the interpretation of the results.
6. Summary of Results - a brief overview of the major findings, espeically those that relate to the replication.

Information about the Replication

As with any experiment, the basic information about the study should be reported. Authors should follow one of the published guidelines to gather this information. This section focuses on the specific information that needs to be reported about a replication. A replication report should contain the following information (at a minimum):

1. Motivation for Conducting the Replication - a description of why the replication was conducted (e.g. to validate results, to broaden the results by changing the participants or the artifacts, etc...
2. Level of interaction with the original experimenters -
if the replication is external (i.e. the original researchers are not involved in the replication), the level of interaction between the replicating researchers and the original researchers should be reported. The level of interaction could range from: none (i.e. simply reading the paper to gather informatino) to much (i.e. the original experimenters acted as consultants in some way). If a published lab package is used, then its use should be described. These guidelines do not address the level of interaction allowed in a true external replication. Rather the guidelines specify that the replicators should be as explicit as possible in reporting the interaction.
3. Changes to the Original Experiment - Any changes the replicators made to the design, participants, artifacts, procedures, data collection and/or analysis techniques should be clearly discussed along with a description of the motivation for the change(s).

Comparison of Replication Results to Original Results

One of the main values of a replication is the comparison of its results with the results of the original study. It is reasonable to expect that authors will embed brief comparisons of results throughout the presentation of the replications results. To make the comparison explicit, it is also important to have a section specifically devoted to comparing the results of the replication with the results of the original study. This section should highlight the following information:

1. Consistent Results - replication results that supported results from the original study.
2. Differences in Results - results from the replication that did not coincide with the results from the original study. Authors should also clearly discuss how the changes made to the experimental design may have caused these differences.

Conclusions Across Studies

Finally, pulling together information about the original study, changes made for the replication and the comparison of results, the authors should provide a discussion of the current state of knowledge. By combining conclusions from the original study with conclusions from the replication, the authors should be able to provide insights that would not have been evident from either study individually. In this section authors should highlight any conclusions of the original study that were strengthened. This section is also the place to propose hypotheses about new context variables that may have become evident through the analysis of multiple studies.

Last Updated on February 17, 2012 by Jeffrey Carver