New Changes to Badging Terminology

Document last revised August 24, 2020

The ACM Publications Board recently approved a change to the terminology used for its artifact review badges to ensure that our current definitions of “reproducibility” and “replication” are consistent with definitions used by other research communities outside the field of Computing. Following discussions with the National Information Standards Organization (NISO), it was recommended that ACM harmonize its terminology and definitions with those used in the broader scientific research community, and as a result the ACM Publications Board voted to interchange the definitions of “Results Replicated” and “Results Reproduced” to adopt the NISO standard.

ACM will be implementing a versioning system for artifact badging definitions. Badges issued until May 14, 2020 will be considered Version 1.0. Please visit the Badging Version 1.0 webpage for complete description and badge definitions. Badges issued after May 15, 2020 will be badged under Version 1.1 with the version noted on the badge awarded. Please visit the Badging Version 1.1 webpage for complete description and current badge definitions. New versions will be identified on the badge itself and documented by updating the version pages. We will be updating citation pages for all badged items in the DL to reflect the badging history with links to version definitions. We will not make any changes to the originally badged documents. 

The changes being implemented are relatively minor and involve swapping the terms “reproducibility” and “replication” with the existing definitions used by ACM as part of its artifact review and badging initiative.

As updated, the definitions are:

• Results Reproduced ( Different team, same experimental setup)

The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts.

• Results Replicated ( Different team, different experimental setup)

The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

All other definitions remain the same.

Lastly, there are many ongoing efforts in the computing research community on reproducibility and replication of scientific results. The ACM Publications Board wishes to thank the many volunteers who have been involved in these efforts. Everyone has been gratified to see these efforts expand and grow. Indeed, other organizations are adopting the badges that ACM volunteers developed and are using them to indicate works that have been independently reproduced or replicated.