Reproducibility of Results in the ACM Digital Library

There is a growing need for access to artifacts and experiments associated with scholarly publications in computer science and engineering. To serve this need, new “active digital curation” platforms are emerging that provide support to create and deliver artifacts and experiments as dynamic, interactive content, which can be associated with scientific articles. These platforms have the potential to radically change the processes and mechanisms for scholarly dissemination.

The ACM Task Force on Reproducibility has been working with ACM conferences and journals to understand and articulate common best practices in preparing and reviewing artifacts, and how to reflect them in publication and enable their re-use.

The Sloan Project

Using three use cases, a study funded by the Alfred P. Sloan Foundation was undertaken to understand the landscape of the emerging platforms and how they might be integrated with the ACM Digital Library (DL).

The use cases considered three typical situations

1. Repeating algorithmic comparisons with different datasets;
2. Sharing and modifying experiments as interactive content; and
3. Deriving a new software artifact from an existing one.

Pilot integration was done for each use case with the ACM DL. The three pilots (corresponding to the use cases) employed different exemplar platforms and papers. Although the use cases do not cover all situations, nor do they exhaust the capabilities of each of the platforms, nonetheless they provide insight into best practices for authors, reviewers and publishers to create, evaluate and deploy content from active curation. And most important, they demonstrate the potential of reproducible experiments with their artifacts when integrated with the published research. Video demonstrations were made for the pilots to illustrate active digital curation and to seek the feedback of the broader computer science and engineering community. A short survey is available for the community to offer feedback on how the ACM DL should grow to accommodate active digital curation. The survey is available from: https://docs.google.com/forms/d/e/1FAIpQLSe8ooV0KhLcB1scw_aU9E66nRrYMl8IXpnhLRuDX7pyBsH98w/viewform?usp=sf_link

This survey will remain active until at least June 30, 2017.

The links to the articles:

• Pilot Integration 1: A parallel connectivity algorithm for de Bruijn graphs in metagenomic applications by Flick et al. [DOI:10.1145/2807591.2807619]
• Pilot Integration 2: An actor-based, application-aware access control evaluation framework by Garrison et al. [DOI:10.1145/2613087.2613099]
• Pilot Integration 3: A permutation-based page interleaving scheme to reduce row-buffer conflicts and exploit data locality by Zhang et al. [DOI:10.1145/360128.360134]