Empowering Repeated Research Initiative

Overview

Repeated research is an important part of scientific methodology. Repeating or replicating research not only brings an independent perspective to the investigation, but also provides a basis for comparing different approaches and extending known results. The process of repeating others’ research also allows us to learn, not only from their insights but also their experimental methodology. Replication is thus an important facet of scientific debate.

Up until recently repeating computer science research, while desirable in principle, was often difficult in practice. This is due to several well-identified factors such as the lack of incentives (e.g., in publication venues) that value such effort, lack of common research platforms and tools, and the difficulty of packaging the requisite digital artifacts in an easily shareable form and the mechanisms to combine and publish them. Change is needed.

The objective of the “Empowering Repeated Research Initiative” is to create a forum that catalyzes such change. We envision two complementary activities to be undertaken in this initiative, in the short and longer term. A first set of nearer-term activities include a series of workshops (see below) educating students and involving faculties in the art of reproducible research, and identifying and developing a set of tools and best-practice-techniques for enabling reproducible research. We would like to invite submissions that repeat results published in past papers and provide a brief analysis of the results as well as the ability to replicate them, similar to “Xen and the Art of Repeated Research” by Clark et al. The repeated results can range over a variety of computer science systems topics such as: wireless, networking, operating systems, distributed systems, cloud computing, and others. With the emergence of generally accessible testbeds, technological innovations such as containers, virtual machines, or computational notebooks, repeatability can be  both easier and an inherent part of the discovery process. Submission on these tools and experience with them are also welcome. Consensus best-practice techniques will emerge from these workshops. The longer-term objective is to to reach out to the main stakeholders, professional societies and conferences, with recommendations arguments and potential solutions for embedding reproducibility in the research life-cycle.

Empower will set up a series of workshop, with regular presentation, demos, panels and discussion. These will be organized virtually but additionally collocated with conferences and workshops when possible. Each event will be prepared by a small group of colleagues who will set up the agenda, organize the debate and synthesize recommendations.  A continuous “challenge” will be organized with an award to recognize important “hands-on” contributions in the field with a strong emphasis on student involvement.

Topics will cover but are not limited to:

  •         Reproducible network and system research
  •         Computation and data reproducibility
  •         Educating the future researchers
  •         Testbed Requirements to support reproducibility

  •         Best Practices for the 3M’s: metadata, measurement and monitoring
  •         Open Data
  •         How other sciences address reproducible research
  •         etc.

Organization and agenda

The EU Empower support action and the NSF PAWR Office will organize the initiative, provide the management and logistics. Participation is free but must be made here:

 

1st Virtual Workshop

  •         Tentative date: September/October 2020
  •         Topic: testbeds and tools
  •         Chair: Kate Keahey

 

2nd Virtual Workshop

  •         Tentative date: December 2020

 

3rd Physical workshop

  •         April 2021, Paris or Athens

Program Committee

Fdida, Serge | Sorbonne University, Co-chair

Keahey, Kate | University of Chicago, Co-chair

Bonaventure, Olivier | UC Louvain

Calvert, Ken | University of Kentucky

Carle, Georg | Technical University of Munich

Claffy, Kc | University of California, San Diego

Crowcroft, Jon | University of Cambridge

Dabbous, Walid | INRIA

Gosain Abhimanyu | Northeastern University

Hunold, Sascha | TU Wien

Kurose, Jim | University of Massachusetts Amherst

Legrand, Arnaud | CNRS

McKeown, Nick | Stanford University

Nakao, Aki | University of Tokyo

Rezende, Jose | Federal University of Rio de Janeiro & RNP

Ricci, Robert | University of Utah

Ruth Paul | RENCI

Seskar Ivan | Rutgers

Taufer, Michaela | University of Tennessee

Vermeulen, Brecht | imec / Ghent University

Zink, Mike | University of Massachusetts Amherst

 

Reproducibility Continuous Challenge

Chair: tbd

Sponsors

 

 

 

 

 

 

 

 

 

 

Contact & Registration

 

Contact: stavmag@the.forthnet.gr

Registration:https://docs.google.com/forms/d/1off8Ijo1_VRVkL–gg5pJ_biJUuKkAFkvyO2TMuFlMo/viewform?edit_requested=true

Information: https://www.advancedwireless.eu/index.php/reproducibility/