David Karger recently posted some thoughts on replication at CHI and WWW, suggesting a mandatory replication paragraph at the end of every paper. It’s a nice idea and while it’s kind of implied in the post, a special section in the review form of reviewers and ACs might be a more manageable thing to push forward. This could offer a smoother transition and progressively help articles supporting replication stand out in PC meetings. It might also be easier to push something in the review system than asking for a specific paragraph in papers. Then the different subcommittees could decide to deal with this differently depending on their research practices.
A review section dedicated to replication could help the authors and the reviewers (who are likely authors) build a discourse on replication and data sharing (beyond people going to workshops and SIGs on replication). Hopefully over time, this would also lead to higher standards on replication and data-set quality.
I submitted a note to CHI last year with all the data and scripts (ended up at ITS). The didn’t receive any comment on the data (even critical). Sadly I ended up improving the paper but not the data which I consider to be at least half of the contribution.
Just for reference here is the text above the review field on PCS (for CHI’13), not a lot on replication:
Write your review of the paper here. Please address each of the following issues:
- Significance of the paper’s contribution to HCI and the benefit that others can gain from the contribution: why do the contribution and benefit matter?
- Originality of the work: what new ideas or approaches are introduced? We want to emphasize that an acceptable paper must make a clear contribution to Human-Computer Interaction;
- Validity of the work presented: how confidently can researchers and practitioners use the results?
- Presentation clarity;
- Relevant previous work: is prior work adequately reviewed?
If you have concerns about the methodological or statistical approaches taken by the authors, or its level of advancement over prior work, please cite a source for your objection (e.g., a definitive paper, a set of professional guidelines or a standard textbook). This is needed to help authors improve their submissions and to enable the Associate Chair to evaluate potentially conflicting reviews.
Please consider making any other recommendations that you think might be of use to the author(s).