{"id":36,"date":"2013-12-06T17:12:42","date_gmt":"2013-12-06T16:12:42","guid":{"rendered":"https:\/\/tabard.fr\/blog\/?p=36"},"modified":"2013-12-06T17:35:43","modified_gmt":"2013-12-06T16:35:43","slug":"building-incentives-for-replication","status":"publish","type":"post","link":"https:\/\/tabard.fr\/blog\/2013\/12\/06\/building-incentives-for-replication\/","title":{"rendered":"Building incentives for replication"},"content":{"rendered":"<p>David Karger <a href=\"http:\/\/haystack.csail.mit.edu\/blog\/2013\/12\/05\/demanding-a-replicability-paragraph-in-conference-submissions\/\">recently posted some thoughts on replication at CHI and WWW<\/a>, suggesting a mandatory replication paragraph at the end of every paper. It&#8217;s a nice idea and while it&#8217;s kind of implied in the post, a special section in the review form of reviewers and ACs might be a more manageable thing to push forward. This could offer a smoother transition and progressively help articles supporting replication stand out in PC meetings. It might also be easier to push something in the review system than asking for a specific paragraph in papers. Then the different subcommittees could decide to deal with this differently depending on their research practices.<\/p>\n<p>A review section dedicated to replication could help the authors and the reviewers (who are likely authors) build a discourse on replication and data sharing (beyond people going to workshops and SIGs on replication). Hopefully over time, this would also lead to higher standards on replication and data-set quality.<\/p>\n<p>I submitted a note to CHI last year with all the data and scripts (<a href=\"http:\/\/dx.doi.org\/10.1145\/2512349.2512794\">ended up at ITS<\/a>). The didn&#8217;t receive any comment on the data (even critical). Sadly I ended up improving the paper but not the data which I consider to be at least half of the contribution.<\/p>\n<p>Just for reference here is the text above the review field on PCS (for CHI&#8217;13), not a lot on replication:<\/p>\n<blockquote><p><small><\/p>\n<p>Write your review of the paper here. Please address each of the following issues:\n<\/p>\n<ul>\n<li>Significance of the paper&#8217;s contribution to HCI and the benefit that others can gain from the contribution: why do the contribution and benefit matter?<\/li>\n<li>Originality of the work: what new ideas or approaches are introduced? We want to emphasize that an acceptable paper must make a clear contribution to Human-Computer Interaction;<\/li>\n<li>Validity of the work presented: how confidently can researchers and practitioners use the results?<\/li>\n<li>Presentation clarity;<\/li>\n<li>Relevant previous work: is prior work adequately reviewed?<\/li>\n<\/ul>\n<p>\nIf you have concerns about the methodological or statistical approaches taken by the authors, or its level of advancement over prior work, please cite a source for your objection (e.g., a definitive paper, a set of professional guidelines or a standard textbook). This is needed to help authors improve their submissions and to enable the Associate Chair to evaluate potentially conflicting reviews.<\/p>\n<p>\nPlease consider making any other recommendations that you think might be of use to the author(s).<\/p>\n<p><\/small><\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>David Karger recently posted some thoughts on replication at CHI and WWW, suggesting a mandatory replication paragraph at the end of every paper. It&#8217;s a nice idea and while it&#8217;s kind of implied in the post, a special section in the review form of reviewers and ACs might be a more manageable thing to push &hellip; <a href=\"https:\/\/tabard.fr\/blog\/2013\/12\/06\/building-incentives-for-replication\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Building incentives for replication<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-36","post","type-post","status-publish","format-standard","hentry","category-thougths"],"_links":{"self":[{"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/posts\/36","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/comments?post=36"}],"version-history":[{"count":8,"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/posts\/36\/revisions"}],"predecessor-version":[{"id":49,"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/posts\/36\/revisions\/49"}],"wp:attachment":[{"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/media?parent=36"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/categories?post=36"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tabard.fr\/blog\/wp-json\/wp\/v2\/tags?post=36"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}