Category: Uncategorized

As I am helping Interactive Surfaces and Spaces conference as an Associate Chair this year, I figured it would be good to summarize a bit the general process and how I go about it. Comments welcome, they are open!

If you land here before having read Ken Hinckley’s “So you’re a Program Commitee Member Now”. You should definitely read it first. This post is more of a personal account.

In the following I’ll likely use AC and PC member interchangeably. AC stands for Associate Chair and PC for Program committee, as ISS shifts to a journal model it’s even called an Editorial Board this year, hence the initials AE (associate editor I guess) you might also see below. While I’m spelling out acronyms, PCS is the submission and reviewing system we use.

The invitation

I was invited by the chairs at the beginning of April. Nic and Fanny gave a super clear overview of the conference reviewing timeline and the workload. It’s the second time I’m doing it for ISS, a conference I really enjoy attending and submitting to.

I guess the first decision happens at this stage. Whether or not you accept the invitation. The review process happens during French holidays which is not great. But given my teaching load during the fall, I would probably say no to most invitations during the period from September to February.

Here is the reviewing timeline we followed:

• Jul 8: Authors’ deadline
• Before July 8: PC members upload a collection of sample papers on PCS.
• July 9-10: PC members declare conflicts and bid on submissions (see instructions below)
• July 10-13: Chairs assign papers
• Jul 14: PC members start assigning external reviewers (1AE: 1 reviewer, 2AE: 1 reviewer)
• Jul 23: All papers should have 2 externals
• Aug 6: External reviews due, 2AC personal review due
• Aug 7: 1AE / 2AE start meta reviews, discussion starts; 3AE requests start
• Aug 14: Discussion ends, 3AE reviews due
• Aug 17: 1AC Meta-review due, 1AE/2AE/3AE tentative recommendation 
• Aug 21: Notifications sent to authors
• Aug 21-Sep 14: Authors revise papers
• Sep 14-28: Second round of reviews
• Sep 28: Individual reviews due
• Sep 28: Discussion starts
• Oct 3: Meta-reviews and 1AE/2AE final recommendation due 
• Oct 5: Notifications sent to authors
• Oct 9: Camera ready due

Workload for ISS

This year ISS is moving closer to a journal like submission process similar to Ubicomp and CSCW. Papers submitted can be accepted, asked for minor revisions, major revisions, or rejected. Papers with majors revisions have to be submitted to the next review cycle (right now, it’s quite close to a full reject).

I reviewed 3 papers and meta-reviewed 3. For each paper, I had to find one reviewer, the other AC on the paper finding the other, leading to 3 reviews + 1 meta-review.

Bidding on papers and managing conflicts

Before the paper deadline, we were supposed to upload some of our own papers. I don’t know if it was for the chairs or for PCS to compute some sort of research profiles/keywords (I’d be somewhat skeptical of this).

After the paper deadline we had two days to bid on papers and declare conflicts. In PCS, we could see a list of people with affiliations and declare whether or not we were in conflict with them.

The ACM has a policy on what constitutes a conflict: https://www.acm.org/special-interest-groups/volunteer-resources/acm-conflict-of-interest-policy. I have a slightly more expansive definition of conflict that includes personal ties. In the end it depends of our appreciation, the chairs sent us as well a list of common conflict cases:

• employment at the same institution or company
• candidate for employment at the same institution or company
• received an honorarium or stipend from the institution or company within the last year
• co-author on book or paper in the last 48 months
• co-principal investigator on grant or research project
• actively working on project together
• family relationship
• close personal relationship
• graduate advisee/advisor relationship
• deep personal animosity

Once you have declared conflicts you cannot get the papers from those people. You are ready to bid on papers. At this stage you can see a list of all the submissions in PCS, and can select one of the following option:

• want: you really want to review this submission, and are qualified to do so
• willing: you are qualified to review this submission, and are willing to do so, though not your top pick
• blank (no option selected): will be considered as a soft willing.
• reluctant: you are not qualified to review this submission.
• conflict: automatically checked for you, after you declare all of your conflicts.

The chairs invited us to select ~10-15 papers as “want” and at least 10 papers as “willing”, so that they could pick from at least 20 papers for each of us. Unfortunately I didn’t keep track of the papers I selected as want or willing to compare them to my assigned papers.

Inviting reviewers

The summer made it maybe a bit harder to get reviewers, or a least to get rapid accept or decline answers.

I typically have an idea of who to invite from the title and abstract. Especially with the bidding phase filtering out papers, it is quite straightforward to think of a few people to invite for reviewing. I still scan the paper to get make sure I know the type of paper it is (system, study, design, theory oriented), and check the references to see if the authors are noticeably citing someone (or not citing people I thought would be relevant). This helps me decide in which order I’ll invite potential reviewers.

One paper turned out to be tricky, five people declining to review or not answering. I asked an old colleague now based in Asia if he had reviewer recommendations on the topic coming from this part of the world where I know less people, and then checked who would be most relevant for the paper. This worked quite well !

Writing reviews

After the invitations, I turned to my reviews on the 3 papers I was AC2. These are “normal” reviews. The only difference I see is that my recommendation for acceptance/revision/reject would necessarily trigger a discussion if it contradicted the recommendation of the primary AC.

Discussions and meta-reviews

After all the papers have their reviews in (I was personally late for some, sorry co-ACs). The discussion and meta-reviewing process starts. One possibility here is to ask for an extra person to review the paper. Given the tight schedule this was probably quite rare (I don’t know how many papers got extra reviews at this stage).

Here starts the synthesis of the reviews. As an AC, I sorted out and weighted the reviewers remarks. In the six papers I was involved in, the reviewers came to the similar conclusions, with different angles and attention to different aspects. But there wasn’t clear conflicts about the quality of a paper (or all the reviewers going in a direction opposite to mine). This made it particularly straightforward to write the meta-reviews, and made our discussions quite simple. For one paper I had a short video-call with the AC to discuss it, and emailed the chairs to discuss the final recommendation.

Finally, I also had to remind some reviewers to ensure that their reviews did not have vague claims about the literature, the supposed knowledge of the authors on topic, or the ability of the authors.

Final decisions

The paper chairs made the final decisions based on our recommendations, and sent them out within a few days. They exchanged with the ACs in case of problem, but not with me (only got an email later to wrapping things up).

The final step will be to check the papers accepted with minor revisions to check if the authors made them.

This is the first time I’m attending VIS, the visualisation conference. We discussed with Romain of blogging on our  experience, here is a personal account, we might post a more structured summary in the near future.

I’ll start with the pre-conference activities. Then will publish a post per day.

The conference really starts on a Tuesday, with Sunday and Monday dedicated to workshops and tutorials. With Romain, Charles Perin, Jeremy Boy and Jean-Daniel Fekete, we are organizing the LIVVIL workshop on Sunday.

Day -2: Friday

We left Lyon on Friday morning. We finalized the workshop plan on the plane from Lyon to Amsterdam, handled last minute things, drafted the workshop introduction presentation.

Compared to other workshops I’ve organized in which only people who submitted papers participated, VIS workshops are open to anyone. As organizers we have no idea how many people will attend. This makes it super challenging to structure the workshop, especially the interactive parts. Discussing only between authors and organizers, is one thing, but involving people who might just drop by for a bit is particularly difficult. So we planned various activities depending on the number of attendees.

The flight was during the day so we also got work/personal stuff done, and worked on a template for blogging during the week.

At arrival I met with Pierre and Yannick who were flying from Paris, we took a cab together to Baltimore and had diner together.

Day -1: Saturday

We booked an awesome airbnb at the edge of Baltimore’s safe area (still less than 15 minutes of walking distance from the conference venue). So my first interaction with a Baltimorean was with a crazy person knocking on the door at 4 in the morning 🎉. To keep up with the spirit I checked some of the Wire places in the morning.

In the afternoon we wrapped-up the workshop organization with Romain, finished the slides. Defined discussion points, and outlined next steps. I was in charge of managing the first panel so I also re-read the papers, and outlined a set of questions, and sketched an introduction to the papers. We also registered, checked the workshop room, and looked for a place to host the post-workshop lunch so that we could have more informal discussions.

Noisy neighbors led me to catch up on videos I had bookmarked in the past few months :

• Robert Kennedy’s talk at  dotScale 2014: Life in the Trenches of healthcare.gov
  It’s an interesting account of a failing IT project. Useful example for project management / software engineering lectures.
• Pat Hanrahan’s talk: People, Data and Analysis at the UW Data Science seminar.
• Antonio Casilli’s talk on Digital Labor and basic income (in french)
  Rather than a precise plan, Antonio Casilli’s raises questions on the relationship between digital labor, taxation and basic income.
• Documentary on Digital Amnesia

 

In this first video from mid-december, Bruce Schneier and Eben Moglen discuss a post-Snowden Internet:

In Protect and Infect – part 2, Jacob Appelbaum presented at CCC the work published in Der Spiegel
http://www.youtube.com/watch?v=vILAlhwUgIU

Last talk is more related to HCI, Michael Brennan asks how to HCI methods can be used to design tools that normal people can actually use. Beyond the Tech: Building Internet Freedom Tools for Real People :

This presentation from Jean Renard Ward covers the history of Pen-Based Computing from the point of view of someone in the 90s. Pretty nice to see all these old pens.