While preparing a post on Big Data for HCI, I started exploring tools for gathering usage data on the Web. There are a lot more tools than I imagined so I won’t list them all. Here is an quick attempt at categorizing them.
Descriptive tools
Descriptive tools give insights on users’ behaviors but do not enable to engage with them. They mostly focus on general stats and support some kind of user segmentation, but it’s mostly about broad quantitative insights.
Site analytics : Tools like Google Analytics and many others. Analytics are rather marketing oriented, but I talked to some startup developers who hooked Google Analytics to their mobile app to know which features were used and which were not.
Page centric analytics : More specialized tools like click trackers generate heatmaps and reports based on mouse events.
Controlled experiments
More widely known as A/B testing, web based controlled experiments have been used for the past 10 years but are increasingly popular. A/B testing enables to run controlled experiments to improve websites or mobile apps often focusing on a specific design point measure with a specific metric (conversion rate, money users spend, etc). An article explaining how to set up experiments (pdf), back in 2005 Jakob Nielsen wrote an article on the limits of the exercise. Google uses this (too?) widely which led some of its top designers to quit in response.
A/B testing tools include Google Web Optimizer, unbounce which specializes in landing pages, and many more. There is also a number of emerging tools for mobile apps such as Facebook‘s one.
Remote usability testing
Remote tests : Unlike A/B testing, remote usability tools enable to study at a distance how a (small) number of users are executing a specific task, often measured in terms of time and errors, sometimes with post-hoc defriefing.
Remote walkthrough : tools like Ethnio support remote recruiting. You can then organize usability sessions by walking users through a task or interviewing them with a variety of tools.
Lightweight tests : a variety of tools support quick data gathering. For instance by asking for rapid feedback or asking what visitors remember after being exposed to a screenshot or mockup for 5 seconds,
Expert feedback : designers ask for the feedback of colleagues/experts. Most often the feedback seems to be on screenshots so still pretty limited.
Surveys
Nothing original here, just surveys served over the web…
Research on the topic
If you know of papers discussing the methodology of running studies (not only controlled experiments) over the distance, leave a comment or send me a ref via twitter @aurelient. Another post will cover research projects once I have enough material.