Unmoderated Remote Usability Testing: Good or Evil? Kyle Soucy — 2011 IA Summit
This recording could really benefit from including the visuals Soucy refers to throughout her speech. She breaks down, for the layman or the unfamiliar UXD moderator the newest tools and ways of remote testing sans the moderator. She brings up the point that I second, “there is no way to say, stop, please think out loud.” This was my foremost concern for some dud remote usability studies that didn’t include any “think out loud” element to them, even when the audio was working, since participant typing was captured. This may have been preventable with further wording and reminders on each task screen. I did a dry run of the test and tasks before sending out and didn’t notice this myself. Interestingly, a user that I tested in person was fully thinking out loud, possibly from that prior experience thinking out loud in a face-to-face test.
She discusses the tools for unmoderated remote usability testing and she thinks the strongest point is metrics, quantitative data versus qualitative data. The web analytics are a key metrics that you get when you test remotely versus in person. Finding out when they abandon the task and survey. She purports the advantages of live intercept recruiting. There isn’t a whole lot in this that Nate Bolt doesn’t cover in his Remote Usability tome but she is very enthusiastic about the richness of and the actionable data delivered. She highlights, don’t send them on a scavenger hunt, and advises finding and exploring the content. During her example of the Lowe’s remote user testing, one of the tasks she wrote was very general, unlike my very specific tasks. She just put a destination out there and the user narrated how they would approach that task and how the site did or didn’t help accomplish this. The nuance may be open-ended questions.
The benefits of unmoderated tests are:
-shorter -less tasks -can supply big numbers -test multiple websites simultaneously -video recordings (but require time note taking) -easy and rapid iterative testing
The disadvantages of unmoderated tests are:
-no spontaneity or adjusting the test or tasks -difference of data -possibly users only testing for honorarium -don’t believe rating themselves -research only is observation -the recruiting process is not optimal -who you test matters