Moderating Usability Tests
Techniques
Concurrent Think Aloud (CTA)
One technique commonly used to understand participants’ thoughts as they interact with a product is having them think aloud while they work. In CTA, the moderator only uses prompts such as “mm hmm,” and “keep talking.” The goal is to encourage participants to keep a running stream of consciousness as they work.
With CTA, we are able to understand participants’ thoughts as they work through issues. CTA is excellent at eliciting real-time feedback and emotional responses. However, the dual task of thinking aloud while working sometimes interferes with usability metrics, such as accuracy and time on task.
Retrospective Think Aloud (RTA)
In RTA, the moderator asks participants to retrace their steps when the session is complete. Often participants watch a video replay of their actions, which may or may not contain eye-gaze patterns. Because RTA allows participants to work in silence, it is an obvious way to control for the negative effects of CTA.
The process of thinking aloud does not interfere with usability metrics (such as time on task), although overall session length increases dramatically. RTA appears to elicit more insight words than CTA; however, memory is fallible, and participants often do not stay on task during RTA. (For a brief review of CTA vs. RTA, see Olmsted-Hawala & Romano Bergstrom, 2012).
Concurrent Probing (CP)
Another technique is probing participants as they work on tasks—when they say something interesting or do something unique, the researcher asks follow-up questions. Usability tests rarely include CP because it interferes with the natural thought process. It can also affect the progression participants would make on their own, without interruptions.
However, CP can be valuable if goals do not include strict user experience measurements. Survey researchers often use CP when pretesting terms and wording to assess how well people understand the questions and concepts.
Retrospective Probing (RP):
Another technique is waiting until the session is complete and then asking questions about the participant’s thoughts and actions. Researchers often use RP in conjunction with other methods—as the participant makes comments or actions, the researcher takes notes and follows up with additional questions at the end of the session.