Dangers of Self-Reported Usability
By Danielle Lottridge, User Experience Researcher
Usability testing is a method for evaluating the ease-of-use of a product or feature by testing with with people, or “users” of a system. The goal of usability testing is to observe behavior during task flows, observe whether and how users complete flows, and note any conceptual misunderstandings that occur. Participants' reactions are important for other methods such as surveys and concept testing however in usability testing we care whether participants can get the task done. Usability testing provides direct feedback by observing how real people use a product. Usability testing tends to be structured as a one-on-one task-based interview where a facilitator prompts a participant to complete a series of tasks; for example, for an ecommerce website, we might test the usability of adding an item to the shopping cart and completing the checkout process.
One-on-one testing takes time and a facilitator. What if, to save time and money, we tried to have users report usability problems – would they be able to do it? Of course it depends! Participants who frequently participate in usability testing for services such as usertesting.com have a disproportinate amount of experience completing tasks and articulating any hiccups that they encounter along the way. They are more likely to be able to describe their mental model of what they expected and how what they encountered was different than what they expected. However, these participants have so much experience with different systems and design patterns that they may no longer be representative of the general population, but instead representative of the extremely tech savvy, or of amateur UX evaluators. Armed with this experience, these types of participants may encounter a different set of usability problems compared to the general population.
Figure 1. Sketch of Elements of a Mental Model
Image Credit https://openclipart.org/detail/187444/andrew-loomis-drawing-the-head-and-hands-potrace-43
Those with less awareness of their mental model and less experience articulating it won’t be able to report usability problems. A case in point: in a recent quantitative diary study with a few hundred participants in a gen pop sample, a market researcher measured agreement with the statement “This app is easy to use.” Perhaps unsurprisingly, most participants reported the app was extremely easy to use! Behavioral data showed that participants were not able to complete basic tasks. Also, dozens spontaneously asked for instructions for the app. The behavioral data and requests for instructions directly contradict the self-report data. Why is this? Participants may’ve genuinely experienced the app as easy to use and didn’t realize that they weren’t able to complete basic tasks. Based on literature that says that Western people feel competent and don’t realize it when they aren’t [1], participants may feel confident in their abilities and thus don’t entertain the possibility that they weren’t able to figure out an interface. The bottom line is that self-reported usability data are easy and cheap to measure but most likely misleading. As we say in our group, bad data is worse than no data. Observing the behavior of representative participants is a gold standard in usability testing and should remain so.
[1] Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6), 1121.