Sep 10, 2014
seen from Malaysia
seen from France
seen from United States
seen from Malaysia
seen from Moldova
seen from China
seen from United States
seen from China
seen from Italy
seen from Malaysia
seen from China
seen from Russia
seen from Italy
seen from Maldives
seen from Malaysia
seen from United States
seen from United States
seen from Yemen
seen from China
seen from Belgium
A Few Discoveries About Remote Usability Testing
I just completed some remote usability testing (3 on usertesting.com, 3 on youeye.com) for a new tool I am working on at my company, Smart Destinations. Sure, I learned about some flaws in the interface, but I also learned a few things about setting up remote usability testing:
1. Be explicit in your instructions; avoid pronouns.
My first task read:
"What can you do on this page? How do you expect it to work?"
In normal writing or speech, "it" would clearly refer to "this page." However, almost all participants assumed "it" referred to the product discussed on the page, the Go Chicago Card. This led to some confusion and frustration for them, and little useful feedback for me.
2. Provide enough information
It's hard to strike the right balance between giving testers sufficient context and bogging them down with details or worse, providing them information you actually want them to get from the page(s) you're testing. However, if your test is focused, especially on a subpage or specific feature, it's better to err on the side of too much information, so the tester doesn't get distracted and veer off course.
3. Make clear what portion of the site the person will be testing.
Most people assume they are starting on (what they refer to as) the "homepage." This struck me as odd, given how few of our site's visitors actually do start on the homepage. Do they assume this only in the context of usability testing? Or whenever people click through to a website from a search engine result page, do they generally assume they're landing on the "homepage"? Perhaps people misunderstand what a homepage is? I'm not sure, but the upshot is you may want to make clear in the introduction where in the overall site hierarchy the tested page sits.
4. Steve Krug is right, 3 is enough.
Obviously, this is not my own discovery. I chose 6 participants for this test, but learned 95% of what I learned overall after the first three participants. It would far more productive to do frequent tests with fewer participants.
5. Seeing people's faces may not be all that helpful
So, when I first tried youeye.com, I thought actually seeing the participants in their webcams was pretty compelling. However, having run this second test on both usability.com (no webcam), and youeye.com, I wonder how much more you learn from their facial expressions than just their tone of voice. In Rocket Surgery Made Easy, Steve Krug says don't worry about recording user's faces. He may be right.
YouEye vs. UserTesting Online Usability Testing
I recently set up two online usability tests, one with usertesting.com, one with the youeye.com. I set them up with the same introduction, tasks, and follow-up questions, making them as similar as possible.
The results were remarkably different. There are functional differences between the two services, mainly:
YouEye records the webcam of the tester and tries to record their emotions (more on that in a later post)
UserTesting.com has simply been around longer, and has developed some really nice functionality like watching videos at 2x speed and exporting annotations to Excel.
But, the stark difference was the testers. I wrote earlier about how the UserTesting.com testers seemed rather expert at web browsing, as well as overly positive about the experience. In contrast, the YouEye testers showed honest confusion and displeasure, and generally seemed to approach the site much more like a real, casual user would. Now, in the end, the majority were still generally positive about our product, but the videos revealed much more about the failings of the webpage they were testing.
Another interesting difference was that the YouEye users took from 3 to 9 minutes to complete the same tasks that the usertesting.com users took from 11 to 25 minutes to do. No time limit or goal is imposed on either site, as far as I know, so I'm not really sure what would cause this difference.
For this round, though both sets of tests were informative, I believe the YouEye tests were more useful to me -- and not because of the "emotion graph" or even the webcam feature, but simply because their testers were more like our users...a little impatient and not deeply engaged.
100% Conversion Rate!
I work on an e-commerce website. I don't think I'm giving away any big secrets to reveal that our online conversion rate is below 10%. Fewer than 10% of people who look at one of our pages buy something.
However, if I were to guess at our conversion rate based on usertesting.com testers (I'll abbreviate UTCT), it would be somewhere around...100%! I've seen this in previous rounds of tests I've done on that website, but in the latest it was particularly evident: the UTCTs' experience with our site is not our typical visitors' experience. I think there are a few factors at play:
UTCTs are not actually shopping online for themselves. Obviously, the whole scenario is unrealistic, as it would be in many different usability testing scenarios. Rather than deciding whether to spend their own hard-earned money, they are actually in the process of earning money.
UTCTs are eager to please. Because higher-rated testers get tapped for tests more often (and thus can earn more money), consciously or unconsciously, the testers are more positive about their experience than they would normally be.
UTCTs tend to be more advanced computer and internet users. One could probably assume this from the fact that they know about and have signed up for usertesting.com. But beyond that, you will see in their computing environment or interactions things that many average or novice users simply wouldn't know about.
What to do about it?
Usertesting.com does let you recruit and use your own testers. I'm not sure how the prices differ or even quite how it works. But it might be better to get your grandma and your neighbor to test than the UTCTs.
Perhaps a better-crafted introductory scenario could put them more in the shopping frame of mind. Rather than "You're planning a trip to Chicago," something like: "You have $500 dollars left in your vacation budget to spend on your trip to Chicago..." etc. Try to make the fact that they're deciding whether to part with cash real to them.
Lastly, I think usertesting.com should change its rating system slightly. Currently, a 5-star system is used to rate the testers. If someone loved your site and had nothing but positive feedback, are you going to give them 1 star? No, more likely 4 or 5. But, how useful was that test? Did you learn anything you could use? Often, not really. usertesting.com should instead allow us to rate each test session. And probably not using a star system but something more "objective" feeling. Like: "On a scale of 1 to 10 how much did you learn from that test session?" Over time, the system could even learn the best type of tester for different types of sites, rather than just who is polite and chatty and has nice things to say.