#uxra

[yahooaccessibility]

Building deeper relationships with users

HC Lai, User Experience Researcher,  Rushani Wirasinghe, User Experience Researcher

Digital advertising is the financial backbone of the Internet industry, enabling many companies to monetize the traffic to their digital products and provide customers with free access to this incredible content. At Yahoo, our user experience (UX) researchers not only work closely with the teams designing consumer products, but also with those managing our advertising platforms. We have identified best practices for conducting effective user research in the context of advertising products, because it’s important to recognize the unique nature of the participants.

Yahoo Gemini Advertiser Council

The B2B UX research team has shifted from one-time interactions to longer term relationships with users of advertising and publishing tools, and actively works to build these relationships. Partnering with the product marketing team, the B2B UX research team developed the Yahoo Gemini Advertiser Council, which engages Yahoo Gemini advertisers annually with the goal of enabling users to be part of the product development process. Yahoo Gemini customers participate in diverse activities such as research studies throughout a product lifecycle, quarterly communications, and biannual in-person meetings with our product team. These activities not only positively impact advertisers’ perception of Yahoo Gemini, but they facilitate deep collaboration between our users, product team, and sales team. Based on direct feedback, Yahoo Gemini Advertiser Council members feel more confident that Yahoo can meet their advertising needs.

Flurry’s Corporate Sponsorship  

Another example of a user panel is Flurry’s corporate sponsorship of Hacker Dojo [1]. Hacker Dojo is a community of app developers and startups located in Silicon Valley. We originally reached out to Hacker Dojo to conduct a one-time usability study on their premises. When we learned that we could expand the relationship from a one-time engagement to a longer-term relationship in the form of a corporate sponsorship, we did not hesitate. The sponsorship has provided Flurry with a user panel that they use to collect user feedback with minimal effort. It has also been a good opportunity to promote Flurry within the local community.

Building a user panel not only helps minimize the lead time to recruit third party users but it also allows researchers to follow a community of users and observe how their usage of a product evolves over time.

1.     HackerDojo. http://www.hackerdojo.com/  (accessed Jan 10, 2017)

[yahooaccessibility]

Agile User Research

Frank Bentley, Senior Principal User Researcher

At Yahoo we move quickly.  Our product teams are often shipping new features every week or two and designers are constantly exploring new solutions. While User Research methods have traditionally been fairly slow to execute, especially in the academic traditions of anthropology and human-computer interaction, we have a need to move more quickly and fit our work into the weekly sprints of our product teams.

For Yahoo Mail, we tried an experiment last year where we conducted weekly user research every Friday.  This gave teams Monday through Thursday to work on their new designs and implementations and Friday for us to conduct research and for them to observe and plan for their next week from our research labs. Concepts could be tested in a design state iteratively from week to week, and then tested again in a working build in a subsequent week once it was implemented.  This allows us to rapidly test new designs, iterate each Friday until we get something that both provides value and is usable by our participants, and test final implementations once they are ready.  We augment this with larger scale surveys and field studies as necessary for bigger features and strategic directions.

Along with testing prototypes, these sessions also include in-depth interviews on particular topics of interest, including walkthroughs of current behavior and understanding of recent experiences.  These questions help us to understand particular domains over several weeks and dozens of participants and help to set strategy when combined with other qualitative data from surveys and usage logs.

These weekly studies have transformed the design and product process to truly be user focused.  We test a wide variety of concepts and implementations each week.  Product Managers and Designers don’t have to worry about setting up a special study for their topic, getting users recruited, and waiting several weeks for studies to be executed.  They can rely on having spots in the weekly study for last-minute additions of new prototypes or qualitatively testing topics that have appeared in logs or online user feedback.

All of this has helped bring the Yahoo Mobile Mail app to its highest ratings ever, hitting and then staying at 4.5 stars on both iOS and Android for most of the year, while shipping a wide variety of new features for our users.

[yahooaccessibility]

Time Is Money!

HC Lai, User Experience Researcher

As a UX researcher working on advertising products, a decent portion of my day is spent observing how advertisers use advertising solutions. During various studies I have conducted, it came to my attention that their frustration often rooted in time and efforts required to complete a task. Additionally, we heard search advertisers’ needs in achieving even better results with their search campaigns on Yahoo Gemini while spending less time and effort managing multiple platforms. We came to realize that “time is money” to our advertisers.

Informed by these insights, Yahoo Gemini recently introduced an account sync feature that addresses advertisers’ needs. Account sync is a powerful tool to help search advertisers manage large-scale search campaigns across multiple search engines. The ability to sync accounts can make it simple for advertisers to import their entire eligible portfolio of campaigns from other platforms into Yahoo Gemini, with just a few clicks. By seeking a deep understanding of users’ values and beliefs, UX Research helped to inform product opportunities and we hope to continue improve the Yahoo Gemini experience.

[yahooaccessibility]

Dangers of Self-Reported Usability

By Danielle Lottridge, User Experience Researcher

Usability testing is a method for evaluating the ease-of-use of a product or feature by testing with with people, or “users” of a system. The goal of usability testing is to observe behavior during task flows, observe whether and how users complete flows, and note any conceptual misunderstandings that occur. Participants' reactions are important for other methods such as surveys and concept testing however in usability testing we care whether participants can get the task done. Usability testing provides direct feedback by observing how real people use a product. Usability testing tends to be structured as a one-on-one task-based interview where a facilitator prompts a participant to complete a series of tasks; for example, for an ecommerce website, we might test the usability of adding an item to the shopping cart and completing the checkout process.

One-on-one testing takes time and a facilitator. What if, to save time and money, we tried to have users report usability problems – would they be able to do it? Of course it depends! Participants who frequently participate in usability testing for services such as usertesting.com have a disproportinate amount of experience completing tasks and articulating any hiccups that they encounter along the way. They are more likely to be able to describe their mental model of what they expected and how what they encountered was different than what they expected. However, these participants have so much experience with different systems and design patterns that they may no longer be representative of the general population, but instead representative of the extremely tech savvy, or of amateur UX evaluators. Armed with this experience, these types of participants may encounter a different set of usability problems compared to the general population.

Figure 1. Sketch of Elements of a Mental Model

Image Credit https://openclipart.org/detail/187444/andrew-loomis-drawing-the-head-and-hands-potrace-43

Those with less awareness of their mental model and less experience articulating it won’t be able to report usability problems. A case in point: in a recent quantitative diary study with a few hundred participants in a gen pop sample, a market researcher measured agreement with the statement “This app is easy to use.”  Perhaps unsurprisingly, most participants reported the app was extremely easy to use! Behavioral data showed that participants were not able to complete basic tasks. Also, dozens spontaneously asked for instructions for the app. The behavioral data and requests for instructions directly contradict the self-report data. Why is this? Participants may’ve genuinely experienced the app as easy to use and didn’t realize that they weren’t able to complete basic tasks. Based on literature that says that Western people feel competent and don’t realize it when they aren’t [1], participants may feel confident in their abilities and thus don’t entertain the possibility that they weren’t able to figure out an interface. The bottom line is that self-reported usability data are easy and cheap to measure but most likely misleading. As we say in our group, bad data is worse than no data. Observing the behavior of representative participants is a gold standard in usability testing and should remain so.

[1] Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6), 1121.

[yahooaccessibility]

Rapid Surveys to Support Design

By Frank Bentley, Senior Principal User Researcher

Often a design or product team will have a question in the course of their work where an immediate answer is needed in order to proceed. Perhaps they want to know if it's worth it to support a feature given people's current behaviors. Or perhaps they want to test some different icons for intelligibility. Ideally, they'd have answers in a few hours so that they could continue their work. Larger product teams might have multiple questions like this per week in the course of their work.

However traditional User Research activities take time. In-person studies have to be planned, participants need to be recruited and data needs to be analyzed. This can typically take a week or more. Large market research studies often take even longer and are very expensive. But product teams in agile environments need to work faster than these traditional methods can support.  

Over the past two years, we've developed a process for quickly designing, running and analyzing mid-scale surveys to support design and product questions within hours or a day at most. Frequently we can have a question one afternoon, run the survey that evening and have data by the morning of what a representative sample of Americans do. We can run dozens of these surveys every month to understand which use cases to support, where current frustrations lie and the intelligibility of designs. The key is to have a few focused questions and to deploy the survey using a large online survey platform (e.g. Survey Monkey or Amazon's MTurk) to a broad audience. These are not marketing-style attitudinal questions, but questions on actual behaviors that people engage in as a part of their everyday life, so that we can quickly quantify whether a use case is worth supporting. This method has helped us to learn that over 90% of Americans receive coupons and deals in their email and that 64% had used one in the past two weeks. We've also learned about the percentage of email users who turn off their notifications by platform and have tested a variety of icons for the Stationary feature of Yahoo Mail, finding one with 7x the recognition rate of the current design within hours of receiving the files from a designer.

Behavioral surveys like these help User Research to work at the speeds of agile teams and provide quick data about user behavior in the wild to designers and product managers to help set product strategy. And we've seen that the results are often within just a few percentage points of larger, much more expensive market research studies that can take months to fully execute. In a study using eight questions covering a range of user behaviors (everything from using coupons received in email to enabling mobile mail notifications to subscribing to particular video streaming sites), we observed an average error of only 3.6% with SurveyMonkey panels and 7.1% with MTurk panels compared to a gold-standard n=1000 market research panel. This is well within the margin of error needed for rapid feedback in the design process. If you have $100 (the approximate cost of a user study) and 2-3 hours, you can quickly get representative data from 150 participants and move design along at a rapid pace! Running multiple studies over time and triangulating findings can serve to further increase confidence in the results.

[yahooaccessibility]

Expanding the Notion of "User"

By Mike Shebanek, Senior Director of Accessibility

Starting today, you’ll notice some significant changes to our blog, both in design and in content, that better reflect who we are and what we do. I’d like to to tell you why we’re making these changes and what you can look forward to as you follow our blog.

One of the things that makes Yahoo’s approach to accessibility unique is that its Accessibility Team and its User Experience Research Team are integrated. That may sound like an obvious approach, but it hasn’t always been this way at Yahoo and we’ve come to find out it’s pretty unusual in the tech industry. In the all-too-rare organizations that actually have a dedicated Accessibility Team, they often report to the chief legal counsel, or are part of an administrative or compliance group. Other times they’re siloed in one division, which prevents them from being able to impact product development or customer support in another. Having a dedicated Accessibility Team and where that team is placed in an organization says a lot about a company’s values, and this is especially true of Yahoo.

In our case, we’re part of a team called User Research and Accessibility or UXRA for short. Our team supports the entire company – every division, every VP, every employee – and it is tasked with representing our users’ unique needs, requirements and points of view in every activity we undertake. We are not just “closely aligned” with our user research team – we are the same team! We share user study labs, training, techniques, findings and the same mission: to accurately represent all users. We work to ensure that users come first in whatever we do, and that this includes people with disabilities, of all ages, of all genders and of all abilities. So, whenever we perform a user study, take a survey, gather product feedback or identify requirements, people with disabilities are included along with everyone else. We can’t imagine any other way.

To better reflect this mission and to give you deeper insight into how we go about understanding and representing our users, you’ll be hearing more from our User Researchers in addition to our Accessibility Team members on this blog.  We have a lot of exciting things to share, and we hope it is informative and inspiring to you and our industry peers as we all continue to reach for greater inclusion.

[yahooaccessibility]

Yahoo UXRA at CHI 2017

Frank Bentley / Sr. Principal Researcher

The Yahoo UX Research and Accessibility (UXRA) team is heading to the ACM SIGCHI Conference on Human Factors in Computing Systems (otherwise known as CHI 2017) in Denver!  

This year we have a series of great talks from our researchers and interns about the research we have conducted across consumer and advertising products. If you’ll be at CHI, you can find the room numbers for talks listed below.  We’re also hiring, so stop by our booth at the breaks. If you can’t make it to Denver, links to download our papers are available below.

CHI is an example of how research in industry builds off of the best academic research available and contributes back to the research community with our own findings and systems that we create. Each of our papers below cites dozens of studies from other corporate labs or universities, and we hope contributes to future scholarship in these venues.  We are an applied research group, and the findings from this research have a direct impact on our products and research culture within the company, as well as helps to put our users first in creating new product experiences at Yahoo for our billion+ users.

“If a person is emailing you, it just doesn't make sense”: Exploring Changing Consumer Behaviors in Email

Monday, 11:30 - 12:50 Room 203

Authors: 

Frank R Bentley, Yahoo, Sunnyvale, United States

Nediyana Daskalova, Yahoo, Sunnyvale, United States

Nazanin Andalibi, Yahoo, Sunnyvale, United States

Comparing the Reliability of Amazon Mechanical Turk and Survey Monkey to Traditional Market Research Surveys

Wednesday, 16:30 - 17:50 Room 302

Authors:

Frank R Bentley, Yahoo, Sunnyvale, United States

Nediyana Daskalova, Yahoo, Sunnyvale, United States

Brooke White, Yahoo, Sunnyvale, United States

Applied Research for Advertising Products: Tactics for Effective Research

Thursday, 11:30 - 12:50 Room 302

Authors:

Hsiao Chun Lai, Yahoo, Sunnyvale, United States

Rushani Wirasinghe, Yahoo, Sunnyvale, United States

It's All About Coupons: Exploring Coupon Use Behaviors in Email

Thursday, 11:30 - 12:50 Room 302

Authors:

Nediyana Daskalova, Yahoo, Sunnyvale, United States

Frank R Bentley, Yahoo, Sunnyvale, United States

Nazanin Andalibi, Yahoo, Sunnyvale, United States

Talks by Other Yahoo Employees

Increasing Activity in Enterprise Online Communities Using Content Recommendation

Tuesday, 11:30 - 12:50 Room 110/112

Authors: 

Ido Guy, Yahoo Research, Haifa, Israel

Inbal Ronen, IBM Research, Haifa, Israel

Elad Kravi, Technion, Haifa, Israel

Maya Barnea, IBM Research, Haifa, Israel

Multimodal Classification of Moderated Online Pro-Eating Disorder Content

Tuesday, 16:30 - 17:50 Room 110/112

Authors:

Stevie Chancellor, School of Interactive Computing Georgia Institute of Technology, Atlanta, United States

Yannis Kalantidis, Yahoo! Inc, San Francisco, United States

Jessica Annette Pater, School of Interactive Computing Georgia Institute of Technology, Atlanta, United States

Munmun De Choudhury, School of Interactive Computing Georgia Tech, Atlanta, United States

David A. Shamma, Centrum Wiskunde & Informatica, Amsterdam, Netherlands

Chair, Session: 360 Video

Wednesday, 14:30 - 15:50 Room 205

Frank R Bentley (Yahoo Sr. Principal User Researcher)