Evaluation of Digital Mock-ups
After viewing the data, I feel it is important to review the demographic data first, as I believe it accidentally ended up being the actual variable being tested (as opposed to the difference in the designs, which was not greatly changed from A to B). This is due to the reason I just mentioned, as well as a small sample size (5 respondents per survey) combined with luck of the draw leading to more sports-inclined individuals taking B vs A.
Please see figures 1a-c below. Survey B clearly had more avid sports fans in its population. As one might expect, these fans are also more avid watchers of sports, and more avid sports streamers.
Figure 1a
A: Mean 2.4 with STDev 1.14 B: Mean 4.2 with STDev 0.84
Figure 1b
Figure 1c
Using this information to set a frame of reference for the meat of the survey (i.e. the questions about the product), one can easily see that more avid sports fans are more likely to find the app useful (Figure 2b). However, it does seem that regardless of demographics, users were generally able to understand what the app does (Figure 2a).
Of the fans who would find the app useful, most do not have any existing workarounds, aside from merely not using their phone:
“Put the phone aside while watching”
“I don't check my phone/laptop while watching sports”
“keep away from your phone!”
“I do encounter it. Just live with it.”
Comparing the two designs, specifically the change in the confirmation page, there is certainly a slant towards users favoring the updated design (Mock B), though the results are not significant - a greater sample size is needed. Results can be viewed in Figures 2c and 2d.
Figure 2a
A: Mean 4.2 with STDev 0.84 B: Mean 4 with STDev 0
Figure 2b
A: Mean 2.2 with STDev 1.1 B: Mean 3 with STDev 1.2
Figure 2c
A: Mean 3.4 with STDev 0.5 B: Mean 4 with STDev 0
Figure 2d
Evaluating the word selection gives additional context into the users’ general thoughts about the product. Again, due to the small sample size and the general similarity between the two mocks, I feel it is more useful to not compare the surveys side-by-side, but rather combine the results together. The top words were:
Effortless - 50% of respondents
Useful, clean, relevant, simplistic, convenient - 30% of respondents
This validates my (intentionally) simple design, and seems to resonate well with users who want a quick and easy experience. This is backed up by the qualitative responses:
“The steps were simple.”
“The vast majority made sense”
“The confirmation screen is clear.”