Observational evaluation
Can be used at both ends of the process:
Beginning: Requirements gathering activity
- Observation of existing system in use
- Trial of putative design solutions
During: Formative evaluation
- Lo-fi and alpha prototypes
End: Summative evaluation
- Observation of beta prototype and production version in use
Gathering empirical data from usage of the software by representative users
Paper prototype stage
Software prototype stage (Alpha) (Functionality may be ‘Wizard of Oz’)
Initial release stage (Beta)
Long term evaluation of installed system
Process and requirements:
1. Subjects are set tasks to accomplish using the system under test
Based on scenarios collected during requirements analysis phase
Background training, motivation and data set up
A precise script should not be given
Task should be encountered as it would in real use
2. Observed attributes of the performance are recorded
Time to complete
Mistakes and errors made
Methods used
Expressions of confusion or frustration
Comments on acceptability, satisfaction and conformance to users mental-model
3. Data from sessions analysed
Data prepared:
Notes and commentary transcribed
Recorded video tagged
Data reviewed to extract information:
Occurrence of critical incidents (this is normally when a user is brought to a halt)
Timings and counts of events
Expressions of dissatisfaction (the number of times an action is completed)
NB Make sure the people who have designed the interface are not visible or become involved with the people testing.
4. Result presented
Formative evaluation
Developers or designers directly observe sessions
Notes and summaries
Video presentation of highlights / low-lights
Summative evaluation
Formal report might well be required for sign-off and checking back on what the initial brief / requirements were
BS ISO/IEC 25062:2006 Software engineering – Software product Quality Requirements and Evaluation (SQuaRE) – Common Industry Format (CIF) for usability test reports
How to run a session:
1. Running a session – Preparation
Design sessions
Establish test objectives
Health & Safety Risk Assessment
Design and write protocol
Prepare consent forms
Prepare test material
Book facilities
Book staff
Facilitator
Recording technician
Receptionist
Recruit subjects
Establish type and number required
2. Running a session - Managing the session
Subject greeting and introduction
Obtain nondisclosure agreement signature (where required)
Explain purpose and procedure - N.B. The software is under test, not the subject
Establish that the participant can withdraw at any time
Obtain consent form signature
Give reward (if any – and get receipts)
Obtain necessary demographic information
Facilitate task/s
Execute protocol in a consistent manner
Be supportive but not directing
Allow subject to give up
Encourage commentary (where protocol requires)
Manage time
Conclude session
Obtain reflection/comments (where protocol requires)
May involve subject reviewing recording of performance
Request that subject does not discuss specifics of test with colleagues (where necessary)
Thank subject for their contribution
3. Running a session – Securing and managing data
Secure data immediately after a session
Complete your own notes
Collect notes from observers
Ensure notes are complete and readable
Ensure any recording was successful
Label and back-up recordings
Approaches
– Straight observation
– Subject gives ongoing commentary
– Subject gives a post-task commentary
– Subject questioned during task
– Structured interview/questionnaire on completion of task
– Subject reviews recording of session and gives commentary
– Pair discovery eg a couple booking something together
Legal/Ethical issues
No undue stress both physical and mental
The purpose of the testing must be adequately explained and assurances of confidentiality must be authentic
Workstations must be set up in compliance with appropriate standards and legislation
Length of sessions and intensity of activity must be acceptable
Subjects must be able to request a break or early termination of the session
Refreshments should be available
– Children as subjects
Be aware of legislative requirements
Agreement given by parent or guardian
Recruitment usually through organisation (school, club, etc.)
– People with disabilities
Mental capability – informed consent must be obtained
Motor or sensory disability – particular attention must be paid to health and safety concerns
Recruitment must be managed appropriately
– Professionalism (duty of care to employer)
Impartiality and objectivity of tester
Economic efficiency of operation
Usability Lab observation is very expensive