CS 3724 Lecture Notes for Sep 20

I. History of usability testing

• Typical excuse: nice idea, but time/resources are limited
• Successful projects: IBM, Microsoft
• Steps: determine tasks and users, design and test activities, develop and test prototypes, collect data, analyze data, repeat
• Techniques: think-aloud, audio/videotaping, note-taking, Nielsen's discount usability engineering, field tests, mockups/demo versions, competitive usability testing

II. Experimental evaluation

VT Usability Lab

Stages of usability testing

III. Observational techniques

• user describes what they believe is happening, why
• simple, little expertise, useful insight

Protocol analysis

• paper and pencil
• audio recording
• video recording
• computer logging
• user notebooks
• automatic protocol analysis tools: EVA, Workplace project

Post-task walkthroughs

• Discuss alternative (but not pursued) actions
• Reflect back on actions

IV. Query techniques

Interviews

• Level of questioning can be varied to suit context
• Evaluator can probe the user on interesting issues
• +: High-level evaluation, info about preferences, reveal problems
• -: hard to plan, need skilled interviewers and willing participants

Questionnaires

• General
  • establish background, gender, experience, personality
  • can overlap other categories but usually not opinion-based
• Open-ended
  • unpromped opinion on a question
  • Can you suggest...? How would you...?
  • often result in brief answers, cannot be summarized statistically
• Scalar
  • judge statement on numeric scale
  • 1-5, 1-7, -2-2 used to balance coarse/fine
  • negative choice (hostile, vague, misleading) on left, positive (friendly, specific, beneficial) on right
• Multi-choice
  • choose one or more from a list of explicit responses
  • ex: How do you get help with the system: manual, online, colleague
  • gather info on user's previous experience
• Ranked
  • place an ordering on a list to indicate user's preferences
  • ex: Please rank usefulness of methods: menu, command line, accelerator
• University of Maryland's QUIS (see Table 4.1), IBM's Post-Study System Usability Questionnaire