Project 5: Formative Usability Evaluation

CS3724, Fall 2000

Due as per class calendar

Activities

The essence of this assignment is to carry out a full set of formative usability evaluations on your computer-based prototype, following the detailed techniques described in the class notes and discussed and practiced in class. 

Before Data Collection Session

  1. The first things to do before you start any usability testing are to pilot test, pilot test, and pilot test. Get at least one person outside your team (it's OK to use people from other teams, but just for the pilot test) to go through your testing routine as a participant. The primary goal for pilot testing is to make sure the prototype is complete and will support usability evaluation without breaking. Also, from just this quick and simple pre-evaluation activity, you will learn a lot and it will help you tune and calibrate your formative evaluation setup. As a result you may make changes to your usability specifications, bench mark tasks, testing procedures, etc. Make sure to do enough pilot testing to shake-down the benchmark tasks and usability specifications you have chosen for evaluation.
  2. Select user participants. Enlist 3 to 5 representative users from your client organization. Your user participants should cover all of your defined user classes.
  3. Prepare several blank data collection forms. You'll need at least one form per benchmark task per user participant.
  4. Decide team roles. One person should be the evaluation leader or facilitator, to meet and greet the participants, to keep the evaluation sessions moving, and to interact with participants. Another person should be responcible for taking quantitative data (e.g. time on task and error counts). Everyone else should record (write down) qualitative data (e.g. critical incidents and comments by the participants).

During Data Collection Session

  1. Before the participant arrives, get your prototype (low-fidelity or high-fidelity) "booted" up and ready to go. 
  2. Greet the participant. You will meet each participant at a pre-determined time and place, often at the client site. Explain the procedure; give the participant any general written instructions about the overall process. Show them any setup you have and have them sign the informed consent form (required). Answer any questions they might have.
  3. Have participant from the appropriate user class perform benchmark tasks using your prototype. Give the participant the written benchmark task description appearing alone on a single sheet of paper, let them read it, and answer any questions. (Reading time is not part of task performance time.) If your plan to collect verbal protocol data, ask the user to "think aloud" while performing the task.
  4. If you are using a paper and plastic low-fidelity prototype, a team member should be designated as prototype "executor", to move parts of the prototype in response top participant actions. Executor must not anticipate participant actions; especially do not give the correct "computer" response for a wrong user action! Executor must only respond to what participants do (with the prototype), and may not speak, make gestures, etc. You may not change the design on the fly! This is the reason you pilot test so carefully, so this will not even be a temptation.
  5. Do not instruct or coach the participant in details of how to perform a task as they are working. If they get completely stuck and frustrated, then give them a hint, but avoid telling them specifics. The idea is not to get them through the tasks but to discover usability problems in your interaction design.
  6. During the usability evaluation session the team member selected for the role (and all others whenever they can) should take extensive notes on critical incidents and verbal protocol data; the Data Collection Form is a good place to put these notes. Also, after the task is completed, talk with the participant about the task, etc., as you saw in the video in class. Also have the participant do some "free play" and generally interact with the design and take some verbal protocol. This can catch some problems not seen in your benchmark tasks, which are by nature more limited in scope.
  7. At the appropriate time, have your user participants fill out your questionnaire, for your selected questions.
  8. At the end of each session, thank the participant, answer any final questions they may have, and give them their "reward".

After the Data Collection Session

  1. After testing is over, do a complete work-up and analysis of the qualitative data, following the techniques described in the class notes and in-class activities.
  2. From your notes, the critical incident data, and the verbal protocol taking, compile a complete list of usability problems discovered across all your usability evaluation sessions. You should easily find a couple dozen usability problems in your usability evaluation. If you don't, I would be suspicious of something (e.g., how you did the process).
  3. Use a cost-importance table to show your estimated importance and cost to fix and your priority ratios and rankings. The table should be presented in sorted order by priority ranking. Assign yourself a fictitious number of person-hours as the available resources to fix problems and draw a line in the cost-importance table representing where you will run out of resources (i.e., the line between "can fix" and "put off").

Deliverables

Inside the binder, create a "tabbed" section labeled "Project 5". Add this section to the front of your team binder. This way, your binder becomes a cumulative record of your whole project, with the most recent parts first. This section should start with its own separate cover page with (mostly the same as on the front of the binder):

Contents of Project 5 section:
  1. To make this report a stand-alone document, repeat your synopsis or a revised version thereof.
  2. Describe your pilot test, the results, what you learned, and what (if anything) it led you to change in your prototype, benchmark tasks, or your formative evaluation plans. After pilot testing, (for this course) the only changes you should make are ones that are helpful to the evaluation process (e.g., bad bug in prototype), not changes that address only usability.
  3. Describe your participants for formative evaluation, how you selected them (very brief), and how they "cover" your user classes.
  4. Describe the team member roles you decided on for the usability evaluation sessions.
  5. Give an executive summary statement highlighting the overall results of the formative evaluation activity, including any high-level conclusions you might have.
  6. Very briefly describe the critical incident taking process that you did. This is mainly to describe how you collected critical incident data and how it worked for you. (The place for describing problems found by this and other techniques is coming up below.)
  7. Very briefly describe the verbal protocol taking process that you did and how it worked for you.
  8. In a cost-importance table list each usability problem discovered. The problem list should contain at least a dozen usability problems and should be sorted by ascending values of priority rank. If you found a great deal more than that, list the dozen or so most significant (or most interesting) ones, showing this summary information:
  9. Indicate the number of person-hours you have allowed yourself as a hypothetical limit and show it on the cost-importance table as a "cut-off" line. Then show your final resolution (to fix or not) of each problem, based on where each usability problem falls in the rankings with respect to that limit.
  10. Conclude the report with a brief statement of any important lessons learned, the kinds of problems encountered by you and/or the participants, what the participants liked and disliked, and reflections on the prototyping and formative evaluation processes.