Overview: During this phase you will gather empirical data regarding the usefulness, ease of use, and user satisfaction associated with the prototype you built in the previous phase. Several different kinds of data will be collected and interpreted with respect to possible design changes. In addition to gathering formative evaluation results, you will explore how use may vary as a function of users' computing experience.

You will be conducting a think-aloud evaluation of the four scenarios you prototyped in the last phase. For convenience, you may use test participants who are friends or acquaintances, but they should be prepared to role-play the actors from your scenarios. You will run the test on six different users, three who are relatively experienced with computers, and three who are relatively inexperienced.

What to do:
1. Meet and assign roles as appropriate: As before, think about the various jobs associated with this assignment. An important goal is to find ways that everyone can contribute to the project. Possible roles include meeting coordinator, meeting notes taker, task developer, instruction writer, test forms developer, testing scheduler, data analyst, report writer.

2. Develop usability test materials: You will need to develop several sorts of materials (all of these are modeled in the book and in the online case studies) to support your usability testing activities, including:

• An informed consent form that will be filled out and signed by each test participant. We will apply for a class-wide consent from the Virginia Tech IRB for this usability testing assignment, but each of you must also develop a form for your own groups' activities. Be sure to include information about the procedures you will use (e.g., think-aloud, survey completion), so that your users will know what they are agreeing to do in advance.
• User background survey. The background survey should gather relevant experience, attitudes, and expectations about systems such as you have built. Think carefully about what you need to know about your users, both with respect to relevant problem domain experience and computing experience. Don't go overboard -- a single page is usually enough for background.
• Task instructions. You will be doing the "exploratory" form of scenario testing, where users are provided with an actor and usage context and then asked to pursue a goal with the system while thinking aloud about their goals, plans, and reactions. This means that you will need general instructions (similar to Fig 7.8, p. 259) and four different "Imagine you are..." scenario instructions (similar to the paragraph in middle of p. 254). Important note: If there are specific pieces of information that users must enter (e.g., a file or customer name) for the prototype to work, be sure to provide this information as well as the role-play instructions.
• Task data forms. You should keep track of scenario start and stop times, write down the verbal comments made as much as possible, and note/describe any errors that are made. You may want to use the example forms in the book as a model or generate your own. Be sure to decide who will be writing down this information and who will be interacting with the user during the test; it may be hard to do both.
• User reaction survey. This survey will assess reactions to the prototype once all scenarios have been attempted. It should have two components: specific Likert scales that are tied to the features highlighted by your earlier claims analysis (i.e., from the Project 2 design scenarios), and a small set of open-ended questions that gathers more qualitative reactions. Again, use the model in the book or other case studies as examples.

Conduct at least one pilot test (see p. 263) of your materials before scheduling your test users, so that you are familiar with your materials, know they will work, and can estimate how much time to allow for each session (probably about 30 minutes per person).

5. Run the usability test sessions: Choose a location and use the same one for all tests. The location should be one that is quiet and where you will not be disturbed (e.g., not the CS undergrad lab!). A team member's apartment is fine, or some other room at Virginia Tech with a computer that can run your software. If you are unable to find such a quiet place, contact Vinoth early on to make special arrangements using departmental or other university rooms.

At least two team members must be present at each session, one to interact with the subject, and another to record times, errors, comments, etc. Begin the session with the informed consent form, answer questions about procedures, and administer the background survey.

Start with the general instructions, emphasizing and answering questions about the think-aloud process; you may want to demonstrate thinking aloud for people who are not sure what to do. Ensure that you have established how they will signal the start/stop of a task.

Then, for each scenario, read aloud the scenario storyline (the paragraph that sets up the role-playing situation and general goals). Be sure that the prototype is in the starting state assumed by the scenario. Remind the user to think out loud when doing the task, then observe what he or she does, and write down verbal comments as best as you can. Carefully r theecord start-stop time and errors for these episodes.

Follow the usability testing guidelines provided in Chapter 7. For example, make sure that you develop and follow a user assistance policy (p. 262) -- you must know in advance when and how you will intervene with help when a user gets stuck (the book recommends using a graduated approach). You should also discuss in advance what will count as an error and what user behavior you will try to observe and record.

After all tasks are complete, ask the participant to complete the user reaction survey. Then debrief him or her about what you are expecting to find, how you will use the findings, and answer questions. Be sure to thank each person for volunteering!

6. Summarize the usability test data: Each team's testing sessions will generate slightly different data, so you should discuss what you have collected and how best to summarize it. In general however, you should create summary tables or charts of the following sorts:

• User background characteristics. You do not need to summarize everything you learned from the background surveys, instead you may select the characteristics you believe are most important to the sessions you conducted. Use tables or graphs to present data that is numerical in form (e.g., averages with standard deviations), use histograms or pie-charts to present categorical data. Be sure to distinguish between your inexperienced and experienced user groups in these tables or graphs, so that we can see the differences between them.
• Time and error data. Create one or more tables showing the times/errors for each user (grouped into low and high experience level), as well as average times and errors for each scenario. Be prepared to discuss what these performance data tell you, either about the individual scenarios, contrasts between the two different experience levels, or the individual users.
• Think-aloud data. Create a text table for each scenario that summarizes the most interesting comments and behaviors you observed. In some cases you may want to report comments verbatim (in this case be sure to indicate the participant number, remember that all test participants are guaranteed anonymity).
• User ratings. Summarize the results of the Likert scales included in the reaction survey, by converting the agreement-level responses to numbers (as in the textbook). Provide the ratings for each user (again, grouped by experience level) as well as averages. You can use tables or charts to show these. Be prepared to discuss these results with respect to your Project 2 claims analyses.
• Open-ended questions. Summarize answers to any open-ended questions included in the reaction survey (i.e., rather than reporting them verbatim). Again, be sure to distinguish among the two groups of users, because they may have had rather different experiences.

7. Write your usability test report: The design reports should have the following labeled sections. Number all pages that follow the Table of Contents, and use tabs or other separators to organize your report so that it is easy to review:

1. Cover Sheet: label the phase as "Usability Evaluation", and include group number, team member names and student numbers, and due date
2. Table of Contents: list page numbers for each required element
3. Overview: a 1-2 page introduction to this group project, including the issues you had identified in advance, the general methods that you used, and a high-level summary of what you discovered in the testing.
4. Procedure: a 1-2 page description of the procedure you followed for testing, including where the testing took place, who participated, your assistance policy, and any unusual characteristics or events.
5. Results: the tables and charts you created to summarize users' background, time and error data, think-aloud comments and other observations, rating scale data, and written comments. In addition to providing these summary representations, include a brief textual summary of each, for example stating in words what a chart or table conveys, pointing to particularly interesting pieces of data. This will be the biggest part of the report, probably 5-10 pages.
6. Interpretation: a 2-3 page discussion of your results. For example, you might discuss how your test data relates to the claims analyzed earlier: did you find what you expected, did you unearth new concerns that should be expressed as claims? Or there may be interesting results with respect to one or two individual scenarios that you want to talk about. You should also discuss any differences observed between the two levels of experience, and what if any implications this has for your system. This is a good place to talk about design changes that you would make, but be sure to provide the usability testing rationale for these changes (e.g., don't just include a list of features you didn't get to yet). I do not expect you to discuss all your findings here, focus on the ones that have the strongest or most interesting implications for usability.
7. Bibliography: cite any sources you used in collecting or analyzing your usability test data.
8. Appendix: include all test materials you prepared and used: informed consent forms, task instructions, completed background surveys, data collection forms, and user reaction surveys.

Grading: The written project will be evaluated for overall completeness, as well as the quality of each component. We will grade the report using this evaluation form.