CS/ISE 5714 - Usability Engineering - Spring 2008
Dr. Pardha S. Pyla
Homework #3 -- Usability Problem Analysis
This homework is to be done as an individual assignment only!
General instructions:
We have selected 10 critical incidents (or usability problems) from the NY Ticket Kiosk that Dr. Hartson encountered during the in-class usability testing demo originally recorded in the HCI lab in McBryde 102 and played back in class (screen video of Dr. Hartson serving as "expert" user participant, using the NY Ticket Kiosk ). We used Morae in the lab to extract 10 video clips (links to each are below) from the one large Morae video/audio we made during that session.
For your class assignment, you are to diagnose each of these usability problems by classifying it within the UAF (identifying the problem type and subtypes and cause(s) within the interaction design), using the UAF Viewer tool. We will send the URL for this Viewer in a listserv message to the class. No username or password will be required to access the viewer; just click on the Login button to get in. To review the critical incident occurrence, task, and context, you can also use the NY Kiosk site, the benchmark tasks (Benchmark Task #1, Benchmark Task #2), and the entire Morae video/audio recording of the evaluation session (screen video of Dr. Hartson serving as "expert" user participant, using the NY Ticket Kiosk )
Deliverable:
A sheet, or sheets (stapled together), of paper with your name and "Homework #3" at the top, printed from a Word (or similar) document. For each problem:
- Give the problem number from the list (no need to copy the problem over) and write your chosen diagnosis path from the UAF, as we did for the examples in class. You can be as terse as you like in writing the path names, as long as we can understand them.
- Write a brief rationale statement about why you picked the category you did and not any other likely one.
- Declare whether you used the Wizard for help with this problem. (We will give you information on how to access the Wizard.)
At the end, please indicate any feedback about the experience, including the UAF, the UAF Viewer and the Wizard. We are especially interested in improving the Wizard via your experience here.
Hints:
- Treat each problem as a single, separate problem (not having to extract multiple problems and not combining related problems this time).
- UP4 actually identifies two problems, but both have essentially the same diagnosis, so treat them together as one.
- UP10 is not about the problem of which object to click on to select the Three Tenors and the fact that the text looks like a link but isn't clickable. This problem is about something that is true in this clip, but the participant didn't comment on it explicitly. This is a critical incident noticed by the evaluator independent of the participant's actions and comments. This is about the fact that the little blue box you have to click on is very small and, therefore, it is more difficult to click on it accurately.
-
While many diagnoses are obvious, there is not always necessarily
one "correct" answer. With this kind of diagnosis, "correctness"
is more a question of reliability, how similar are your results to those
of experienced usability engineering practitioners.
- If you think it is a toss-up between two classification categories and you definitely cannot decide between the two, give both as your answer (will be considered your first and second choices in order), showing them as separate answers. (Don't give more than two possibilities, please.)
- Grading will be based first on how well your classification matches the expected norm (by experienced practitioners).
- If your classification doesn't match, then we look at your rationale and any second choices. If one of your choices is a good match, you will get at least partial credit. If your choice is not a match to the "norm" but your rationale is good, you can get up to full credit.
- Since the clips are from a real evaluation session, expect realistic distribution of diagnoses across the Interaction Cycle.
- IMPORTANT: If you have trouble making your diagnosis at the top level, or you feel you're having to guess, or don't feel confident about your top-level diagnosis choice (a common occurrence for beginners, as you saw in class), try using the Wizard, which we will make available to you.
The Morae Video/Audio Clips Defining the Usability Problems
Caution: These clips are in .wmv format. They may not work with Mac computers
and they might not play equally well with all media players (so we highly
recommend Camtasia Player, which will be made available to you).
Apologies for the fact that the header for each video says Video Clip #1.