Usability Evaluation

This project will be done in teams of three (one team may have four or two). You will be assigned to a group based on how you fill out the questionnaire on the first day. You will then form teams with one person from each group. Subsequently, you will select a system for evaluation from the list maintained by the GTA. Just as described for the selection of papers for individual presentations, teams will sign up on a first-come basis. Any requests to evaluate a system not listed on the GTA's page must be cleared with Dr. Tatar ASAP, as the deadline for system selection is less than two weeks after the semester begins.

1. System selection: Review the brief descriptions and/or follow the links to get more information. All of the systems listed have demo or evaluation versions available. Choose one that is intrinsically interesting to you and your partners, or perhaps one that has 1-2 novel or unusual features you would like to study in use.

Selection of a system for study must be done by the beginning of class on Monday, Feb. 9.

2. Study design: Although a prime goal of your study will be to evaluate the usability of your selected system, you must also design it to be a pilot study of at least one CSCW research question. Research questions may involve user variables (e.g., experience, intimacy, goals, time pressure), system variables (e.g., tools used, number of windows, size of database), or task variables (e.g., complexity, task type, dependencies). Because meaningful research questions will depend on the system studied, and because many students in the class have little or no HCI background, each team is required to meet at least once with Dr. Tatar during office hours or at some other mutually-convenient time to discuss your ideas and plan your study. A schedule will be posted on the CoWeb (as soon as it becomes available) to facilitate this (when the Coweb becomes available!).

Note that this will be a relatively "small" study. If you propose a within-subjects design (i.e., where all users will be treated the same), six groups will be sufficient to carry out a small analysis. If you develop a between-subjects design (i.e., users are assigned to different conditions), please collect data from at least five groups in each condition.

You must have your usability study design approved by Dr. Tatar by the beginning of class on Monday, February 16. At this time you will also be required to develop and submit an IRB form (the GTA will consult on the IRB process as needed).

3. Data collection and analysis: The facilities in McBryde 102 will be available on request for data collection, but it is possible that some studies will be carried out in more informal settings. Note that if you do choose to use the usability lab, you will be responsible for scheduling times only as needed, so that others may share the space. For access or information about this lab facility, please contact the GTA.

4. Study report: After your usability data have been collected, you will analyze them according to the plan approved by Dr. Tatar. You will then write a paper using the CSCW conference proceedings format, with a limit of 6 ACM double-column pages (the page limit includes figures and references). The paper should be modeled on the CSCW and/or CHI papers that you have been reading for the course. It should begin with motivation and relevant background work, including citations to published papers or other resources. It should then have a methods section that describes your study design and procedures. This should be followed by a section reporting results, and a final section of discussion and conclusions. There should also be a reference section at the end.

The final report must be submitted on paper by the beginning of class on Wednesday, April 7^th.

5. Informal demo: In the second half of the semester, we will schedule out-of-class demonstrations of the systems that were evaluated. Each pair who worked together on this project will run the demo, organizing it however they choose. Although these demos will not be graded, work toward organizing and attending the demo sessions will figure into the class participation grade.