Preece et al Chapter 12 & 14

Question Words of Evaluation
The need for evaluation can be answered in

1. Why?
   Evaluation is necessary to see if a certain design or product is liked or even usable by users. In the business world, evaluation can help designers to improve a product before it is fully released.
2. What?
   What to evaluate is completely governed by the problem space. It can really be anything from websites to toys to traffic controls.
3. Where?
   Similar to the “What,” the where to evaluate is dependent on the problem space. Some situations need to be evaluated in a structured, controlled lab environment or a more natural environment. The text uses the example of a web page being better evaluated in a lab setting, while a toy or a phone would be better in a natural setting.
4. When?
   Once again, when to evaluate depends on “What.” However, there are two general times to evaluate. Summative evaluations are done after the design process and formative evaluations are done during the design process.

Usability Testing

Usability testing is completely controlled by whatever person is performing the evaluation. It is normally used to check for consistency in a product (usually software). Commonly, a user is observed closely and the time it takes to complete a certain task is recorded (quantitative data collection). In addition to performing tasks, users may fill out satisfaction questionnaires or be interviewed for more information (qualitative data collection). Usability testing is especially useful for products that have multiple generations (word processors, etc.). Users are often recorded on camera and microphone so that more than just timing information can be collected.

Usability labs can range in size from an entire facility to the contents of a suitcase. For a real life example, one can be found in Informatics West. Some testers have begun to use remote testing methods as well, eliminating the need for a central usability lab.The final usability section covers how to conduct a usability test and basically equates to an excerpt from a statistics course book. It covers variable choice, control experiments, null and alternative hypotheses, participant designs, and t-tests.

Field Studies

Field studies are more often observed outside the lab setting and are less controlled. The data collected can be used for any number of analyses. Data can also come in many forms (notes, video, interviews, etc) both from the observer and the participant. The amount of time it takes to conduct a field study can be anywhere from a couple of minutes to years. While on site, observers must take into consideration the system or product breaking down and whether or not it can be easily repaired. The results of field studies can be written up in a number of ways, including narratives and vignettes.