jump to navigation

Preece Ch. 12 (pp. 584-595), 14 June 11, 2009

Posted by maggertj in Uncategorized.

CH. 12 – Introducing Evaluation:

Evaluation is an integral part of the design process.  It is how you collect information about the users’ interaction with a prototype or product.  It focuses mostly on the overall experience of a design and its’ usability.  It is interesting to note that this is how you differentiate how specific target users interact with a design.  The example given was that a generalized design will different for a 25 year old versus an 85 year old, or an expert interacts differently than a novice.  These interactions should be tested on top of theorized.

12.2 – The why, what, where, and when of evaluation

Users want interaction to be easy, effective, safe, and satisfying.  This is an overall experience, and can only be tested via evaluation.  Previously in history, users only expected functionality, now they expect an experience.  Businesses and marketing also need evaluation to maximize profit.

When determining what to evaluate, it largely depends on the product itself.  Government traffic sites might ask for fewer accidents, whereas toy makers might ask if a toy is infant safety proof.  These questions come from more than a usability standpoint.  They involve aesthetic, emotional, engaging, and motivating qualities in their bases.

Where evaluation also depends on the product.  testing the key sizes on a cell phone can be done quantitatively in a laboratory, whereas testing the the fun of a new toy is better done in a natural setting.  Testing how often a person might use their cell would clearly have to be done while watching a person’s normal life.

The same can be said on when a product should be evaluated.  Brand new product lines are tested after much research and data has been collected, while a software upgrade might be tested immediately after it has been created.  The two types of evaluation done are summative and formative.  Summative  is testing done after a product is finished.  Formative is done during a design process.  The purpose of evaluating is to ensure a product accurately interprets the users’ needs, and that the design embodies these needs.

12.3 – Evaluation approaches and methods

The three main approaches to evaluation are usability testing, field studies, and analytical evaluation.   Each have their positives and negatives and are suited for specific purposes.  Usability testing measures the performance of users doing specific tasks.  Users are asked to perform, while the tester records the user’s actions via video and sound.  Questionnaires and interviews usually follow the session.  The defining characteristic of usability testing is that the evaluator controls the environment and the format of the test.  This is a problem in that it perpetuates a non-realistic scenario.  The everyday routine of people’s lives does not cause the test to be delayed in any way.  Also the person being tested may or may not behave normally during the test.  They may feel like they are being tested, rather than the product, which in turn causes them to behave abnormally.

Field studies are done in a natural setting, which aims at understanding what people do normally and how products mediate their activities.  They are used to help identify opportunities for new technology, establish the requirements for new design, and facilitate the introduction of technology.  Data is taken in the form of events and conversations.  These are recorded as notes or audio and then taken for later analysis.

Analytical evaluations are separated into two categories, inspections and theoretically based models.  Inspections are based on heuristic evaluation and walkthroughs.  Theoretically based models are used to predict user performance.  Heuristics are based on common-sense knowledge and usability guidelines. These can sometimes lead evaluators astray because finidings are not always accurate.  Cognitive walkthroughs simulate a a user’s problem-solving process through dialogue.  these are mostly useful when optimizing efficacy of interfaces.

These 3 types may or may not be used separately.  It is perfectly normal to use all 3 in a single design process.  Using these 3 types, there are 5 main methods in evaluation basically. They are observing the users, asking their opinions, asking expert’s opinions, testing performance, and modeling performance.  It really just depends on what is the most opportunistic at the time in correlation to the product.

Ch. 14 – Usability testing and field studies

14.2 – Usability testing

Usability testing varies in the amount of control the evaluator has. Due to the testing environment and the type of tasks given, results vary greatly.  The two main components to testing are the user test and the satisfaction questionnaire. The purpose is the measure performance and the reasoning/methods are achieved those results.  As was mentioned earlier, results and interaction are recorded at all times to allow later analysis.  Time and number are the main things measured.  Quantitative results are the easiest to determine this way.

There are multiple ways of recording data, via key loggers, video cameras, audio recordings, or indirect survey.  Usability labs are very expensive and as a result, evaluators have been trying to find cheaper alternatives, such as mobile kits or smaller home cameras.  The other “x” factors of these tests come from finding/enticing users and the development of the tasks for them. The procedure of running the test also changes the results.  Evaluators can also run usability tests like experimental testing.  They formulate a hypothesis to test, then setup control groups and variables.

14.3 – Field studies

Field studies serve to find out a product’s overall scope within the confines of everyday life.  This is done by allowing a user to interact with a product during their normal everyday routine.  The positive of this is that evaluators get real-life data based on likely scenarios playing out.  The negative of this is that evaluators get mounds of irrelevant data that may not be useful.  They also cannot test a specific hypothesis in this sense, because the events that play out are totally random.

The time range used for a study can be a few minutes or several months.  Data is done via observation and interviews.  It’s recorded by audio, video, and field notes.  Users may be asked to fill out questionnaires.   The behavioral aspects of field studies is certainly much larger in range than in usability testing.  This however can be aided by identifying the artifacts a user interacts with during their day.  These artifacts help quantify the exact nature of the routine users take.



1. seanryanconnolly - June 15, 2009


This is an excellent summary of the work. You have a clear understanding of the steps and pieces involved in evaluation. I know this is your last blog for this class, but for a stronger score, we would like to see you connect the readings directly to the work in class, or, in the projects you are working on. Don’t just intellectually absorb the content. Match the concepts up with your activities and articulate how one informs the other.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: