How Do We Test Usability?

In my last post I brought up the main factors that determine usability for a product or system. Now it’s time to consider various approaches to testing these components and gathering practicable information on usability.

There are different types of usability tests depending on development stage, goals for testing, and resources.

(1 ) A Heuristic Review involves having a trained usability tester evaluate a program and compare it to standards or guidelines that are generally agreed upon in the industry. is a good article on effective Heuristics testing.

(2) In a monitored or formal usability test, a facilitator will set up a physical testing environment. Participants complete tasks while being observed by the facilitator. Data is collected from video/audio of the test session, along with the facilitator notes and any post-test evaluation.

(3) An ad hoc usability test is less prepared than a formal usability test. The participants may be chosen randomly off the street (referred to as hallway testing) and/or the testers may simple be observed using the program without prompts on exactly what tasks to perform. This type of testing will be cheaper and easier to repeat throughout the design process.

(4) A competitive usability test will have participants compare two or more similar programs and answer questions about the performance of each. This type of testing will give the development group a sense of how the program is stacking up to other existing ones.

(5) A usability questionnaire, often administered after a physical usability test of some variety, is a way of getting systematized written feedback from testers. The Software Usability Measurement Inventory (SUMI) is one example of a standard usability questionnaire.


Because the GNOME usability testing internship involves conducting a formal usability test, I want to mention several indispensable components to this type of research. I can’t say that these are hard and fast rules, but they seem to be the precedent for any good monitored usability test.

(1) Usability test are conducted with human participation. The subjects are often going to be representative of the user base for a program, though professional usability testers are needed for some types of tests.

(2) Practical, real world tasks are given for the participants to perform. These tasks are carefully considered and remain consistent for all test subjects.

(3) The participants are monitored while completing a usability test when possible. It’s important that the data come from observing and interacting with the subject performing the tasks rather than a mediated post-task evaluation of the testers experience alone(though that is valuable too).

(4) The combination of facilitator notes, video/audio recording of the test, and tester feedback (verbal or written) is used to create a report providing both qualitative and quantitative data about the usability of the program. This information directly affects the next iteration of the programs development.

As you can see there is a lot of variety when it comes to usability testing. It is crucial that a development team do the research and decide what type(s) of usability testing will work best for the specific circumstances of any given project.

One thought on “How Do We Test Usability?

  1. This is a great list and shows that there’s more than just one way to evaluate the usability of a program. You can do interviews, group review, formal usability tests, questionnaires, or a usability expert can review the system and provide comments based on best-practices (“heuristic review”).

    Alice Preston described 11 different ways to evaluate usability in a design. Unfortunately, I cannot find the link online anymore. Her list:

    1. Interviews/Observations: One-on-one sessions with users.
    2. Focus Groups: Often used in marketing well before there is any kind of prototype or product to test, a facilitated meeting with multiple attendees from the target user group.
    3. Group Review or Walk-Through: A facilitator presents planned workflow to multiple attendees, who present comments on it.
    4. Heuristic Review: Using a predefined set of standards, a professional usability expert reviews someone else’s product or product design and presents a marked checklist back to the designer.
    5. Walk-Around Review: Copies of the design/prototype/wireframe are tacked to the walls, and colleagues are invited to comment.
    6. Do-it-Yourself Walk-Through: Make mock-ups of artifacts, but make the scenarios realistic. Walk through the work yourself.
    7. Paper Prototype Test: Use realistic scenarios but a fake product.
    8. Prototype Test: A step up from a paper prototype, using some type of animated prototype with realistic scenarios.
    9. Formal Usability Test: Using a stable product, an animated prototype, or even a paper prototype, test a reasonably large number of subjects against a controlled variety of scenarios.
    10. Controlled Experiment: A comparison of two products, with careful statistical balancing, etc
    11. Questionnaires: Ask testers to complete a formal questionnaire, or matching questionnaire.

    Of course, there’s some overlap in that list.

    The heuristic review is interesting. A “Usability Analysis” or “Usability Critical Analysis” is basically a heuristic review, and is essentially a “plus/delta” group exercise, focused on what is working in the design (plus) and what needs to be improved (delta). In this review, a panel of usability experts provide comments on a program’s user interface design, and predict how users might respond to such a design. The interface might be a working prototype, or a mock-up on paper. A parallel in other fields of academia is the “Literary Critical Analysis,” a discussion of a work in literature. The use of criticism doesn’t imply disapproval or a negative review, but instead a full review of the work.

    A more compelling way to achieve usability “buy in” with developers is to perform a usability test. There’s nothing quite like seeing a user experience problems with your program.

Leave a Reply

Your email address will not be published. Required fields are marked *