First Impressions


Wow, it’s great to say that I have completed the paper prototype testing on 10 participants!  I’d like to share my first impressions about how the testing went before sharing a deeper analysis of the test cycle. Let’s hear some things that went well!

1. I’m confident that each tester understood the task at hand and chose the setting that they first associate with the task to the best of their ability.

2. Many of the testers found that they liked GNOME settings and indicated an interest in using GNOME in the future!

3. The test uncovered a lot of patterns of user behavior, even between testers with completely different computer usage history.  I’ll obviously go into more depth on this point in my analysis as this is the most interesting aspect of the test.

4. The tests all went smoothly and consistently for the most part.

5. Testers often had questions about open source and linux operating systems.  I think the test got people more comfortable with the idea that open source software is and can be as good or better than its proprietary alternative.

6. Testers were able to give their choice on each task without really much difficulty.  Unlike my last usability test, where testers had to complete tasks, this test was a “where would you click first?” scenario for each task.  I  think that leveled the playing field between experienced users and novice ones in terms of time and successful completion.

7. Every tester completed all 23 tasks!


Allow me to sort of walk you through a few points where my expectations didn’t match up with the reality of testing.

1.The introduction-  I spent a lot of time trying to think of an introduction that would be informative, but not complicated or overly long.  It turns out not many participants were really able to listen and absorb information  when I was reading from a script.  Something about that interaction made a lot of people’s minds go blank I guess.  I ended up explaining the test again in my own words to many of the testers.  So, I just think it’s an interesting observation that even though the content of my explanation vs. the script didn’t vary much, people heard me clearer when I talked candidly about the test.

2. Think Out Loud-  It was quite difficult to get some testers to consistently follow the think out loud protocol.  I had to prompt a lot of testers to speak about the choices they were making, and then it seemed to weaken their confidence in their choices.  It seemed as though some of the testers associated my prompt for more info with an incorrect answer.  I don’t know how to better prepare folks for using this protocol, but I’d love suggestions from those who have had good test responses in the past.  There were a few really awesome testers who got it though, so I don’t want to make it sound like everyone ignored think out loud.  It was just a lot harder to pull thoughts from testers than I had hoped.

3. This is not a test of the participants skills!- I had expected that the disclaimer that the test was about the software and how useful it is, not how well the tester can do the tasks, would be sufficient to help participants relax.  It seemed really hard for some testers to accept the idea that whatever choice they made would be legitimate and useful to the usability test.  I told testers to act like they would were they using the application at home.  You wouldn’t labor over which setting to click on, right?  You would just choose one and move on if it wasn’t correct.  I believe that the fact that this test was on paper put more space between the tester and normalcy.

4. I would have changed some of the wording and order of tasks after seeing how testers interpreted them.  It can be hard, as someone who uses GNOME, to put myself in the position of someone who doesn’t.  I think I could definitely improve on the scenario task creation in a future iteration of the test.

So there you have it, a taste of what’s to come in my analysis of this usability test.  Overall I am pleased with how the tests went and I really think there will be some useful usability info to draw on for the designers and developers of GNOME settings.  That’s what it all about, right?  It feels very satisfying to (hopefully) make the new setting app more useful!
More to come soon…


One thought on “First Impressions

  1. Great first impressions!

    I especially loved that you highlighted “It can be hard, as someone who uses GNOME, to put myself in the position of someone who doesn’t.” That’s an important takeaway for any usability testing. When you are familiar with something and know how to use it, it can be hard to imagine how easy it will be for someone who doesn’t know the software. As you commented, you use GNOME every day, so these tasks seemed easy to you. It wasn’t as easy for the testers.

    I hope other open source developers also learn from this. A usability test isn’t that hard to do; you need to figure out what you want to learn from the test, and you need to organize it, but actually *doing* the test isn’t that difficult. And you don’t need very many testers to get useful results. Most of the time, about five testers will give you “good enough” results that you can tweak the design. And then you need to repeat the usability test. Usability and design are iterative: create a design, do a usability test, tweak the design, repeat the usability test, etc.

    I also loved your comment that testers “internalized” the test. This is common. People just want to do well, and it can be frustrating when they can’t do something we’ve been asked to do. You had a great reply: “You wouldn’t labor over which setting to click on, right? You would just choose one and move on if it wasn’t correct.” That’s exactly the right suggestion to make. In a “work” setting, people generally don’t agonize over which menu item might contain some functionality. People tend to click on different menus until they find the function they want – and if they can’t find it in a few clicks, they may give up and assume the functionality isn’t there.

    Great job! I’m looking forward to your analysis! A reminder: I find it’s best to do the analysis in two parts: a heat map, and a discussion of themes. To create your heat map, use the method I describe on my blog about how to create a heat map. (Search for “heat map” if you don’t have the link.)

    You may already have an idea of themes: What did testers find easy to do? What did testers find to be more difficult? Starting with the heat map, themes may become obvious. Look for “hot” rows (lots of black, red or orange). “Cool” rows with lots of green or yellow are likely to be areas with good usability. Pull quotes from users during the tests to give examples of problems in “hot” rows. What was common between these “hot” rows? Were users confused by the same or similar design patterns in these difficult areas? Also look at what went well: What was common between the “cool” rows? Why did users find these easy to do?

Leave a Reply

Your email address will not be published. Required fields are marked *