Wow, it’s great to say that I have completed the paper prototype testing on 10 participants! I’d like to share my first impressions about how the testing went before sharing a deeper analysis of the test cycle. Let’s hear some things that went well!
1. I’m confident that each tester understood the task at hand and chose the setting that they first associate with the task to the best of their ability.
2. Many of the testers found that they liked GNOME settings and indicated an interest in using GNOME in the future!
3. The test uncovered a lot of patterns of user behavior, even between testers with completely different computer usage history. I’ll obviously go into more depth on this point in my analysis as this is the most interesting aspect of the test.
4. The tests all went smoothly and consistently for the most part.
5. Testers often had questions about open source and linux operating systems. I think the test got people more comfortable with the idea that open source software is and can be as good or better than its proprietary alternative.
6. Testers were able to give their choice on each task without really much difficulty. Unlike my last usability test, where testers had to complete tasks, this test was a “where would you click first?” scenario for each task. I think that leveled the playing field between experienced users and novice ones in terms of time and successful completion.
7. Every tester completed all 23 tasks!
Allow me to sort of walk you through a few points where my expectations didn’t match up with the reality of testing.
1.The introduction- I spent a lot of time trying to think of an introduction that would be informative, but not complicated or overly long. It turns out not many participants were really able to listen and absorb information when I was reading from a script. Something about that interaction made a lot of people’s minds go blank I guess. I ended up explaining the test again in my own words to many of the testers. So, I just think it’s an interesting observation that even though the content of my explanation vs. the script didn’t vary much, people heard me clearer when I talked candidly about the test.
2. Think Out Loud- It was quite difficult to get some testers to consistently follow the think out loud protocol. I had to prompt a lot of testers to speak about the choices they were making, and then it seemed to weaken their confidence in their choices. It seemed as though some of the testers associated my prompt for more info with an incorrect answer. I don’t know how to better prepare folks for using this protocol, but I’d love suggestions from those who have had good test responses in the past. There were a few really awesome testers who got it though, so I don’t want to make it sound like everyone ignored think out loud. It was just a lot harder to pull thoughts from testers than I had hoped.
3. This is not a test of the participants skills!- I had expected that the disclaimer that the test was about the software and how useful it is, not how well the tester can do the tasks, would be sufficient to help participants relax. It seemed really hard for some testers to accept the idea that whatever choice they made would be legitimate and useful to the usability test. I told testers to act like they would were they using the application at home. You wouldn’t labor over which setting to click on, right? You would just choose one and move on if it wasn’t correct. I believe that the fact that this test was on paper put more space between the tester and normalcy.
4. I would have changed some of the wording and order of tasks after seeing how testers interpreted them. It can be hard, as someone who uses GNOME, to put myself in the position of someone who doesn’t. I think I could definitely improve on the scenario task creation in a future iteration of the test.
So there you have it, a taste of what’s to come in my analysis of this usability test. Overall I am pleased with how the tests went and I really think there will be some useful usability info to draw on for the designers and developers of GNOME settings. That’s what it all about, right? It feels very satisfying to (hopefully) make the new setting app more useful!
More to come soon…