Paper Prototype Test Analysis

Introduction

This is an analysis of a recently performed paper prototype usability test for GNOME settings. The usability test was conducted by myself (Ciarrai) with the help of 10 participants between the dates of July 29-Aug 14 2016 in Tucson Arizona. The test used two printed images of the proposed changes to the GNOME settings application panel, and one printed image of the WIFI panel. Each tester was given 23 tasks and asked to respond with their first menu choice for completing each task. The test was conducting using the think aloud protocol. The responses were recorded for note taking purposes. Tester demographics regarding gender, technical skill, and preference were taken for future analysis.

Methodology

Each test started with an introduction:

You have been asked to participate in a software usability test for GNOME. Similar to the way Windows and Mac OS are desktop systems, GNOME is a desktop system that provides for user access to applications, files and management tools in Linux operating systems. GNOME is free to anyone and follows the principles of open source software.

GNOME is currently modifying the Settings application. We want to see how these changes affect users before implementation. Because the updates are so new, we won’t be testing Settings on a version of the program. We will use a paper list that represents the order, naming and some details of the new Settings application. I will explain a number of tasks to you, one at a time, and would like you to tell me where in the settings you would go first to accomplish each task. I would also like you to explain your decision out loud as make the settings choices. Talking out loud will help me understand your choices better.
This is entirely a test of the Settings application and how well it works. We are not testing you and there is no wrong answer. If you can’t figure something out, that’s okay and will still provide us with useful information. So, for every task I would like you to and tell me the setting with which you would first try to complete the task. I will write down the setting you pick and then we will move on to the next task. There are 23 tasks in all. After we have gone through every task I will ask for some feedback on the test. I would like to record each test to ensure the results and subsequent report accurately reflect what happened during the test. If recording presents a problem for you, please say so now. Let me know if you have any questions. Thank you for your help in improving GNOME.

After the introduction the testers were given a chance to ask any clarifying questions. Many of the testers were still confused on the specifics of the test at this point and a more procedural explanation of the test was given. At this point the test began. The usability test evaluated paper prototypes for the newest iteration of GNOME settings. Three images were used in all to represent the proposed changes to the settings application.

These paper prototypes represent the look of the settings application design update, including full scroll of the drop-down menu and sub menus for Devices and Details. The sub menus were cut out and hidden from view until the participants chose either Devices or Details from the left-side menu to complete a task.

Participants were asked 20 questions using these two images:

1. You set an alarm on your laptop. Now you want to make sure that it will be loud enough to hear from across the room. Where would you look in the settings app to change the volume of the alarm?

2. You want to simultaneously be to able to view your web browser and a text editor where you are working on a document. You have two monitors and want to spread your work space across them. Which setting would you use to enable this function?

3. You want to have your Thunderbird account to be the email that opens whenever GNOME is asked to open an e-mail client. Which setting would you try to make Thunderbird your official email?

4. You would like to know which version of GNOME you are running so that you can see if you are up to date. Which setting would you try to find out this information?

5. You are about to watch a movie on your computer. The internal sound is too low so you get out a Bluetooth speaker. Where in the settings app would you go to connect the speaker to your computer?

6. You are working on a few design projects and want to adjust the way your monitor handles colors so that none of the image quality is lost. Which setting would you try first to change the default colors?

7. You would like to be able to access photos from GNOME shell search. Which setting would you try first to see if photos can be indexed in a search?

8. You are a very skilled typist. You notice that a lag on your keyboard and want to speed up the computers’ response to your key strokes. Where would you look in the settings app for a way to change the keyboard speed?

9. You are going out of town for a week and having a friend housesit for you. She wants to use your desktop computer while you are gone and you are happy to oblige. Your friend’s native language is Spanish while yours is English. You think it would be nice to set your computer to Spanish for her so that she can use it more efficiently. Where would you look to switch the language to Spanish?

10. You prefer watching movies with the video player VLC. You would like to set VLC as the video player that GNOME connects to automatically. Which setting would you try first to make the change?

11. In a few minutes you are expected to be in a two hour long lecture. You want to take notes on your laptop during the talk. Where in settings would you look to see how much battery charge you have left in order to know whether you have to run to your car and get the charger before the lecture starts?

12. You’re at the office and you want to show a work presentation with a projector. Which Setting would you try first to connect your laptop to the projector?

13. You’re preparing for a remote work meeting. You wish to set a password that you will give to your coworkers so they can see and control your screen during the meeting. With which setting would you first attempt to do this task?

14. You find that you don’t like only having 5 minutes of idleness before your screen locks on your system. Where in the settings would you go first to adjust the wait time from 5 to 10 minutes.

15. You just bought a new printer. Which setting would you choose first to connect the new printer to your computer over WiFi?

16. You are living abroad for several months in a country with a time zone 6 hours ahead of yours. Where in setting would you go to adjust the time zone to reflect your new location?

17. You are listening to music on your computer. At the same time, you are torrenting a number of files. You don’t want the notification sound to disrupt your music each time a file successfully downloads. Which setting would you try first to mute the volume of the notification?

18. You don’t want to have alert messages come up when your screen is locked. Which setting would you try first to turn off these messages?

19. You are tired of looking at the default lock screen image on your computer. Where would you look first in the settings application in order to replace the lock screen image with a picture of your family that you have saved on your computer?

20. You want to be able to scroll with two fingers on your touchpad. Which setting would you try first to enable this scrolling on your touchpad?

 Each tester was encouraged to think out loud throughout the test. When nothing besides menu selections were forthcoming, the testers were prompted with the question “What made you choose X setting to complete the task?” Usually the tester was able to give an reason for the choice or an association that they had with a particular panel. For many testers, the choice was made in less than a minute. This seemed in line with general user patterns. Most users won’t spend more than a minute making an initial selection in a settings application. After submitting the menu option representing the testers’ first choice click in completing each of the 20 tasks, testers were shown a third image.

Looking at this WIFI panel, testers were asked which of the four menu options represented their first choice click for the completion of the additional questions.

21. You have the WiFi password for your favorite local coffee shop saved on your computer. While you are there one day working on your laptop the person next to you asks if you know the WiFi password. Where in settings would you look to locate the saved password in order to share it?

22. You’re at the office trying to get work done. The WiFi has gone down three times already today and you are frustrated with waiting. You have an Ethernet cable plugged in and set up already, which panel would you try first in the Network settings to switch over from WiFi to Ethernet?

23. You’re not getting a great signal to the internet. Where in the Network settings app would you go to change which WiFi network you are connected to?

 

After selections were recorded for the final three questions, participants were asked additional follow up questions (these questions & responses will be addressed later on in the analysis). After the follow-up questions, the tests were completed.

Demographics

An attempt was made to get a variety of testing participants. Below are graphs representing gender, technical skill and preferred operating system breakdowns for the ten participants.

There are other demographics where specifics were not collected but are still worth mentioning. All participants were over 20 and under 40 years of age. Every participant claimed to use the internet daily. Only three of the participants had ever seen gnome before. No other discernible demographics were recorded.

Test Results

This usability test was unique in that it did not require participants to complete tasks with software. Testers were asked to select a setting from a drop-down menu that would represent the first place they would go to complete a task. Because the task was a simple selection, all participants completed ever task. Also based on the nature of the test, accuracy of responses could really only be put into a yes/no category, giving a higher failure rate than is perhaps normal in a usability test. One exception was made to the yes/no categorization of task completion. When a participant correctly chose the Devices or Details menu option, but failed to select the correct sub-menu option, the answer considered that partially correct. I could see arguments as to why it makes sense to record those as incorrect responses, but I wanted to have a way of differentiating when it was the menu or sub-menu that created confusion for the tester.

Below is a heat map representing the ten test subjects responses to the 23 tasks in order. The first column describes each task with a few keywords. The second column shows the correct responses to each task question. The ten subsequent columns represent the responses from each of the then testers color coded based on accuracy. Green represents a correct response, red correlates to an incorrect response, and orange is a correct selection of Devices or Details, but an incorrect sub-menu selection.

Key

Test 1

Test 2

Test 3

Test 4

Test 5

Test 6

Test 7

Test 8

Test 9

Test 10

Alarm Volume

Sound

Monitor Mirror

Devices, Displays

Default Email Client

Details, Default Applications

GNOME Version

Details, About

Bluetooth Connection

Bluetooth

Monitor Colors

Details, Color

Search Index

Search

Keyboard Speed

Devices, Keyboard

Switch Language

Region & Language

Set Default Video Player

Details, Default Applications

Battery Life

Power

Projector Connection

Devices, Displays

Screen Sharing

Sharing

Screen Lock Time

Privacy

Printer Connection

Devices, Printers

Time Zone

Date & Time

Mute Notifications

Sound

Alert Messages

Notifications

Lock Screen Image

Background

2 Finger Scrolling

Devices, Mouse & Keyboard

Find Saved WIFI Pass

WIFI

Ethernet Connection

Networks

Switch WIFI Networks

WIFI

I want to break down this heat map into three categories: tasks that were easy for everyone, tasks that were hard for everyone, tasks that were easy for some and hard for others.

Easy for everyone

#1. The volume settings for an alarm are located in the Sound panel in settings. This was an easy task for all but one participant. Tester generally didn’t think long before making this selection either. Testers said that it was an easy choice because alarms have a sound, and if you are trying to adjust that sound, then the Sound panel is the most obvious option. The tester who got this task wrong thought that an alarm is a notification and so the relevant adjustments for it would be in the Notifications panel.

#9. All participants knew to change the desktop language in the Region & Language panel. It was the only option that mentioned language and seemed the clear choice for everyone.

#11. All testers knew to look under the Power panel for access to info on the battery life of their laptop. The reasons given were related to past experience with laptop power settings (regardless of OS).

#15. Each tester successfully identified Devices as the first place to click when looking to set up a printer, and then Printers as the sub-menu option. This is interesting considering that there are other hardware questions where the testers did not correctly identify Devices as the most likely setting to choose from. So either all testers had viewed the Devices panel at least once before this question and remembered that Printers was a sub-option, or printers are more clearly associated in people’s minds as a device.

Overall, I think that testers did well with tasks that they have done before and with aspects of the settings panel that reflect the organization of other systems. The battery life task(#11) is a good example of this. Most if not all testers were familiar with checking the remaining battery charge, as it is a common issue for laptop batteries to be insufficient for users’ needs. Power is a term they were used to associating with the setting that deals with battery charge on their own systems. So, this task was no problem for any of them. All testers responded within seconds to this task. When prompted for a reason, it seemed harder for them to come up with one right away, as if the answer was obvious.

Hard for everyone

#6. None of the testers knew how to update color profiles in GNOME. I think this has to do with the fact that none of the participants had tried to complete that task on any system before. I think this task was a bit more obscure and no participant associated color with the Details menu option.

#14. None of the participants successfully identified Privacy as the settings panel to use to change the wait time on the lock screen option. Many of the testers chose the Power panel and several testers explained this by stating that they consider the lock screen to be a power saving mechanism. I think people have that association because perhaps on their systems the screen dimmer comes into effect at the same time that the screen locks.

I think it’s good to note that there were twice as many tasks that were across the board easy as there were difficult for everyone.

Tasks that proved somewhat difficult

#3. Several testers thought that they would set their e-mail client in Online Accounts.  Their reasoning was generally that an e-mail account qualifies as an an online account.  I think if they were able to see the Default Applications sub menu in Details without having to open a new panel some testers would have chosen the correct answer.  Still, I can understand why there was confusion over what type of action could be taken in the online accounts panel.  Even though the question was regarding the e-mail client and not the e-mail account itself, to many users the e-mail account is the important part and so online accounts would seem the better option.

#5 A lot of testers didn’t associate the Bluetooth panel with the task of connecting a speaker that uses the Bluetooth standard. Some testers did not know what Bluetooth meant.  Others just didn’t see it in the menu.

#16. Many of the testers thought they could change their time zone in Region & Language. Those who chose Region & Language did so because they said that they associated time zones with region. There didn’t seem to be any other option that addressed the location of the user, so they chose Region.

#17. Testers who didn’t choose Sound as the place to adjust the volume of a file download message thought that Notifications would be the place to go. They generally saw Notifications as the place to go for any type of tasks for a system message.

#22. Several testers had trouble figuring out where to switch over to an Ethernet connection. Of those who got it wrong most thought that it would be part of the WIFI sub panel.  Some of the testers had not dealt with Ethernet cables and so they didn’t know where to look, though everyone was familiar with the concept of Ethernet connection.

Similar to the analysis for what went well in the test, I think testers had trouble with tasks that they were unfamiliar with and those where the settings selections they are used to making were not available. Testers who had never connected a computer to a projector mentioned that they were just sort of shooting in the dark when choosing a setting. This is an unfortunate part of the testing process. It is hard to anticipate tasks that cover the breadth of settings you want to test and that at the same time seem easy and familiar enough for all testers.

When we are learning new interfaces we don’t generally figure things out in the first click, so some of the incorrect answers on this test need to be attributed to individual user behavior. The trick to usability testing is separating these isolated errors from the patterns that illuminate where the software is doing well and where it is falling short. Below in the recommendations section I have outlines the places where I think error patterns are strong enough that the design for these aspects should be reconsidered.

Follow-Up Questions

Directly after each usability test, I asked the participants the following questions.  Not every participant had a response to every questions, and some overlapped, but here are the general responses:

1. What is you general impression of GNOME setting?

Positive overall impressions:

It’s minimalist, tidy

It’s clear and easy to use, straightforward

It’s way better than the old settings application

Negative overall impressions:

It could overwhelm elderly people

Nothing about it was intuitive

2. Is there anything about the organization of the settings that you think should be different?

Labeling could improve, especially for Region & Language

Devices and Details options were too far down the drop-down menu

Bluetooth should be under devices

3. What (if anything) do you like about the layout of the settings?

One tester liked that there were fewer pathways to remember than in other settings apps.

Another tester said it was less cluttered than the previous version of GNOME settings

Another tester mentioned that they liked how Privacy and Sound were available without having to scroll down on the menu

One participant mentioned liking that there is a Network panel

Another said that they liked that all of the setting fit into three panels (main, Devices, Details)

Many testers echoed the fact that they liked the organization and clarity of the selections

4. It seemed like you were having trouble with x (when applicable). What would have made that task easier for you?

Testers didn’t really struggle to select something for any of the tasks so this follow-up question did not prove to be very useful

5. Would you want to use this settings application in place of your existing one?

4 people would use this instead of their existing settings applications

2 answered maybe

4 answered no

6. Do you have any other thoughts on the settings application?

One tester said that the WIFI panel looks like Mac software

7. Do you have any comments on the test in general?

One tester said they would have liked to see what all of the panels looked like.

Another said that it was a straightforward test

A participant said that they were fixated on getting the answers right

 

Recommendations For GNOME settings

1. Rename the Region & Language panel. Many testers assumed that they when they moved physical locations they would change their location and time zone in the Region & Language panel because the word region is in there. I think it would be suitable to call this panel Language. Users who want a specific regional language will still identify this choice under a panel called Language, and users with a different conception of the word region will not mistakenly go to Region & Language to change their location. Of the users who have previously used Gnome, the feedback from all three was that this settings panel always confuses them.

2. Move the Bluetooth panel next to Sound. I know that in the prior version of settings one could use both the Bluetooth panel and the Sound panel to connect to Bluetooth once the system recognizes the speaker, but as far as I understand it, pairing of Bluetooth happens in the Bluetooth panel. In the proposed changes to the Sound panel there is an option to test Bluetooth, but I didn’t see any to connect it. If the future settings panel is not going to have a way to pair Bluetooth in the Sound panel, I would suggest moving the Bluetooth panel to a more visible location. I believe that testers didn’t recognize the Bluetooth panel because of it’s location and association in the drop-down. I think were it near sound, people would see it as a option when considering Bluetooth tasks.

3. Consider changing the name of the Universal Access panel. Most testers were unfamiliar with this panel and several mentioned that the term universal access did not adequately describe the panel. No one gave a suggestion for a better name.

4. Definitely stick with the drop-down menu concept. Those who had used GNOME previously referred to the switch as a vast improvement. I also showed the current settings app to some of the testers and they too said that this update makes the settings application way easier to use.

5. Consider putting the Devices and Details panels higher on the list so the user doesn’t need to scroll down to access their sub menus.

 

Recommendations for future testing

1. Participants didn’t seem to listen very well to the test script. They all seemed to comprehend the specifics of the test better when I explained it to them in my own terms in the moment. I think some peoples’ brains shut off when they hear a script being read. I would in the future tell the testers about the test in my own words and then hand them the script to read. This way there is an element of consistency with the content, but also participants’ first introduction to the test won’t be scripted.

2. Explaining the think aloud protocol and giving multiple examples might help emphasize its importance to testers. Many testers forgot once they began the test and had to be prompted to talk out loud. Also, since they weren’t completing a task, testers had less to say in general. I mostly prompted them for reasons regarding their choices. Perhaps there is more useful lines of questioning for a prototype test. I could try to ask something more like “what does X settings make you think of?” “What does it do?” instead of “Why did you make this choice?” which makes the tester question herself instead of the setting.

3. Change the wording of task #7 because it uses the word search which is the name of the associated settings panel.

4. Separate task # 17 & #18. They are too similar and confused testers as to why they would be asked in succession.

5. I would add in a task that involves universal access. Testers were not familiar with this setting and it is an important one to be familiar with.

6. I would consider showing testers the GNOME desktop environment prior to performing the usability test. I think it would be good for them to see the whole picture. Also for question #7 I had to explain how an application search happens in GNOME for a number of testers. It would have been easier to show them.

7. I noticed that not all testers read through the menu items before the test began, and therefore failed to notice some of the settings panel options. They sometimes noticed a panel a few questions after it would have been useful to them. This made me consider whether it would be best for testers to read through all material before the test. Perhaps it is better not to ask participants to engage in any behavior they wouldn’t engage in at home using the software, but it is worth considering.

8. I think it would be useful to test a future iteration on users who are already familiar with GNOME. There was a sense that non-GNOME users didn’t feel an investment in a desktop environment they are not likely to use.

9. It was noticeable in the follow-up questions that testers wanted to know how the new settings app compared to the old one. I would consider showing all testers the previous version as part of the follow-up.

 

Thanks for reading.  As always I welcome questions, comments and suggestions for how to improve upon this usability test.