Inexpensive Ways to Target Problem Areas

by Todd Elliott

October 6, 2009

Until fairly recently, when designers wanted to test an idea or design, they sought out an outside usability agency or, rented a room, some expensive equipment and recruited users to come into an artificial environment to participate in a usability study. In the past few years, technology has become available that has brought the cost and complexity of user testing down to a level where it is available for anyone. In addition, it allows designers to be exposed to users, in their native environment, not a sterile lab across town. You can run a test in only a week for less than a few hundred dollars.

There are numerous reasons remote user testing is valuable:

1. It’s cheap - most tests can be run for little more than the cost of your time and the incentive to the user.

2. It’s fast - done right, a study can be put together, executed and summarized in about a week. You also get immediate feedback that can be rolled directly into your designs.

3. Mediate conflict - User testing can be used to resolve internal design conflicts.

4. Test ideas - A quick user test can often help resolve a difficult design choice, like labeling or determining whether users can figure out how to use the new drag and drop system for adding items to their shopping cart.

5. Reduce release cycles - Instead of putting a new idea out in the wild and then having to scramble with a follow-up release, running a quick user test can often yield valuable insight that will save you time in the long run.

One of the best reasons to do remote research however, is that it offers you a window into the users environment. In addition to seeing how they tend to navigate a page you will see what kind of electronic interruptions occur, such as IM and email. You get to experience a little bit of their physical environment, is it noisy or quiet? Do they have kids competing for their attention? This is the kind of real-life testing environment that can yield unexpected and extremely valuable results.

One user test we conducted not long ago included a large segment of users that were relatively new parents, meaning there were often kids of all ages from in the background while we were testing. The product had been tested extensively in QA, and the usability tests were almost an afterthought. During the course of our testing, we discovered one very important issue: If the users walked away from the sign-up process for more than a couple of minutes, their session would expire and they would lose all of the information they had input. They had to start all over again. This only came up because one of the users we tested had a small child that demanded constant partial attention, and caused her to walk away from the computer for several minutes. It was one of the most valuable outcomes of the test. More importantly, it was a wholly unanticipated outcome that was only possible because we were testing the application.

Whether your client is an external one, or an internal group, invite them to watch the tests. Clients that listen in on user tests almost always come away with a fresh perspective on how users use their site.

There are many different types of remote user testing, the case above was the result of standard moderated remote usability study. However there are a number of other ways of collecting feedback from users without having them come into a lab.

Card Sorting

Card Sorting is a valuable tool in its own right to help inform the top level navigational structure of a site. It’s a common tool used by Information Architects everywhere, and is a cheap and reliable way to quickly get feedback on navigation and findability. This is done by… yes, sorting cards. Each card is labeled with a piece of information or a function. The cards are given to the users who are asked to sort them into logical groups. This helps uncover users expectations, create logical categories and help with naming of those categories. This type of data is very helpful early in the project when you’re trying to establish the overall navigation.

The best part is that tools are available online that make it easy to do this sort of testing remotely, which means you can design a card sorting study for dozens, even hundreds of users without the time consuming overhead of managing so many in person interviews.

Online tools like Optimalsort and Userzoom make it easier than ever to run card sorts. Userzoom is newer to the space and appears to offer a really nice output for easy analysis. I’m looking forward to using it at the next opportunity.

Remote Moderated Usability testing

In the case of moderated remote user testing, you can test specific ideas or solutions for a small audience: How long does it take users to find things? What do users think your site labels mean? Can users figure out how to get where they want to go? You can even test multiple designs on the fly, and test improvements as you go along.

It is also valuable to have your clients observe and listen in on the sessions using Adobe Connect or other meeting applications. Meeting applications like these are what allow you to see the users screen. As I mentioned earlier, the insight into the users experience can often foster empathy for the user, which is can be very valuable, if you need their support to improve the user experience.

There are a number of tools available for the lab environment that aren’t viable remotely. It’s sometimes desirable to do eye tracking or to record the users expressions while running the session. Though we don’t typically do this, there are more and more tools becoming available that add this type of functionality for fans of remote usability studies.

One example for recording users is Silverback which uses the cameras built into the Macbook and Macbook Pro to record the user as well as their screen. This is very handy if capturing the users expression while they are going through the test.

Recruiting For Your Session

It’s important to test your designs and solutions with the intended audience. For that reason, often the best audience to take advantage of is the one already coming to you. In order to recruit them, all you need is a survey and a survey tool. Your survey should be short and designed to capture a spread of your current audience. It’s useful to offer an incentive. Amazon gift certificates make great incentives, and are easy to manage because they can be emailed in bulk from Amazon.com.

There are a whole host of survey tools available for practically free out there. They include Survey Monkey, Wufoo.com, even Google Docs has a survey form option now, and User Experience experts, Bolt Peters , have a very nice, supported solution for a small fee. Once you create the survey, all you need to do is identify the page or pages you want to recruit from, then just put a short line of javascript in the code at the bottom of the page. Depending on how much traffic your site gets, you’ll usually want to select the pages that get the most traffic.

Once the survey is up and running, you can start scheduling a pool of candidates, or you can try to recruit them on the fly as they come to the website, and have them join the online meeting.

Designing A Script

One key to a good user test is a flexible script. If you force the user through a pre-determined set of tasks, you’ll risk miss out on those unexpected opportunities. Letting the user direct the study. I often like to start the study by asking the user what they came to the site to do originally. This is often referred to as their “passionate task” or the task that they came to the site to do. It can often give you surprising insight into users thought process.

That being said, I recommend taking the time beforehand to map out a series of questions. A good rule of thumb is to try to limit the test to 45 minutes. People often start to lose focus after that long. It doesn’t take many questions to fill up an hour, so it’s usually a good idea to pick about 3-4 main tasks.

A script should also include all the logistical information up front. Introduce yourself, and give them a little background on the study’s subject. You want to ask their consent, give them specific details on how long the test will last and how the software works. When you’re done, don’t forget to thank them and confirm the address you’ll be sending their incentive to.

Screen Sharing & Capturing the Output

The three keys to a successful test are:

• You can talk to the user
• You can see what they’re doing
• You can record both the audio and the video for later reference (optional).

This is where the field has gotten pretty interesting lately. There are a number of inexpensive solutions out there for screen sharing. UserVue by Techsmith is still one of the best screen sharing and recording solutions out there, but the rumor is that they will be going away in 2010.

Both iChat and Skype offer screen sharing, and allow you to voice chat. If quick and dirty is your focus, Skype has a large installed user base and is already free. Other solutions include Adobe Connect, which has a little more up front cost, but also a whole range of uses beyond user testing.

If it’s important to record your session, Camtasia by Techsmith is one of the best solutions out there, and is now available for the Mac and the PC. And of course, Silverback, which I mentioned earlier, although it’s currently for Mac only.

Finally, I’d like to mention a few tips not covered in this overview. The first is that it’s very handy have someone sit in with you to take notes. It take a lot of practice, but it is possible to run a test and take notes at the same time. Secondly, save at least 15 minutes between each test to summarize what you heard. This will help you quickly include those insights in your next design.

There you have it, all the basic tools you need to run your own user tests. How about it readers, are there tools out there that you like that I haven’t mentioned? If so, let me know.

Todd Elliott is a project manager for Adaptive Path. Todd’s experience spans online advertising, product feature development, and editorial management, with expertise in usability testing and product quality assurance.