In these
tough economic times it seems just about everyone is trying to do
more with less. When it comes to getting the most out of your
design investment, improving user experience can be a cheap way to
increase profits. The challenge is often figuring out what part of
the site needs the investment so you can get the biggest bang for
your buck. This week I’m going to talk about some inexpensive ways
to identify and target problem areas, using a variety of readily
available tools, specifically, remote user testing.Todd Elliott
Inexpensive Ways to Target Problem Areas
Until fairly recently, when designers wanted to test an idea or
design, they sought out an outside usability agency or, rented a
room, some expensive equipment and recruited users to come into an
artificial environment to participate in a usability study. In the
past few years, technology has become available that has brought
the cost and complexity of user testing down to a level where it is
available for anyone. In addition, it allows designers to be
exposed to users, in their native environment, not a sterile lab
across town. You can run a test in only a week for less than a few
hundred dollars.
There are numerous reasons remote user testing is valuable:
1. It’s cheap - most tests can be run for little more than the cost of your time and the incentive to the user.
2. It’s fast - done right, a study can be put together, executed and summarized in about a week. You also get immediate feedback that can be rolled directly into your designs.
3. Mediate conflict - User testing can be used to resolve internal design conflicts.
4. Test ideas - A quick user test can often help resolve a difficult design choice, like labeling or determining whether users can figure out how to use the new drag and drop system for adding items to their shopping cart.
5. Reduce release cycles - Instead of putting a new idea out in the wild and then having to scramble with a follow-up release, running a quick user test can often yield valuable insight that will save you time in the long run.
One of the best reasons to do remote research however, is that it offers you a window into the users environment. In addition to seeing how they tend to navigate a page you will see what kind of electronic interruptions occur, such as IM and email. You get to experience a little bit of their physical environment, is it noisy or quiet? Do they have kids competing for their attention? This is the kind of real-life testing environment that can yield unexpected and extremely valuable results.
One user test we conducted not long ago included a large segment of users that were relatively new parents, meaning there were often kids of all ages from in the background while we were testing. The product had been tested extensively in QA, and the usability tests were almost an afterthought. During the course of our testing, we discovered one very important issue: If the users walked away from the sign-up process for more than a couple of minutes, their session would expire and they would lose all of the information they had input. They had to start all over again. This only came up because one of the users we tested had a small child that demanded constant partial attention, and caused her to walk away from the computer for several minutes. It was one of the most valuable outcomes of the test. More importantly, it was a wholly unanticipated outcome that was only possible because we were testing the application.
Whether your client is an external one, or an internal group, invite them to watch the tests. Clients that listen in on user tests almost always come away with a fresh perspective on how users use their site.
There are many different types of remote user testing, the case above was the result of standard moderated remote usability study. However there are a number of other ways of collecting feedback from users without having them come into a lab.
Card Sorting
Card Sorting is a valuable tool in its own right to help inform the top level navigational structure of a site. It’s a common tool used by Information Architects everywhere, and is a cheap and reliable way to quickly get feedback on navigation and findability. This is done by… yes, sorting cards. Each card is labeled with a piece of information or a function. The cards are given to the users who are asked to sort them into logical groups. This helps uncover users expectations, create logical categories and help with naming of those categories. This type of data is very helpful early in the project when you’re trying to establish the overall navigation.
The best part is that tools are available online that make it easy to do this sort of testing remotely, which means you can design a card sorting study for dozens, even hundreds of users without the time consuming overhead of managing so many in person interviews.
Online tools like Optimalsort and Userzoom make it easier than ever to run card sorts. Userzoom is newer to the space and appears to offer a really nice output for easy analysis. I’m looking forward to using it at the next opportunity.
Remote Moderated Usability testing
In the case of moderated remote user testing, you can test specific ideas or solutions for a small audience: How long does it take users to find things? What do users think your site labels mean? Can users figure out how to get where they want to go? You can even test multiple designs on the fly, and test improvements as you go along.
It is also valuable to have your clients observe and listen in on the sessions using Adobe Connect or other meeting applications. Meeting applications like these are what allow you to see the users screen. As I mentioned earlier, the insight into the users experience can often foster empathy for the user, which is can be very valuable, if you need their support to improve the user experience.
There are a number of tools available for the lab environment that aren’t viable remotely. It’s sometimes desirable to do eye tracking or to record the users expressions while running the session. Though we don’t typically do this, there are more and more tools becoming available that add this type of functionality for fans of remote usability studies..
One example for recording users is Silverback which uses the cameras built into the Macbook and Macbook Pro to record the user as well as their screen. This is very handy if capturing the users expression while they are going through the test.
Recruiting For Your Session
It’s important to test your designs and solutions with the intended audience. For that reason, often the best audience to take advantage of is the one already coming to you. In order to recruit them, all you need is a survey and a survey tool. Your survey should be short and designed to capture a spread of your current audience. It’s useful to offer an incentive. Amazon gift certificates make great incentives, and are easy to manage because they can be emailed in bulk from Amazon.com.
There are a whole host of survey tools available for practically free out there. They include Survey Monkey, Wufoo.com, even Google Docs has a survey form option now, and User Experience experts, Bolt Peters, have a very nice, supported solution for a small fee. Once you create the survey, all you need to do is identify the page or pages you want to recruit from, then just put a short line of javascript in the code at the bottom of the page. Depending on how much traffic your site gets, you’ll usually want to select the pages that get the most traffic.
Once the survey is up and running, you can start scheduling a pool of candidates, or you can try to recruit them on the fly as they come to the website, and have them join the online meeting.
Designing A Script
One key to a good user test is a flexible script. If you force the user through a pre-determined set of tasks, you’ll risk miss out on those unexpected opportunities. Letting the user direct the study. I often like to start the study by asking the user what they came to the site to do originally. This is often referred to as their “passionate task” or the task that they came to the site to do. It can often give you surprising insight into users thought process.
That being said, I recommend taking the time beforehand to map out a series of questions. A good rule of thumb is to try to limit the test to 45 minutes. People often start to lose focus after that long. It doesn’t take many questions to fill up an hour, so it’s usually a good idea to pick about 3-4 main tasks.
A script should also include all the logistical information up front. Introduce yourself, and give them a little background on the study’s subject. You want to ask their consent, give them specific details on how long the test will last and how the software works. When you’re done, don’t forget to thank them and confirm the address you’ll be sending their incentive to.
Screen Sharing & Capturing the Output
The three keys to a successful test are:
o You can talk to the user
o You can see what they’re doing
o (optional) You can record both the audio and the video for later reference.
This is where the field has gotten pretty interesting lately. There are a number of inexpensive solutions out there for screen sharing. UserVue by Techsmith is still one of the best screen sharing and recording solutions out there, but the rumor is that they will be going away in 2010.
Both iChat and Skype offer screen sharing, and allow you to voice chat. If quick and dirty is your focus, Skype has a large installed user base and is already free. Other solutions include Adobe Connect, which has a little more up front cost, but also a whole range of uses beyond user testing.
If it’s important to record your session, Camtasia by Techsmith is one of the best solutions out there, and is now available for the Mac and the PC. And of course, Silverback, which I mentioned earlier, although it’s currently for Mac only.
Finally, I’d like to mention a few tips not covered in this overview. The first is that it’s very handy have someone sit in with you to take notes. It take a lot of practice, but it is possible to run a test and take notes at the same time. Secondly, save at least 15 minutes between each test to summarize what you heard. This will help you quickly include those insights in your next design.
There you have it, all the basic tools you need to run your own user tests. How about it readers, are there tools out there that you like that I haven’t mentioned? If so, let me know.
There are numerous reasons remote user testing is valuable:
1. It’s cheap - most tests can be run for little more than the cost of your time and the incentive to the user.
2. It’s fast - done right, a study can be put together, executed and summarized in about a week. You also get immediate feedback that can be rolled directly into your designs.
3. Mediate conflict - User testing can be used to resolve internal design conflicts.
4. Test ideas - A quick user test can often help resolve a difficult design choice, like labeling or determining whether users can figure out how to use the new drag and drop system for adding items to their shopping cart.
5. Reduce release cycles - Instead of putting a new idea out in the wild and then having to scramble with a follow-up release, running a quick user test can often yield valuable insight that will save you time in the long run.
One of the best reasons to do remote research however, is that it offers you a window into the users environment. In addition to seeing how they tend to navigate a page you will see what kind of electronic interruptions occur, such as IM and email. You get to experience a little bit of their physical environment, is it noisy or quiet? Do they have kids competing for their attention? This is the kind of real-life testing environment that can yield unexpected and extremely valuable results.
One user test we conducted not long ago included a large segment of users that were relatively new parents, meaning there were often kids of all ages from in the background while we were testing. The product had been tested extensively in QA, and the usability tests were almost an afterthought. During the course of our testing, we discovered one very important issue: If the users walked away from the sign-up process for more than a couple of minutes, their session would expire and they would lose all of the information they had input. They had to start all over again. This only came up because one of the users we tested had a small child that demanded constant partial attention, and caused her to walk away from the computer for several minutes. It was one of the most valuable outcomes of the test. More importantly, it was a wholly unanticipated outcome that was only possible because we were testing the application.
Whether your client is an external one, or an internal group, invite them to watch the tests. Clients that listen in on user tests almost always come away with a fresh perspective on how users use their site.
There are many different types of remote user testing, the case above was the result of standard moderated remote usability study. However there are a number of other ways of collecting feedback from users without having them come into a lab.
Card Sorting
Card Sorting is a valuable tool in its own right to help inform the top level navigational structure of a site. It’s a common tool used by Information Architects everywhere, and is a cheap and reliable way to quickly get feedback on navigation and findability. This is done by… yes, sorting cards. Each card is labeled with a piece of information or a function. The cards are given to the users who are asked to sort them into logical groups. This helps uncover users expectations, create logical categories and help with naming of those categories. This type of data is very helpful early in the project when you’re trying to establish the overall navigation.
The best part is that tools are available online that make it easy to do this sort of testing remotely, which means you can design a card sorting study for dozens, even hundreds of users without the time consuming overhead of managing so many in person interviews.
Online tools like Optimalsort and Userzoom make it easier than ever to run card sorts. Userzoom is newer to the space and appears to offer a really nice output for easy analysis. I’m looking forward to using it at the next opportunity.
Remote Moderated Usability testing
In the case of moderated remote user testing, you can test specific ideas or solutions for a small audience: How long does it take users to find things? What do users think your site labels mean? Can users figure out how to get where they want to go? You can even test multiple designs on the fly, and test improvements as you go along.
It is also valuable to have your clients observe and listen in on the sessions using Adobe Connect or other meeting applications. Meeting applications like these are what allow you to see the users screen. As I mentioned earlier, the insight into the users experience can often foster empathy for the user, which is can be very valuable, if you need their support to improve the user experience.
There are a number of tools available for the lab environment that aren’t viable remotely. It’s sometimes desirable to do eye tracking or to record the users expressions while running the session. Though we don’t typically do this, there are more and more tools becoming available that add this type of functionality for fans of remote usability studies..
One example for recording users is Silverback which uses the cameras built into the Macbook and Macbook Pro to record the user as well as their screen. This is very handy if capturing the users expression while they are going through the test.
Recruiting For Your Session
It’s important to test your designs and solutions with the intended audience. For that reason, often the best audience to take advantage of is the one already coming to you. In order to recruit them, all you need is a survey and a survey tool. Your survey should be short and designed to capture a spread of your current audience. It’s useful to offer an incentive. Amazon gift certificates make great incentives, and are easy to manage because they can be emailed in bulk from Amazon.com.
There are a whole host of survey tools available for practically free out there. They include Survey Monkey, Wufoo.com, even Google Docs has a survey form option now, and User Experience experts, Bolt Peters, have a very nice, supported solution for a small fee. Once you create the survey, all you need to do is identify the page or pages you want to recruit from, then just put a short line of javascript in the code at the bottom of the page. Depending on how much traffic your site gets, you’ll usually want to select the pages that get the most traffic.
Once the survey is up and running, you can start scheduling a pool of candidates, or you can try to recruit them on the fly as they come to the website, and have them join the online meeting.
Designing A Script
One key to a good user test is a flexible script. If you force the user through a pre-determined set of tasks, you’ll risk miss out on those unexpected opportunities. Letting the user direct the study. I often like to start the study by asking the user what they came to the site to do originally. This is often referred to as their “passionate task” or the task that they came to the site to do. It can often give you surprising insight into users thought process.
That being said, I recommend taking the time beforehand to map out a series of questions. A good rule of thumb is to try to limit the test to 45 minutes. People often start to lose focus after that long. It doesn’t take many questions to fill up an hour, so it’s usually a good idea to pick about 3-4 main tasks.
A script should also include all the logistical information up front. Introduce yourself, and give them a little background on the study’s subject. You want to ask their consent, give them specific details on how long the test will last and how the software works. When you’re done, don’t forget to thank them and confirm the address you’ll be sending their incentive to.
Screen Sharing & Capturing the Output
The three keys to a successful test are:
o You can talk to the user
o You can see what they’re doing
o (optional) You can record both the audio and the video for later reference.
This is where the field has gotten pretty interesting lately. There are a number of inexpensive solutions out there for screen sharing. UserVue by Techsmith is still one of the best screen sharing and recording solutions out there, but the rumor is that they will be going away in 2010.
Both iChat and Skype offer screen sharing, and allow you to voice chat. If quick and dirty is your focus, Skype has a large installed user base and is already free. Other solutions include Adobe Connect, which has a little more up front cost, but also a whole range of uses beyond user testing.
If it’s important to record your session, Camtasia by Techsmith is one of the best solutions out there, and is now available for the Mac and the PC. And of course, Silverback, which I mentioned earlier, although it’s currently for Mac only.
Finally, I’d like to mention a few tips not covered in this overview. The first is that it’s very handy have someone sit in with you to take notes. It take a lot of practice, but it is possible to run a test and take notes at the same time. Secondly, save at least 15 minutes between each test to summarize what you heard. This will help you quickly include those insights in your next design.
There you have it, all the basic tools you need to run your own user tests. How about it readers, are there tools out there that you like that I haven’t mentioned? If so, let me know.
Get the FeedAdaptive Path News
Did You Miss UX Week?
Then you really missed out - it was a great program, drawing
broadly from outside the companies, industries, and product types
that we typically expect to care about crafting a great user
experience. Last year’s workshops were such a hit that we had even
more hands-on goodness, and new for this year was our Doctor-Is-In
booth, a place to get advice, insight, or just a sympathetic ear
for your UX challenges.
See what the twitterverse had to say, and start making your plans for next year.
Where Else Can You Meet Us?
Jesse James
Garrett is speaking at Discuss & Discover in Munich,
October 20-22. The roster of
speakers looks awesome, so make it if you can.
Peter Merholz and Lane Becker will both be speaking at the Canadian User Experience Workshop in Banff November 12-14. Come hear about Lane’s experiences getting a customer satisfaction startup off the ground and running over the past year, and Peter’s thoughts about upgrading our mandate from user experience to customer experience!
Does Your Company Want to Offer Great Experiences?
Designing great experiences has become progressively more complex.
People expect more from companies and the technologies they employ.
They expect companies to coordinate across channels, product lines,
and services.
The design challenge becomes both deeper and broader, and we have been growing to meet it. If you worked with us years ago, you might be surprised at the capabilities and perspectives that we have brought into our studios. Why not contact us and see how we can help craft an amazing customer experience for your company?
Get the FeedSelections From Our Blog
-
Adaptive Path
Deliverable Quality: Avatar or Facade? -
Adaptive Path
Off the Desktop and Into the World -
Dan Harrelson
Rapid Prototyping Tools Revisited -
Andrew Crow
The Psychology of Achievements
What We’re Reading
-
AIGA.org
Case Study: NPR.org -
Ecofont
Take a Look at the Ecofont -
A List Apart
Testing Search for Relevancy and Precision -
Mashable
20+ Mac Apps to Increase Your Productivity
Subscribe to the Newsletter
Sign up to receive essays, appearance dates, and other news from Adaptive Path.
Recent Newsletters
- October 7, 2009
- September 22, 2009
- September 8, 2009
- August 25, 2009
- August 11, 2009
Where do great ideas come from?
At Adaptive Path, our ideas are driven by the work we do. We do consulting for user interface and user experience design, and offer conferences, training and education for UX designers.
From field ethnography, UI wireframes and task flows, to visual design and implementation, we do it and we teach it.
Learn more in our video, Adaptive Path in 2 ½ Minutes:
Want to know more about Adaptive Path? You should read more about our services or contact us to find out how we can help you!