• 90% of All Usability Testing is Useless

    Ninety percent of all usability testing performed on Web sites is useless. This is not to say that it doesn’t have a significant role to play in user experience design. When done right, usability testing will improve your Web site and your development process, but the current culture surrounding Web site usability testing is such that it rarely benefits the design. Worse, this misapplication can undermine the acceptance of this important technique throughout an organization.

    Usability testing has its background in Human-Computer Interaction (HCI), whose practitioners remain its most ardent supporters. This approach–a researcher, a participant, a list of tasks, and sometimes even a stopwatch–was developed for mainframe and then shrink-wrapped software as an evolution from even earlier task-based analysis techniques. It involves rigorous procedures that result in statistically oriented, graduate-degree-guaranteed results. When used to analyze Web-based user behavior, however, this approach suffers because the Web is not just software. User research on the Web needs to reflect this.

    Context Not Metrics

    The Web is a Frankenstein mix of creativity and code, inspiration and plumbing, form and function, art and science. Good Web design does well with all of these, and it’s often difficult to tell where each stops and the next begins. Trying to determine the time to complete a specific task is a fool’s errand. The real question is, why do users pause? What are they looking at? What are they thinking about? Did your navigation system fail them because of the categories you created, the words you used, or is it the placement of the navigation? Perhaps it’s because of inconsistencies across the site or poorly implemented CSS tabs. Traditional usability testing will not give you these answers.

    We need to abandon the idea that user testing on the Web is a quantitative process. Focusing on numbers to the exclusion of other data leaves researchers with nothing more than noticeably dubious statements like, “Design A is 5% more usable than design B” (or “90% of all usability testing is useless”). Instead, user research for the Web should delve into the qualitative aspects of design to understand how and why people respond to what has been created, and, more importantly, how to apply that insight to future work.

    Bring It In-House

    There is an outdated notion that usability research requires a blank slate–that outside observers will somehow comprehend what your users are doing in a more objective manner than yourself. But when you can’t rely on statistics, and you need to understand the context of your site’s use, all you have to go on is what you yourself can observe.

    So it’s time to get your hands dirty. Usability testing for the Web doesn’t require outside firms that hold themselves separate from the design process. Understanding context means embracing this research as part of the design process. It should be performed with the full participation of the design team, so that everyone can internalize the user’s perspective. Developing and leading usability tests should be a core competency owned by someone within your team (though many folks believe it ought not to be the original designer).

    And it shouldn’t be exclusive to this team. Individuals from across the organization should participate. Anyone who might have a stake in what’s being tested should be present for at least a part of the process. If the right people in your company see real users having real problems to which you can provide solutions, they’ll understand both your value to the organization and the need for the time and resources to make sure that good design happens.

    Usability Testing Is Design, Not QA

    Changing the approach to usability testing also changes what you should expect to get out of it. Rather than a validation done once before completing a product, this internal, qualitative usability testing is done earlier, more frequently, and as part of the design process–not separate from it.

    When viewed as a sort of quality assurance, as it classically is, usability testing becomes a late-stage “nice-to-have,” less important than getting the newest version of the Web site out the door. It usually results in a thick document that outlines everything that’s wrong with a Web application, including fundamental design issues that can’t be fixed in the few weeks left before launch. This isn’t the best way to effect positive change.

    When viewed as part of the design process, the outcome is very different. Some of the most inspired work I’ve seen has happened on whiteboards in the observation room while testing is going on several feet away. In some situations we’ve even made changes and applied them between rounds of testing in order to gauge the next user’s response to the new solution. From a statistical point of view, this approach is a disaster; from a design point of view, it’s a boon.

    More Economical

    Doing user testing within your design team means that it becomes less costly and less time-consuming–No more expensive consultants!–which means it can happen more frequently within your design process, on a schedule that works for your team. This approach allows usability to move from being a one-time, report-oriented process to an iterative, action-oriented one. Completed some basic wireframes? Put them in front of a couple of your users. Built a simple prototype? Put it in front of some users.

    This approach also allows you to always make changes when they’re least expensive. It’s much easier to change a whiteboard drawing than a wireframe; easier to change a wireframe than a prototype; and easier to change a prototype than a finished product.

    More Creative

    Finally, this approach to usability testing also helps you become a better designer. Designers sometimes worry that having to alter their work in response to user feedback will limit creativity, but when done as part of the design process, user research allows you to form new ideas and improve your design based on what people actually need. User research becomes a platform for inspirational, usable innovation. As a designer committed to improving your users’ experience, nothing could be more satisfying.

    Further Reading

    I’m not the first person to make the suggestion that usability testing change to accomodate the Web. Here are some excellent sources for guidance and inspiration around this topic:



  • Close
    Team Profile