Jared Spool on User Research Methods

by Peter Merholz

July 15, 2005

Adaptive Path’s Peter Merholz recently talked to the founder of User Interface Engineering Jared Spool about user research.

The Business Behind the Research

Peter Merholz: So someone comes to you, and needs to better understand why their site is or is not working. How do you set up the approach that you’re going to take? How do you know what kinds of methods and what kinds of research will get the desirable results?

Jared Spool: Well, we start with trying to understand the business. One of the first questions I ask them is why do they think they need to do any research, which interestingly enough they almost never volunteer without prompting. What I’m looking for is the business rationale behind the research. You know, research costs money, there’s no way to do it without costing money, there’s expensive research, and there’s inexpensive research, but it always costs some money. So the question is: Why spend any money at all? So when we’re looking at working with a client, we want to understand why they want to spend the money. It’s almost always because they figure they’re going to make it back with the information they’ll gain, but we need to understand what that information could be. Doing research just for the sake of doing research is a losing proposition. We’re looking to enhance people’s business.

Testing Setups

PM: And so when is fidelity important, when is it not important? Can a lab environment truly control the conditions enough so that you have repeatable findings, you know that everything was similar enough that you can say that there’s truly trends emerging?

JS: I think lab testing is a tool. When you try to make a tool into a religious artifact, you tend to use it for things it’s inappropriate for. We don’t look at user testing as a religious artifact. And, for us, it’s not lab testing. In fact we don’t use a lab. I hate labs, personally. But we use laboratory setups frequently. But to me there’s a huge difference between the two.

PM: Talk a little about that. What’s the difference between a lab and a laboratory setup? Are the tests conducted at your offices?

JS: We almost never conduct them in our office. We usually conduct them in a place that is neutral ground and it’s chosen to be either close to the developers or close to the user.

PM: Do you use focus group facilities?

JS: No, no, we typically use a conference room at the client’s, or we’re a big fan of what are known as executive office suites, which are sort of these furnished offices that come with receptionists, and lounges, and coffee machines, all for a nice neat monthly fee. Interestingly enough, you can get one of those for a month for what you’ll pay for a focus group lab for a day.

PM: So you’re using these executive suites. How do you see your approach to something seemingly well understood as lab-style user testing? How is your approach different from what would be considered standard?

JS: We don’t have one-way mirrors, or television hook-ups, we typically have the observers in the room. The users are let completely in on the whole thing in terms of they understand why we’re there and what we’re doing. We have to keep the client anonymous, we do, but the users are partners with us. But other than that, it runs basically the same as any test. There’s a facilitator, a script, depending on the nature of the test, we use, for instance a lot of interview-based task techniques, so there isn’t so much a script as there is what we call a focus list.

Hiring Theater People

PM: You mentioned that when you see a resume with a theater background, you find that encouraging. What is the perspective of theater people that you find illuminating in doing this kind of work?

JS: Theater people have an interesting viewpoint of the world and it changes our viewpoint towards them.

Theater, particularly live theater, as opposed to film for example, is a process where you iterate, you see what works, you try it in rehearsal, and then you make changes, and then you try it again. So theater people inherently understand vast iterations, and moving toward an objective. Theater is also very much about an experience, so quality theater people understand the experience design in that regard, and they understand elements of the user design, such as the illusion, and subtlety, and the back-channel communication sort of stuff. Theater people all know how to work on a deadline because the curtain goes up at eight, and so you either have everything in place when the curtain goes up or you just make stuff up, but the curtain is going to go up. Theater people also understand the difference between on stage and backstage, which in a consulting practice or a research business is actually very important.

Common Mistakes

PM: What are typical mistakes, or misguided notions that you see when observing others engaged in user research?

JS: I think probably the biggest thing is not understanding the difference between observation, inference, opinion, and recommendation. Those four things are quite distinct and independent of each other. And if you don’t realize there’s a difference, you tend to muddle them up, and then things get very confusing.

PM: Any other mistakes, in terms of how tests are structured?

JS: There are many problems I see in terms of people not understanding the effects of testing affecting results. Here’s a simple example that we discovered. We were working with a client, testing a website that sold furniture. And the client had already done some testing. They had created some simple tasks, and one of the tasks was basically asking users to come to the site and buy a bookcase they might want to buy.

Every user in that study went to the search box, and typed in the word “bookcase,” immediately. So they were off trying to improve the hits on the word “bookcase,” and get the results better. We did a subsequent test, and instead of asking people to look for a bookcase, we gave them a slightly different task.

We said imagine you have 300 different paperback and hardcover books in cardboard boxes, all strewn through your living room. You need a way to organize this stuff such that you can find the stuff you like, and people who come over can be impressed with your collection. What do you do? People would go to the site, and then click on links. They’d click on furniture links; they’d type into the search engine storage systems, and then type in shelves. Not a single user in our phase of the study typed in the word bookcase. So by changing the description of the task, we got a completely different result.

To hear more thoughts from Jared Spool on innovative user research methods, join us on Day Three of User Experience Week.

A software developer and programmer, Jared Spool founded User Interface Engineering in 1988. He has more than 15 years of experience conducting usability evaluations on a variety of products, and is an expert in low-fidelity prototyping techniques.

Peter Merholz is Adaptive Path’s President, and a founder of Adaptive Path.