Crafting a User Research Plan

by Mike Kuniavsky

May 14, 2003

Every piece of user research is part of an ongoing research program, even if that program is informal. However, making a program formal provides a number of advantages: It gives you a set of goals, a schedule that stretches limited user-research resources, and results when they’re needed most. It also helps you avoid unnecessary, redundant, or hurried research.

Over the next few weeks, we’ll introduce the three major parts of a user research plan:

• Goals: Why you’re doing the research.
• Budget: How much it’s going to cost.
• Schedule: When to do the research.

This week, we’ll outline the goal-setting process and explain how to develop a workable budget for your user research plan.

Goals

Before you begin writing a research plan, determine corporate priorities and set research goals that can meet them. Hopefully everyone in an organization knows the general goals of a project (“Get more revenue,” “Bleed less money,” “Get more exposure for the brand,” etc.), but there will be a natural spectrum of understanding about what this means. Each department team invariably has a different method for figuring out how goals apply to them and for measuring their success at completing goals. Thus, you’ll need to do some internal discovery before you can make a complete research plan.

Identify Stakeholders and Their Needs

Stakeholders are the people in a department who represent what that department wants and needs from the project. The departments working on the project may not be the only ones affected by it, and the people you talk to don’t have to be top managers. Seek out groups that are directly affected and people who can provide accurate, honest representations of how your plans will impact their group. Ask them these three sets of questions:

1. In terms of what you do on a day-to-day basis, what are the goals of the product?
2. Are there ways that it’s not meeting those goals? If so, what are they?
3. Are there questions you want to have answered? If so, what are they? If you get answers, how will you use them?

Stakeholder answers may be contradictory or conflicting, but they will give you an idea of what aspects matter most and what questions your research needs to address.

So, for example, say you identify three main needs as stated by stakeholders:

• Help visitors use the search engine better and more often.
• Tell people about related services of which they may be unaware.
• Guide visitors to country-specific information from the worldwide homepage.

Now let’s take these needs and prioritize them.

Prioritize

If you’re not sure how to prioritize, try the following exercise, borrowed from Total Quality Management:

Make a column next to your list and label it “Importance.” Go down the list and rate each item on a scale of one to five, where five means that you must have an answer to this question in order for the project to succeed and one means it would be nice to have an answer.

Next, make a second column and label it “Severity.” This will reflect how bad the problem is today. Rate each item from one to five again, where five represents terrible problems affecting the bottom line right now, and one means information that would be good to know.

Now multiply the two entries in the two columns, and write the result next to them in a third column called “Priority.” Ordering the list by the third column amplifies the differences between priorities and often helps sort items that would otherwise appear similar.

Next, rewrite the needs as user-specific research questions or information to be gathered. Broaden narrow questions into general topics to get at the root causes of problems.

The questions should be simple so that they give the most bang for the buck. Pick one or two questions for each need that, when answered, will address the underlying goal.

For example: If visitors need to use the search engine better and more often, “How do people navigate the site, especially when they’re looking for something specific?”

To inform people about related services of which they may be unaware, “Where do people find related service information? Where do they expect it?” or “Where would inclusion of service information be intrusive?”

To guide visitors to country-specific information from the worldwide homepage, “What information is country-specific?” or “Where are non-U.S. visitors expecting to find that information? How much is language-specific versus country-specific?”

When you’re doing user research, don’t try to prove things right or wrong. Never create goals that seek to justify a position or reinforce a perspective. The process should aim to uncover what users really want and how they really are, not whether an opinion (yours or a stakeholder’s) is correct.

For more information about setting priorities, read Janice Fraser’s essay on the subject.

Budgets

The budget will be based on the cost of resources available to you, but it’ll probably come in three big chunks: people’s time (including your time and the research team’s time), recruiting and incentive costs, and equipment costs.

In my experience, useful times for calculating the duration of a qualitative user research project such as a usability test (including project management time and typical inefficiencies) are roughly as follows:

• Preparation for a single project — ten hours
• Recruiting and scheduling — two to three hours per person
• Contextual inquiry/task analysis — five hours per person
• Focus groups — three hours per group
• Usability tests — three hours per participant
• Analyzing contextual inquiry/task analysis — five hours per person
• Analyzing focus group results — four hours per group
• Analyzing usability tests — two hours per person
• Preparing a report for email delivery — twelve hours
• Preparing a one-hour presentation — six hours

The effort necessary for quantitative research, such as surveys and log analysis, will vary greatly based on the complexity of the task and the tools and expertise available. What a good statistician can do in a couple of hours can take days for someone with less training.

Incentive costs for interviewees vary, but as of Summer 2003 in San Francisco they tend to fall around $100 per person per ninety-minute session.

Likewise, equipment costs will vary based on how much documentation and space you require. It’s possible to spend $5,000 a day renting a usability lab for a series of highly documented usability tests, or testing can be free if you conduct a focus group in your conference room with a borrowed video camera and use the tape from last years’ forgettable holiday party.

Maintenance

Revise your research plan every time new knowledge comes in. Everything will change as your team and the company’s understanding of the user’s experience grows. The research goals, especially, should be re-evaluated, refined, and rewritten to take every piece of additional knowledge into account.

Since the research you do is likely to affect your understanding of a number of different research goals, you should consolidate your knowledge about the user experience whenever possible. A good way to do this is to create a mini-site for your intranet that contains all the reports and goals. Link each goal to the information that applies, and enter all problems into the development group’s bug-tracking software.

Eventually, you should have a set of interlinked documents that, together, make up a more-or-less complete picture of your user population.

This week’s essay is adapted from a chapter in Mike’s new book, Observing the User Experience: A Practitioner’s Guide to User Research, an indispensable book full of real-world guidelines for how research should be done.

Intruiged? Continue reading part II of Mike’s “Crafting a User Research Plan” »