Conducting International Usability

by Peter Merholz

March 12, 2003

As business on the Web matures, organizations increasingly pay attention to the first two Ws in WWW — World Wide.

Companies with international sales offices crave a distinct Web presence in each locale to demonstrate seriousness in that local market. Multi-national companies often use intranets to unify global teams. Maintaining global consistency requires centralizing these Web efforts (usually within corporate headquarters), yet this must accommodate distinct approaches to working which vary from region to region.

User testing is a valuable tool in such situations, but how does one conduct user tests internationally? This essay draws from my experience leading an international user testing project, and I hope you can learn from my mistakes and successes.

Hiring Decisions

Choosing a local consultant is perhaps the most important decision you’ll make. Professional mailing lists help greatly, and I found Frederic Fuchs, a Parisian usability engineer, by asking around on CHI-Web. You can also try market research organizations?the kind of folks who do focus groups. With a little training you can turn a focus group moderator into a user test facilitator, and market research firms typically have good databases of potential subjects. Ideally, you’ll work with someone bilingual—it can be awfully difficult to express the nuances you’re seeking in the test if you have to do so through an interpreter.

Depth and Timing

As someone accustomed to planning and running my own user tests, I find that international tests must be far more thorough and timely. For starters, communicating with people overseas, even in these days of fax and email, usually means twelve to twenty-four-hour lags.

The screener (Word .doc) and testing script (Word .doc) require precision and clarity to make sure specifics don’t get lost in translation. Instead of the one week it usually takes to find subjects, you’ll want to budget for two. All this time adds up, so begin planning at least a month before you want to test.

Testing Day

The testing days are when this all comes together. For a successful foreign language user test, you’ll need these people:

• A facilitator who is a native speaker in the local language, in the testing room
• An interpreter giving real-time translations in the observation room
• You, or some other person from the main project team, sitting in the observation room to watch and take notes
• Enough test subjects (six to eight, usually) that you can witness trends in behavior

You’ll also need this equipment:

• A video camera, preferably digital, in the testing room (with one video tape per session). Watch out for PAL/SECAM/NTSC video standard compatibility!
• A microphone and tape recorder (with one audio tape per session) to capture the interpreter’s work in the observation room
• Brief non-disclosure agreements in the local language
• A television in the observation room, showing what the camera is recording. (In my experience, a video camera and television set-up is more comfortable for test subjects than the ominous one-way mirror.)

Your Findings

Once the tests are complete, you can start to analyze your observations. Obviously, include the test facilitator in this process. Performing user tests overseas reminds you of how culturally rooted “usability” can be. While I dealt with typical usability concerns of clarity, ability to complete tasks, and show-stopping obstacles, what I found most intriguing were differences in perspective.

In the U.S., people critique color on an aesthetic basis — “I like this shade of blue,” “Those colors clash.” In Japan, the subjects didn’t discuss aesthetics, but meaning, “I wonder why they chose to use this green color.” Americans are used to their companies having global dominance; in France, we heard critiques that our client seemed like “another Anglo-Saxon company.”

Such observations might turn out to be the most valuable results of international user testing. Employing usability may not only improve a site’s interface, but help you and your company better understand your place in the world.

Peter Merholz is President and a founding partner at Adaptive Path. When he isn’t keeping the world safe for good user experiences, he’s writing about the San Francisco East Bay at the Beast Blog.