“A-ha Moments?”

I always get a little miffed when I hear user-experience folk describe their data analysis process as looking for “a-ha moments.” It seems like an evasive answer to a simple process question. But more importantly, it leaves one with the impression that coming up with research insights is an experience of epiphany, like Newton sitting under the apple tree discovering gravity.

In a recent talk about his current book, The Myths of Innovation, Scott Berkun emphasized that humans love stories of epiphanies because they diminish the sweat-effort, failure, and plain old hard work that goes into coming up with something that is truly innovative. He explained how the story of Newton sitting under the apple tree conveniently de-emphasizes his 15 years of dedication and study to the subject.

Similarly, “a-ha” moments seem like an easy epiphany story; and that at the end of all your field interviews, research insights will come to you like apples falling from a tree. In truth, the end of field interviews leaves most teams with a mountain of videotapes, transcripts, and digital photos—and the daunting task of translating it all into something that is insightful, thought-provoking, and actionable.

So what do you do? What process do you follow? I’ve asked user experience industry folks these very questions. They invariably described their process as “deep thinking” and “marinating in the data” or “cranking it out.” Descriptive? Yes. Helpful? Not really.

A recent project allowed me to reflect on my own data analysis. I found there are three general phases that I follow when climbing the mountain of data in search of research insights.

What We Saw / Heard

Part of what makes data analysis scary is: 1) There is a lot of data and 2) It is all in disparate forms. Slogging through all that data can feel intimidating because there is simply so much stuff. I often relate this phase of analysis to the television show Clean Sweep. It’s a show about people who have a house so crammed with crap that it’s unlivable. Professional organizers come in and help them clean up their house and get their act together. Just like a house crammed with clutter, at the end of all the field interviews, there is a mess of videotapes, transcripts, and photos that have to be culled through and made sense of.

The first thing all guests on Clean Sweep do is unload the contents of their home onto their front lawn. This is effective for two reasons: 1) It forces people to face the madness and take a good hard look at all their stuff and 2) It presents “the stuff” in an apples-to-apples format. Similarly, because the data from field research is often in disparate forms (video tapes, transcripts, digital photos, etc.), it’s important to present data in an apples-to-apples format by transforming it into something tangible and visible. Only then can you take a good hard look at everything that’s been collected and start making choices about what is important.

Making Your Data Visible—Memoing

Data analysis starts during the debriefing of every interview. I don’t take notes during an interview so I usually hit a coffee shop after to capture my thoughts. A cup of coffee later, I have a stack of Post-it Notes filled with quotes and keywords. Each Post-it gets a thought—things like “called customer service everyday for a week” or “pissed off when blood sugar is high or low = what am I doing wrong?” If I don’t go on an interview, I try to watch the videotapes. Whenever a participant says something I think is interesting or important, I write it down on a Post-it Note and record the timestamp from the video, which makes video-editing clips much easier. In Grounded Theory, this is known as “memoing.”

The “Why” of Post-it Notes

I know of a few people who like to work digitally and enter all their memo information in tools like Excel or Word. This perplexes me. It seems akin to pounding a nail into a wall with a screwdriver—the wrong tool for the job. The goal for ethnographic field data analysis is to build a shared understanding with a team and that requires collaboration. Digital memos prove really challenging to share amongst a team. Unlike data trapped in an Excel spreadsheet, the tactile quality of Post-it Notes is easy for people to scan and engage. Memoing on Post-its also provides a shared sense of ownership in the process, making it easy to build upon the ideas of others. Of course, there are downsides to Post-its: you can’t email them and they are difficult to share with teams working remotely, but the benefits of easy collaboration far outweigh the trade-offs.

Participant Boards

It’s very easy to generate more notes than you know what to do with. I use the participants themselves as organizing principles for all the notes. I dedicate a 36” x 70” sheet of cardboard to each participant in the study, filling it with the Post-it Notes and photos from the interviews. Each board is like a small homage to a participant, making it easy to remember the important things they have shared with the team.

Why It Matters

This juncture of the analysis process is all about editing to relevance and expressing a point of view. The goal is to come up with design implications with accompanying insights that both tell the story of your research and recommends actions you think someone should take.

The most challenging part is editing. Typically at this phase, there are more ideas and themes than you know what to do with. Unfortunately not all are as relevant to the project as others. Repeating to the team and myself the phrase: “Why does this matter?” can bring perspective. Another technique is to share themes with people who know nothing about the project. Sharing the story of each theme helps determine the ones that resonate and the ones that don’t. Fresh eyes also help in honing the “talking points” and crafting the story or its emerging insights.

Emerging Insights

Writing the story of the theme is a subtle dance between exposing data found in the field and interpreting what you think it means. In big headline-style type, I use the Post-it Notes, frameworks, photos and clusters of data to craft this message in three to five presentation slides. I call these the Emerging Insights. They illustrate your thought process, and function as evidence for your implications.

Implications

Each theme or series of Emerging Insights should have an implication—a directive that clearly tells the client what to do. Implications tell people why the research matters—how it is relevant to them. I found the most effective implications to be the ones that reframe the problem in an interesting and thought-provoking way. The most boring and uninspired research findings are ones that either tell people what they already know, e.g., people have powerful relationships with their mobile phones. Or they don’t tell people what they should do because they’re missing an implication or a point of view.

Just like good design, good research findings express a point of view. Don’t be afraid to have a point of view; it’s an essential part of making sure the research is actionable. You’ve gone out into the field and talked to real-live people. Therefore, the work you do should tell their story and advise your client of the actions that should be taken next.

Appropriateness

Different people will have different approaches to data analysis, and some approaches may be better suited to particular challenges. My point is, there is always a process. Despite claims that “a-ha” moments are the basis of how they analyze field data, I would bet dollars to doughnuts they have a process too. They’re just not doing a very good job at communicating it.

Process isn’t a panacea. It doesn’t make that mountain of data at the end of field interviews any smaller. However, having a process that you can clearly communicate is a powerful skill. It allows people to engage with your work on a deeper level because you can show both the insights and implications you’ve discovered as well as the work you’ve done to get there. Process gives your team a road map to follow, making collaboration easier. In addition, it builds confidence within your organization for your effort, ensuring the hard work you’ve done won’t be dismissed.