They say, if it moves measure it. I’ve been finding myself thinking a lot about how to measure UX lately—mainly to realize that I halfway don’t believe it’s possible. My gripe is that even if you can regularly track and look at numbers (itself no small feat), it’s hard to know what they mean exactly. And yet, without understanding the quantifiable effects, it’s very difficult to know or show what impact user experience is having in an organization. And that’s important.
So I guess it’s time for me to learn more about UX metrics. Happily, others in our field have been some doing interesting thinking in this area. I chatted recently with Richard Dalton, who manages a UX team at Vanguard and will be speaking about measuring user experience at Adaptive Path’s MX conference on March 6th.
Leah: At MX you’ll be talking about measuring UX, which I assume means that you believe you can measure UX. What leads you to make such a bold and audacious claim, where so many others have failed?
Richard: I think we’re having some success where others have struggled for two reasons: 1) We’ve developed a framework of tasks that we use to define what the capabilities in our user experience are trying to achieve -
we use this to identify *what* we should be measuring, and 2) We’re measuring our UX at a much more granular level than i’ve seen many others try it - that gives us some understanding of *why* our experiences are succeeding or failing.
Leah: So, you’ve done the work to come up with some interesting measurements. Good numbers. What do you do with them? How do they change the way people think? In your experience, how does data change designs?
Richard: Numbers and data can make UX professionals uncomfortable - we’ve all heard the stories of Google’s 41 shades of blue - but we treat the data as just another piece of research. The quantitative data needs to be balanced against qualitative feedback and against the experience of UX professionals. We’ve found this especially important because interpreting the quantitative data is not an easy task, you need to constantly guard against making false assumptions. When you can view the data with an open mind though, we find that its a great eye-opener - forcing us to examine design possibilities that we hadn’t previously considered.
Leah: In a recent discussion on the Information Architecture Institute discussion list, you talked about measuring drivers, not outcomes. What are drivers, and what makes them so special?
Richard: In a nutshell, while ‘outcome measures’ tell you *if* your experience is succeeding or failing, ‘driver measures’ tell you *why*. Drivers represent the ‘cause’ to the Outcomes ‘effect’ - they’re generally much lower level measures that are specific to your experience, whereas outcomes are higher level and a little more generic.
Leah: In your presentation on dealing with a UX mid-life crisis, you talk a bit
about the emotional considerations for users. Do you measure emotion in any
way and use that to drive outcomes?
Richard: We haven’t tried to measured emotion yet - mainly because it would very likely require users to self-report their emotions and although we do sometimes resort to actually asking users questions in order to get data for measures we try and reserve it for a last resort (we prefer behavioral data). We did do an AB test a while ago which
included a section on how the users perceived the look and feel of the interface (we asked users to pick from a list of adjectives) - that’s probably the closest we’ve come (so far) to measuring emotions.
Leah: If someone is interested in starting to measure UX, what should they do? How do you get started?
Richard: You mean in addition to attending MX and seeing my presentation? ; - ) I have three specific suggestions:
1) Figure out your user tasks and use them as a basis of what to measure
2) Make sure you have a mix of driver and outcome measures
3) Try it - pick something small, start, and be prepared to learn (probably from failure) as you go!
Richard will be speaking at MX (as will I). Up until a few days ago, the event was sold out. But the magicians on the Adaptive Path Events team have secured additional space, which means there are now additional spots available.
If you’re thinking of attending MX, you can use the code “BLOG” for 10% off the price of registration.