Usability expert, me? Never.

Often when I'm introduced to a new project team I'm introduced as the usability expert, which makes me cringe a little. Why is that I hear you ask, well the evidence suggest that there is no such thing as a usability expert.

Our main conclusion is that our simple assumption that we are all doing the same and getting the same results in a usability test is plainly wrongBack in 1994 a chap called Rolf Molich (and friends) decided to see if us usability people were consistent in our ability to identify usability problems. He devised a simple test. Ask 9 usability experts to evaluate the usability of Microsoft’s free email service Hotmail and come up with a list of usability issues.

Rolf then aggregated all 310 usability problems uncovered from the experts and found something rather surprising.

Usability experts are not equal

It seems not all usability experts agree. Of the 310 usability issues found only two, that’s 2, were found by more than 6 experts. Of all the problems identified 75% were identified by only one expert. Usability it seems is in the eye of the beholder.

So what are we usability folk to do?

Rolf’s old partner is crime, Jakob Nielsen (Jakob and Rolf came up with the original usability heuristics) conducted a wonderful piece of research and found that 5 users will identify 70% of all usability problems. Notice the similarity in those numbers? 75% of problems are identified by 6 experts vs. 70% of problems found by 5 users.

The recommendation: run a usability test with real users. It need not be expensive, only 5 users are needed after all, it need not be exhaustive, a hour is more than enough.

Us usability people can help organise and interpret the results and most important of all make recommendations to fix the usability problems identified. A usability expert has techniques and theory but can’t always uncover usability issues any better than a user can.

Finally, be wary of people describing themselves “usability experts”, we now know there is no such thing.

8 Responses to “Usability expert, me? Never.”

  1. Harry Brignull

    Hi Joe

    I’d love to see the Molich 1994 study run again. It should be easily replicable, and we can find out if things moved on a lot since then.

    Would you be interested in participating in a re-run of this study? I’m sure we can get 7 other companies interested without much trouble!

Leave a Reply