Back in 1994 a chap called Rolf Molich (and friends) decided to see if us usability people were consistent in our ability to identify usability problems. He devised a simple test. Ask 9 usability experts to evaluate the usability of Microsoft’s free email service Hotmail and come up with a list of usability issues.
Rolf then aggregated all 310 usability problems uncovered from the experts and found something rather surprising.
Usability experts are not equal
It seems not all usability experts agree. Of the 310 usability issues found only two, that’s 2, were found by more than 6 experts. Of all the problems identified 75% were identified by only one expert. Usability it seems is in the eye of the beholder.
So what are we usability folk to do?
Rolf’s old partner is crime, Jakob Nielsen (Jakob and Rolf came up with the original usability heuristics) conducted a wonderful piece of research and found that 5 users will identify 70% of all usability problems. Notice the similarity in those numbers? 75% of problems are identified by 6 experts vs. 70% of problems found by 5 users.
The recommendation: run a usability test with real users. It need not be expensive, only 5 users are needed after all, it need not be exhaustive, a hour is more than enough.
Us usability people can help organise and interpret the results and most important of all make recommendations to fix the usability problems identified. A usability expert has techniques and theory but can’t always uncover usability issues any better than a user can.
Finally, be wary of people describing themselves “usability experts”, we now know there is no such thing.
Leave a Reply