Usability Test Results Too Diverse?
Rolf Molich has done a lot to make the usability profession what it is today. He keeps asking questions to what we thought would be common wisdom 5 years ago, and he keeps asking questions. Now, addressing the fact that usability test results can be very diverse (and often are).
“If you have 500 problems to choose from on a non-trivial website, and the usability test basically finds 40 problems more or less at random, then it’s no longer a surprise when you get these different problems.”
From Usability Professional’s Organisation Thumbnail on Rolf Molich, via Jared Spool. And the article continues:
What’s the solution? Interestingly, Rolf doesn’t see it as increasing the number of users, or other methodological changes. “The solution is a much more radical one,” he states firmly. “It’s prevention. Many of the usability problems we have seen should have not occurred on the website in the first place.”
Well, oddly enough, I remember that Rolf put it differently at the Danish usability conference a year ago. At that time his conclusion was to test with 50 users. This is what I wrote after his presentation:
Five users are not enough to catch 85% of the usability problems.
I asked him to elaborate on the latter, as Jakob Nielsen originally posted the article back in 2000 (why you only need to test with five users). Rolf Molich is states that you need to test with more than 50 users to catch all major usability problems. I asked how he could say that a usability problem was major if only one of 50 had a problem?
His point is that a test with five users never gets around all the corners of a website. Five users is not enough to find usability problems in rare error messages, rarely used functionality, etc.
That leaves us with two solutions on a common problem: The risc of finding the wrong results. What are your experiences here? Any thoughts or opinions?
Technorati Tags: usability, rolfmolich, rolf molich, jared spool, testing, diverse, jakob nielsen
February 18th, 2007 at 04:26 (GMT-1)
Evaluate with key flows and iterate. Its impossible to evaluate everything, but you can determine the parts of the product that will determine its success. For example, if people are not registering to use your product – this may have to be addressed before looking at other functions.
February 20th, 2007 at 02:21 (GMT-1)
“Many of the usability problems we have seen should have not occurred on the website in the first place” Agreed completely. Usability test your prototypes and you could probably just get by with 5 people. iRise, Axure, Lucid Spec, and GUI Design Gallery are some great higher fidelity prototyping tools out there.
Most of my 30 or so usability test have had 3-6 people. The one that had 12 ended up seeing the same usability problems after about 5 people (this had a diverse user group too).
February 28th, 2007 at 06:56 (GMT-1)
It always depends on the project. Sometimes 5 people is all you need because it isn’t a critical application roll out anyway. What I find really effective as an additional kind of testing is having the application reviewed by another designer that is not part of the project and given only the context a user would have. As you design your system, you come to accept some weird business rules or get too close, having another designer go through it can help you challenge yourself on those points.