from uie:
Usability Testing Best Practices: An Interview with Rolf Molich
rolf discusses the art (more) & science (less) of usability testing.
It depends. The number of users needed for web-testing depends on the goal of the test. If you have no goal, then anything (including nothing) will do.
goal: selling usability in your org = 3-4 users
goal: find "catastrophic probelms to drive an terative process" = 5-6 users
goal: "find all usability problems in an interface" = as many as you can get
Rolf has conducted tests with 50 users and not come up with an exhaustive list of problems and has also had 9 usability teams test Hotmail and of the 310 problems found, only 6 were identified by more than half the teams and 232 (75%) were reported only once. basically, usability testing is an art.
regarding heuristic evaluations:
Heuristic inspections are cheap, simple to explain, and deceptively simple to execute. However, I don’t use this method very often and I don’t recommend it to my clients. In my opinion, the idea that anyone can conduct a useful heuristic inspection after a crash course is rubbish. The results from my studies showed that inexperienced inspectors working on their own often produce disastrous amounts of “false alarms”.
Another problem is that heuristic inspection is based solely on opinions. No one has given me a good answer to the question that I’ve heard several times from disbelieving designers: “Why are your opinions better than mine?” I think that’s an excellent question, particularly knowing that users often prove me wrong whenever my heuristic predictions are put to a real usability test.
tips for creating usable usability reports:
Above all, a good usability report must be usable. The main recommendations I give clients for creating a usable usability report are:
- Keep it short.
No more than approximately 50 comments and 30 pages. It’s the job of the good usability professional to limit the comments to the ones that are really important.
- Provide a one-page executive summary on page 2.
Include the top three positive comments and the top three problems. Four of the nine CUE-2 teams did not include an executive summary in their reports.
- Include positive findings.
The ideal ratio between positive findings and problems is 1:1, but I have to admit that I rarely do better than 1:3. The CUE-2 teams ranged from no positive comments at all to an excellent ratio of 7:10.
- Classify the comments.
Distinguish between disasters, serious problems, minor problems, positive findings, bugs, and suggestions for improving the interface. Three of the nine CUE-2 teams did not classify their comments at all. The remaining six each invented their own classification scheme.
Reports are of course useful, but even a perfect report is useless if it doesn’t cause beneficial changes to the user interface. For example, good communication with the development team through effective consensus building is far more important than a good test report.
sample usability report
rolf's CUE (Comparative Usability Evaluation) reports are available online.