Mar 30, 2021 | by Budi Tanrim
Inspecting interface design — HE vs. UT
In 2009, a group of researchers puts a test to heuristic evaluation and usability testing. They asked one big question: Which one is better between HE and UT?
Which method will 1) cover more problems, and 2) which method will identify severe problems that potentially stop users from completing their tasks? They divided the participants into two groups:
- The first group is the “UT group,” which consists of 12 novice users recruited for usability testing.
- The second group is the “HE group,” which consists of 9 expert evaluators to perform the heuristic evaluation.
Both groups analyzed the same interface. There are four commercial websites they analyzed.
On average, the UT group identified 30% of the total problem, and the HE group identified 60% of the entire problem. It means, by performing the heuristic evaluation, the group could actually identify more problems. Dozens of studies had reproduced the same result as well. Some found that heuristic evaluation found 72% of problems while usability testing only found 10%.
That is big.
But we can’t stop there.
Even if HE can identify more problems, does it identify some serious problems? Or most of them are cosmetic problems, which meaningless problems that wouldn’t stop users from completing their tasks.
Many studies found that HE would either found the most critical problems or almost identical compared to UT. However, some studies produced a different result. They found HE would find only half critical problems compared to UT. This inconsistency means that HE could either be as good as UT or can be worse than UT.
These findings are essential for us as design practitioners. It means, if the heuristic is done correctly, we could found some critical problems.
We can save time.
We can iterate more.
We can improve our work.
This leads us to an important question: How would we perform the heuristic evaluation then?
Footnotes
- Tan et al., Web evaluation: Heuristic evaluation vs. user testing (2009)
- Doubleday, A., Ryan, M., Springett, M,. Sutcliffe, A.: A comparison of usability techniques for evaluating design (1997).
- Kock E.D. et al., Usability evaluation methods: mind the gaps (2009)
- Jeffries et al., User interface evaluation in the real world: a comparison of four techniques (1991)
Hi, this blog is no longer active. I move to substack. You can subscribe below or go to newsletter.buditanrim.co
