CHI paper accepted

glad to tell you, that our full paper “Intrducing Item Response Theory for Measuring the Usability Evaluation Process” is accepted for the CHI 2008 in Florence, Italy.

In this paper we treat the problem of heterogeneity in the usability evaluation process. Former stochastic models of Virzi, Nielsen&Landauer and Lewis employ the curve of diminishing returns P=1-(1-p)^n for predicting the outcome of an evaluation process. The problem with this model is the unique parameter p: It is unlikely that the probability of an individual evaluator detecting a usability defect is always the same.

As a solution we introduce the Rasch model for modeling the process. This model allows for a unique p_ij for each pair of evaluator and defect. With the Rasch model two basic parameters are introduced: the difficulty of defects and the skills of evaluators. The Rasch model allows for measuring both on a truly metric scale.

Admittedly, this is kind of very theoretical stuff (although I prefer the term “groundbreaking” ). But we tried hard to explain the basic concepts and procedures and describe three practical scenarios. At the time of the CHI, these scenarios will be available for download as runnable programs in the statistical computing environment R.