MeMo: Towards Automatic Usability Evaluation of Spoken Dialogue Services by User Error Simulations

By Sebastian Möller, Roman Englert, Klaus Engelbrecht, Verena Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Reithinger (2006)

Proceedings of INTERSPEECH 2006, the Ninth International Conference on Spoken Language Processing, Pittsburgh, PA.

Abstract

Proper usability evaluations of spoken dialogue systems are costly and cumbersome to carry out. In this paper, we present a new approach for facilitating usability evaluations which is based on user error simulations. The idea is to replace real users with simulations derived from empirical observations of users’ erroneous behavior. The simulated errors must cover both system-driven errors (e.g., due to poor speech recognition) as well as conceptual errors and slips of the user, because neither alone is predictive of perceived usability. The simulation is integrated into a workbench which produces reports of typical and rare errors, and which allows usability ratings to be predicted. If successful, this workbench will help designers in making choices between system versions and lower testing costs at early phases of development. Challenges to the approach are discussed and solutions proposed.

Citations

• Google Scholar page

Download

Full Publication:  

BibTeX entry

@inproceedings{MoellerEE+06,
  year = {2006},
  author = {{M\”{o}ller}, Sebastian and
            {Englert}, Roman and
            {Engelbrecht}, Klaus and
            {Hafner}, Verena and
            {Jameson}, Anthony and
            {Oulasvirta}, Antti and
            {Raake}, Alexander and
            {Reithinger}, Norbert},
  title = {{M}e{M}o: Towards Automatic Usability Evaluation of Spoken
    Dialogue Services by User Error Simulations},
  booktitle = {{Proceedings of INTERSPEECH 2006, the Ninth International
    Conference on Spoken Language Processing}},
  address = {Pittsburgh, PA}}