Understanding and Supporting Modality Choices

By Anthony Jameson and Per Ola Kristensson (2017)

In S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, & A. Krüger (Hrsg.), The handbook of multimodal-multisensor interfaces (vol. 1) (S. 201–238). New York: Association for Computing Machinery and Morgan & Claypool.

Abstract

(From the Introduction:)

One of the characteristic benefits of multimodal-multisensor processing is that it gives users more freedom of choice than they would otherwise have. The most central type of choice concerns the use of input modalities: When performing a particular task with a multimodal system, users will often have a set of two or more input modalities to choose from. They may choose to employ just one of these; or they may use two or more modalities simultaneously or in alternation. In general we can say that they must somehow choose some combination of modalities from the available set.

It has often been pointed out in the multimodal literature that this freedom of choice is a major benefit of multimodality in that it gives users the flexibility to deal with a wide range of needs and situations. But this additional freedom of choice can be beneficial to users only if they are able to make reasonably good modality choices. If they make choices that actually lead to less satisfactory results than they could get with a system that offered more limited interaction possibilities, then multimodality will be a drawback rather than an advantage.

We begin this chapter by summarizing what can be learned from the research literature about the modality choices that people make, concluding by arguing that there is a need to focus more on the processes by which people arrive at such choices and on strategies for helping them to choose more effectively. In the subsequent sections, we introduce relevant psychology-based models for this purpose and illustrate their applicability to the problem of modality choice.

Note