Affective multi-modal interfaces: the case of McGurk effect

This study is motivated by the increased need to understand human response to video-links, 3G telephony and avatars. We focus on response of participants to audiovisual presentations of talking heads, and examine the effect of noise and temporal misalignment of channels. We show that misalignment of audio and visual channels not only cause strange perception phenomena - the McGurk effect but also cause participants to apply extra mental effort, which is detectable from the physiological data collected. These data allow inferences to be drawn about the impact of the McGurk effect, thus providing indication of stress levels. This illuminates both to the mental and physical aspects of users interacting with multi-modal interfaces