Birdbox: Exploring the User Experience of Crossmodal, Multisensory Data Representations

We contribute to an improved understanding of how physical multisensory data representations are experienced and how specific modalities affect the user experience (UX). We investigate how people make sense of Birdbox, a crossmodal data representation that employs combined haptic-audio, audio-visual, or visual-haptic output for data about birds. Findings indicate that participants preferred haptic output for the bodily experience it triggered. Participants further created their own mappings between data and modality; haptic was mapped to aggression, and audio to speed. Especially with (soft) haptic output, Birdbox was experienced as a living entity. This can also be seen in participants’ bodily interactions, holding Birdbox as if it were a small bird. We contribute to a better understanding of the UX of different modalities in multisensory data representations, highlight strengths of the haptic modality, and of metaphorical understandings of modalities.