Multimodal literacy

This paper describes a novel framework for literature that presents the reader with a multimodal experience (visual, haptic and auditory). We present an initial prototype, extending an iPad with a surface vibration transducer for haptic feedback to augment the reading of a short story. Text provides the reader with verbal information, whereas other senses perceive the context through non-verbal cues. We use our own framework to create this storytelling across senses, integrating text, sound, animation and tactile sensation. We believe with a multi-modal reading experience, users will enjoy reading more, will feel more immersed and remember the narrative better.