Enhancing Non-Speech Information Communicated in Closed Captioning Through Critical Design

The communication of non-speech information (NSI) in closed captioning is essential in providing full access and increased enjoyment of video content, particularly for d/Deaf or Hard-of-Hearing (DHH) viewers. We identified the limitations and frustrations of current NSI captioning through needfinding interviews and then employed a critical design framework to develop a medium-fidelity audio-reactive animated overlay prototype to explore the opinions and values of DHH users regarding NSI communication through surveys and interviews. The results show that current NSI captioning strategies lack consistency across platforms, adequate temporal information, clarity in emotional conveyance, and lack customization options. The study suggests that novel sound communication technologies show promise in enhancing certain aspects NSI communication and that there is a strong desire to move beyond the current, inflexible format of single-track closed captions for NSI communication.