人机交互
虚拟现实
计算机科学
用户体验设计
系统设计
多媒体
用户界面
沉浸式技术
增强现实
计算机中介现实
系统体系结构
计算机图形学(图像)
沉浸式(数学)
虚拟机
用户界面设计
作者
SIYUAN ZHAO,Xiangkun Liu
摘要
Gesture- and voice-based interaction in VR has shown promise in various applications; however, existing systems are hindered by their emphasis on navigation tasks, dependence on two-handed gestures, lack of fine-grained precision, and testing on small, homogeneous populations. These limitations limit applicability to intricate, design-centric workflows. To address these limitations, we introduce a Gesture- and Voice-Controlled Virtual Reality System for immersive home design, which incorporates the multimodal fusion of hand-tracked gesture input and natural language instructions for precise furniture placement, material selection, and spatial adjustment. Three aspects of novelty in the system are (i) adaptive modes of interaction to support one-handed usage, numeric voice commands, and coarse-to-fine adjustment of precision; (ii) multimodal fusion architecture merging gesture, speech, and context information for sub-centimeter precision; and (iii) task-aware user experience evaluation framework tailored to home design processes. The development process follows a systematic pipeline, from requirements analysis and design of gesture and voice lexicons, to multimodal intent recognition, VR scene assembly with snapping and alignment, and iterative testing. Experimental results on a wide variety of participant populations demonstrate a 30% decrease in placement error and substantial improvements in usability and workload ratings over controller-based interaction. The results suggest the potential of multimodal VR for creating accessible and engaging home design experiences.
科研通智能强力驱动
Strongly Powered by AbleSci AI