计算机科学
计算机视觉
人工智能
地标
渲染(计算机图形)
高斯分布
高保真
平滑的
偏移量(计算机科学)
混合模型
合成数据
计算机图形学(图像)
发电机(电路理论)
语音识别
可用的
运动捕捉
稳健性(进化)
视图合成
抖动
忠诚
变压器
可视化
作者
Zhen Xiong,Haozhi Huang,Zhen-Sheng Yuan,Guanghua Yang,Jundong Tan
标识
DOI:10.1109/ijcnn64981.2025.11227339
摘要
Synthesis of talking portraits is a challenging task. Recently, neural radiance fields (NeRF) have been utilized to achieve real-time synthesis and high-fidelity 3D modeling of talking portraits, demonstrating exciting success. However, besides the slow running speed of NeRF, most methods tend to produce artifacts and visual jitter due to the lack of detailed control over NeRF’s implicit expressions. In this work, we propose a novel method for pose-driven talking portrait synthesis based on 3D Gaussian Splatting (3DGS). The pose is composed of three dynamic data types: facial landmark, audio, and camera viewpoint. Our method consists of two modules: the facial landmark generator module and the Gaussian rendering module. The facial landmark generator module is based on a transformer network, which employs a cross-attention mechanism to capture the relationship between audio and facial landmark, outputting the facial landmark corresponding to any given audio. In the 3DGS part, the motion canonical Gaussian of the digital human is first obtained through static reconstruction. The spatial perception module is then used to ascertain the relationship between each type of pose data and the each Gaussian, predicting the offset of the Gaussian. The canonical Gaussian and the offset are combined to obtain the dynamic Gaussian. Extensive experiments show that our method outperforms existing methods in terms of the fidelity and synchronization of speech and video.
科研通智能强力驱动
Strongly Powered by AbleSci AI