计算机科学
编码器
人工智能
计算机视觉
语音识别
钥匙(锁)
模式识别(心理学)
自动化方法
自动化
集合(抽象数据类型)
信号处理
作者
S Wang,Zhaoming Pang,S Wang,Xinzhu Xu
出处
期刊:
[Institute of Electrical and Electronics Engineers]
日期:2026-04-21
卷期号:: 4741-4745
标识
DOI:10.1109/icassp55912.2026.11462928
摘要
Accurate diagnosis of cardiovascular diseases requires a deep understanding of electrocardiogram (ECG) signals, including their complex temporal dynamics, structural characteristics, and multi-lead spatial correlations. However, existing unimodal methods often fail to capture these diverse characteristics simultaneously, which further limits the performance of downstream tasks such as automated report generation. To address this challenge, we first propose a dual-channel sequence-image collaborative embedding framework. This framework enables robust multimodal learning by concurrently extracting raw time-series features and time-frequency image features from ECG data. Despite the effectiveness of this dual-channel encoder in capturing ECG features, general multi-modal large language models (MLLMs) still struggle with ECG report generation—primarily due to the inherent complexity of ECG interpretation and the modality gap between text and ECG signals. We thus develop ECG-ReportGen, an ECG-focused MLLM that integrates our dual-channel encoder. We further design lightweight adapters to bridge the gap between ECG features and the MLLM’s semantic space, incorporate a Retrieval-Augmented Generation (RAG) module to mitigate hallucinations in medical knowledge, and adopt LoRA (Low-Rank Adaptation) for lightweight MLLM fine-tuning to generate structured reports. Experiments validate the effectiveness of our innovations: the dual-channel encoder outperforms unimodal baselines in feature representation, and ECG-ReportGen achieves strong performance in automated report generation.
科研通智能强力驱动
Strongly Powered by AbleSci AI