Interactive Enhanced Network Based on Multihead Self-Attention and Graph Convolution for Classification of Hyperspectral and LiDAR Data

高光谱成像 计算机科学 激光雷达 卷积(计算机科学) 人工智能 遥感 图形 模式识别(心理学) 理论计算机科学 人工神经网络 地质学
作者
Hongmin Gao,Hao Feng,Yiyan Zhang,Shuyu Fei,Runhua Sheng,Shufang Xu,Bing Zhang
出处
期刊:IEEE Transactions on Geoscience and Remote Sensing [Institute of Electrical and Electronics Engineers]
卷期号:62: 1-16 被引量:27
标识
DOI:10.1109/tgrs.2024.3467674
摘要

The fusion of multimodal data plays a crucial role in classification tasks. However, existing research typically mines and analyzes the individual features of each data source separately before considering how to fuse them. In contrast, our approach first constructs interactive enhanced fusion features (IEFFs) for initial fusion while considering the extraction of individual features and, finally, integrates them effectively to utilize the information from each data source more comprehensively. To this end, we propose a novel interactive enhanced network based on multihead self-attention (MSA) and graph convolution. Specifically, we extract individual features from hyperspectral image (HSI) and light detection and ranging (LiDAR) data and then construct IEFFs based on the row and column features of the central pixel. Individual features focus on the local characteristics of a single data source, while IEFFs strengthen the feature expression of the central pixel through matrix operations, integrating the complementary information of multimodal data. Subsequently, we use graph convolutional networks (GCNs) to construct graph structures for four types of features (interactive enhanced HSI features, interactive enhanced LiDAR features, HSI individual features, and LiDAR individual features), modeling the pixels as nodes and capturing spatial relationships. On this basis, we apply an MSA mechanism to mine spectral dependencies, further extracting global spectral features. Finally, we design a multimodal gated fusion module (MGFM) that effectively integrates these features through its weighting mechanism. The weight allocation is adjusted dynamically according to the characteristics of the feature, achieving optimal fusion of multimodal data. Extensive experiments on three popular HSI and LiDAR datasets verify the superior performance of our method. Our code will be available athttps://github.com/haofeng0003/MSA-GCN.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
Daodao发布了新的文献求助10
刚刚
xtt发布了新的文献求助10
1秒前
1秒前
三木发布了新的文献求助10
1秒前
彩色脆霜发布了新的文献求助10
2秒前
www完成签到,获得积分10
2秒前
研友_VZG7GZ的应助被shymuaiu采纳,获得10
3秒前
3秒前
Feny发布了新的文献求助10
3秒前
GingerF的应助被Tuniverse_采纳,获得100
3秒前
科研通AI6.4的应助被miao采纳,获得10
3秒前
3秒前
千贝儿完成签到,获得积分10
3秒前
wy发布了新的文献求助10
4秒前
sunstar完成签到,获得积分10
4秒前
悦耳安萱完成签到 ,获得积分10
4秒前
Ava的应助被111采纳,获得10
5秒前
hui完成签到,获得积分10
5秒前
111发布了新的文献求助10
5秒前
6秒前
6秒前
科研通AI6.3的应助被酱鱼采纳,获得10
7秒前
lunna发布了新的文献求助10
7秒前
ss完成签到,获得积分10
7秒前
学习牛马发布了新的文献求助30
7秒前
8秒前
甘甘甘甘的应助被科研通管家采纳,获得30
8秒前
Jasper的应助被cai采纳,获得10
8秒前
科研通AI2S的应助被科研通管家采纳,获得10
8秒前
无极微光的应助被科研通管家采纳,获得20
8秒前
打打的应助被科研通管家采纳,获得10
9秒前
情怀的应助被科研通管家采纳,获得10
9秒前
04d的应助被科研通管家采纳,获得10
9秒前
田様的应助被科研通管家采纳,获得10
9秒前
9秒前
9秒前
小蘑菇的应助被wei采纳,获得10
9秒前
852的应助被科研通管家采纳,获得10
9秒前
斯文败类的应助被bam采纳,获得10
9秒前
orixero的应助被科研通管家采纳,获得10
9秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Agyptische Geschichte der 21.30. Dynastie 510
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
基于界面调控的高效湿气发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8192209
求助须知:如何正确求助?哪些是违规求助? 9578801
关于积分的说明 21861940
捐赠科研通 7791695
什么是DOI,文献DOI怎么找? 3436846
关于科研通互助平台的介绍 2560562
邀请新用户注册赠送积分活动 2483617