MMTSCNet: Multimodal Tree Species Classification Network for Classification of Multi-Source, Single-Tree LiDAR Point Clouds

Trees play a critical role in climate regulation, biodiversity, and carbon storage as they cover approximately 30% of the global land area. Nowadays, Machine Learning (ML)is key to automating large-scale tree species classification based on active and passive sensing systems, with a recent trend favoring data fusion approaches for higher accuracy. The use of 3D Deep Learning (DL) models has improved tree species classification by capturing structural and geometric data directly from point clouds. We propose a fully Multimodal Tree Species Classification Network (MMTSCNet) that processes Light Detection and Ranging (LiDAR) point clouds, Full-Waveform (FWF) data, derived features, and bidirectional, color-coded depth images in their native data formats without any modality transformation. We conduct several experiments as well as an ablation study to assess the impact of data fusion. Classification performance on the combination of Airborne Laser Scanning (ALS) data with FWF data scored the highest, achieving an Overall Accuracy (OA) of nearly 97%, a Mean Average F1-score (MAF) of nearly 97%, and a Kappa Coefficient of 0.96. Results for the other data subsets show that the ALS data in combination with or even without FWF data produced the best results, which was closely followed by the UAV-borne Laser Scanning (ULS) data. Additionally, it is evident that the inclusion of FWF data provided significant benefits to the classification performance, resulting in an increase in the MAF of +4.66% for the ALS data, +4.69% for the ULS data under leaf-on conditions, and +2.59% for the ULS data under leaf-off conditions. The proposed model is also compared to a state-of-the-art unimodal 3D-DL model (PointNet++) as well as a feature-based unimodal DL architecture (DSTCN). The MMTSCNet architecture outperformed the other models by several percentage points, depending on the characteristics of the input data.