TSCMDL: Multimodal Deep Learning Framework for Classifying Tree Species Using Fusion of 2-D and 3-D Features

Accurate tree species information is a prerequisite for forest resource management. Combining light detection and ranging (LiDAR) and image data is one main method of tree species classification. Traditional machinelearningmethods rely on expert knowledge to calculatea large number of feature parameters.Deep learning technology can directly use the original image and pointclouddata to classify tree species. However, data with different patterns require the use of different types of deeplearningmethods. In this study, a multimodal deeplearningframework (TSCMDL) that fuses 2D and 3D features was constructed and then used to combine data from multiple sources for tree species classification. This framework uses an improved version of the PointMLP model as its backbone network and uses ResNet50 and PointMLP networks to extract the image features and pointcloudfeatures, respectively. The proposed framework was tested using UAV LiDAR data and RGB orthophotos. The results showed that the accuracy of the tree species classification using the TSCMDL framework was 98.52%, which was 4.02% higher than that based on pointcloudfeatures only. In addition, when the same hyperparameters were used for training the model, the efficiency of the model training was not significantly lower than for models based on pointcloudfeatures only. The proposed multimodal deeplearningframework extracts features directly from the original data and integrates them effectively, thus avoiding manual feature screening and achieving more accurate classification. The feature extraction network used in the TSCMDL framework can be replaced by other suitable frameworks and has strong application potential.