In order to improve the recognition accuracy of vibration signals in distributed optical fiber vibration sensing systems, this paper proposes a method combing Dual-Layer Variational Mode Decomposition (DL-VMD) and Improved Compensation Distance Estimation Technology (ICDET). Firstly, this paper proposes the DL-VMD method to achieve a more refined decomposition of optical fiber vibration signals, which finally obtains three optimal Intrinsic Mode Functions (IMFs) with richer information; Secondly, the time-domain and frequency-domain features of the three IMFs are extracted as feature vectors. Then the proposed ICDET is used to optimize the extracted features; Finally, the Support Vector Machine (SVM) acts as the classifier to realize the recognition of optical fiber vibration signals. In order to verify the effectiveness of the proposed method, this paper carries out experiments on four common optical fiber vibration signals, and the results show that the recognition accuracy of this scheme on four vibration signals is 98.3%. This shows that the method proposed in this paper has great potential for application in the field of distributed optical fiber vibration sensing system.