Based upon the application of the time domain reflectometry(TDR) in the monitoring of the defective coaxial cable, a calculation model of TDR system was presented to study the propagation characteristics of electromagnetic wave in defective coaxial cable system. The propagation characteristics of electromagnetic wave in three types of sheared coaxial cables were tested. The influences of coaxial cable's shear deformation on the characteristic impedance of coaxial cable and the TDR signal were analyzed. The theoretical and experimental results show that the ratio of the impedance of a coaxial cable to its shear deformation is almost a constant, which depends on the type of the cable; and that the spike of reflection wave increases with the shear deformation. Therefore, it is applicable to monitor the slope stability by observing the reflection signal of TDR system.