Long-wavelength interdomain phonons and instability of dislocations in small-angle twisted bilayers

Abstract We develop a theory for long-wavelength phonons originating at dislocations separating domains in small-angle twisted homobilayers of two-dimensional materials such as graphene and MX 2 transition metal dichalcogenides ( M = Mo, W; X = S, Se). We find that both partial and perfect dislocations, forming due to lattice relaxation in the twisted bilayers with parallel and antiparallel alignment of the unit cells of the constituent layers, respectively, support several one-dimensional subbands of the interdomain phonons. We show that the spectrum of the lowest gapless subband is characterized by imaginary frequencies for wave numbers below a critical value, dependent on the dislocation orientation, which indicates an instability for long enough straight partial and perfect dislocations. We argue that the pinning potential and/or small deformations of the dislocations can stabilize the gapless phonon spectra. The other subbands are gapped, with subband bottoms lying below the frequency of the interlayer shear mode in the domains, which facilitates their detection with the help of optical and magnetotransport techniques.