Coal seam gas drilling system as research object is studied. Considering the structural features of drill bit-drill string, boundary conditions and drill press, drill speed, gravity, friction, drilling fluid buoyancy and other loads, then model the nonlinear dynamic model of drill tool-coal rock system. The drilling system is modeled by FEM software ABAQUS which include drilling string, wall, drill bits, coal rock. Analysis for gradual change of coal rock and motion, force of drill string and drill bit which in the process of drilling. Finally, the failure law of coal rock and the motion of drill bit have been demonstrated by simulation results,it is both effective and useful.