Global bifurcation analysis in a predator–prey system with simplified Holling IV functional response and antipredator behavior

In this paper, we study a predator–prey system with the simplified Holling IV functional response and antipredator behavior such that the adult prey can attack vulnerable predators. The model has been investigated by Tang and Xiao, and the existence and stability of all possible equilibria are determined. In addition, they performed a bifurcation analysis and showed that the system undergoes a Codimension 2 Bogdanov–Takens bifurcation. In this paper, for the same model, we further show that the cusp‐type Bogdanov–Takens bifurcation can be of Codimension 3, which acts as an organizing center for the whole bifurcation set. In addition, we propose the existence of Hopf bifurcation of Codimension 2 and the coexistence of stable limit cycle and unstable limit cycle. In particular, we show that the antipredator behavior has great effect on the dynamics of the model, it may cause the predator population to extinct while the prey population will increase up to the carrying capacity. Numerical simulations including bifurcation diagrams and phase portraits are performed to illustrate and confirm the theoretical results. These results may enrich the dynamics of predator–prey systems.