The existence of nonlinearity is an inevitable frequent occurrence that should be considered to accurately identify the modal parameters of a vibration system using operational modal analysis. A problem is that the traditional operational modal analysis method based on the linear modal theory is not applicable to modal parameter identification of vibration systems with nonlinearity. A solution is as follows: this paper is aimed at solving the problem by proposing a new operational modal analysis method to carry out modal parameter identification for a nonlinear vibration system. The new operational modal analysis method, based on the forced response and symbolic regression method without assuming any pre-existing information and only using mathematical symbols, is introduced to solve the problem by automatically searching for the expression structure and modal parameters of a system in nonlinear normal modes. The simulation result of a three-degrees-of-freedom nonlinear system reveals the high accuracy of the proposed operational modal analysis method in extracting the modal parameters. Then, a rod fastening rotor model is considered, and the capability of the proposed operational modal analysis method to precisely extract its modal parameters is further evaluated.