Firstly, the set of dynamics and kinematics equations of motion is simplified according to the features of three-dimensional constrained ascent trajectory of RLV, which improves the efficiency of optimization iteration algorithm greatly. Secondly, the trajectory optimal control problem is converted into constraint nonlinear programming program by adopting parameter optimization method, and then this constraint nonlinear programming program is transmuted into unconstraint nonlinear programming program using an augmented-lagrange multiplier method. Consequently, the BFGS method is used to solve this problem. At last, the ascent trajectory optimization problems of two Launch modes of RLV have been simulated numerically based on this algorithm, namely air-launch mode and rocket-aided-launch mode. Successful results show this algorithm is able to generate a feasible ascent trajectory of about 700 s flight time in 20-30 s on the desktop computer.