An autonomous optical navigation method based on images of asteroids during interplanetary cruise is presented.In this method,firstly,images of asteroids with star background are taken by the spacecraft navigation camera.Then images are processed to determine the center of the asteroid and stars in the camera frame.Finally,the locations of the centers of asteroids and stars,the inertial attitude of spacecraft and the ephemeris of asteroids at the time of observing are used to determine the orbit by using a new nonlinear filter combining Unscented transformation and batch least square.The method is demonstrated with simulation according to the real trajectory of Deep Space 1 mission,and comparison between results of this new method and batch least square filter(BLS) used by Deep Space 1 shows that the former has higher accuracy.