An algorithm is proposed to estimate the general motion of multiple moving objects in an image sequence. The general motion is described by a general motion model which is specified by number of parameters called motion parameters. By estimating the individual motion of the objects, segmentation according to motion is achieved at the same time. The algorithm works directly on an image sequence without using any higher level information such as corners and edges. Furthermore, it is not necessary to go through the estimation of optical flow as an intermediate step, avoiding the error caused by estimating optical flow.