In this paper, we present an extension to the Recurrent Motion Image (RMI) motion-based object recognition framework for use in development of automated video surveillance systems. We extended the object classes of RMI to include four-legged animals (such as dog and cat) and enhanced the preprocessing and shadow removal algorithms for better object segmentation and recognition. Under the new framework, object blobs obtained from background subtraction of scenes are tracked using region correspondence. In turn, we calculate the RMI signatures based on the silhouettes of the object blobs for proper classification. This new framework is tested on several real world 320 x 240 resolution color image sequences captured with a low-end digital camera, and all of the moving objects in our samples are properly detected, tracked and classified - indicating the applicability of the new framework in similar task environment.