Abstract—In this paper we recommend a novel method for detecting and tracking objects in the presence of cluttered background such as movements of leaves of trees, under various types of occlusion (object to object and object to scene occlusion) and scale change of object (small or large object) in real-time video. Object detection and tracking are two main terms of developing any tracking system. In our approach firstly we apply filters to remove noise and to avoid minute changes in the scene then we are using the frame differencing method to detect and segment the moving object. Contour tracking approach is applied to track the object of interest in all consecutive video frames. For checking superiority of this method we use it on different type of dataset: KTH and Own dataset. Key Terms—Frame differencing, Contour tracking, Average filter, cluttered background, object tracking.