A segmentation of moving object in video sequence is a core technique of intelligent image processing system such as video surveillance, traffic monitoring. A typical method to segment a moving region from the background is the background subtraction. One of famous background modeling is Gaussian mixture model (GMM). Even though the method is known efficient and exact, GMM suffers a problem that includes false pixels in ROI (region of interest), specifically shadow pixels. These false pixels cause fail of the postprocessing tasks such as tracking and object recognition. This paper presents a method for removing false pixels included in ROI. First, we subdivide a ROI by using shape characteristics of detected objects. Then, a method is proposed to classify pixels from using histogram characteristic and comparing difference of energy that converts the color value of pixel into grayscale value, in order to estimate whether the pixels belong to moving object area or shadow area. The method is applied to real video sequence and the performance is verified.