Violent behaviors occurred in elevators have been frequently reported by media in recent years. It is necessary to provide a safe elevator environment for passengers. A new visual surveillance system with the function of abnormal behavior detection is proposed in this paper. Firstly, human objects in surveillance video are extracted by background subtraction, and meanwhile the number of people in each image is counted. Then, some algorithms are presented to deal with different abnormal behaviors. For one person case, we pay attention to whether the person fell down or not. And for two or more people case, we use the image entropy of Motion History Image (MHI) to detect if there is violent behavior. Experimental results show that the proposed algorithms can offer satisfactory results.