Identifying shoplifting behaviors and inferring behavior intention based on human action detection and sequence analysis

• Identifying abnormal behaviors affecting public safety such as shoplifting. • Framework based on detection and sequence analysis of human actions. • Better inference of the intentions underlying abnormal behaviors. • Enhancing public safety by identifying abnormal scenarios rapidly and efficiently. Identification of abnormal behaviors affecting public safety (e.g., shoplifting, robbery, and stealing) is essential for preventing human casualties and property damage. Many studies have attempted to automatically identify abnormal behaviors by detecting relevant human actions by developing intelligent video surveillance systems. However, these studies have focused on catching predefined actions associated explicitly with the target abnormal behavior, which can lead to errors in judgment when such actions are undetected or inaccurately detected. To better identify abnormal behaviors, it is essential to understand a series of performed actions to capture behaviors’ pre- and post-indications (e.g., repeatably looking around and spotting CCTVs) and infer the intentions underlying such behaviors. Thus, in the present study, we propose a framework to identify abnormal behaviors through deep-learning-based detection of non-semantic-level human action components segmented with a window size of several seconds (e.g., walking, standing, and watching) and performing sequence analyses of the detected action components to infer behavior intentions. Then, we tested the applicability of the framework to the specific scenario of shoplifting, one of the most common crimes. Analysis of actual incident data confirmed that shoplifting intentions could be effectively gauged based on distinct action sequence features, and the intention inference results are continuously updated with the accumulated series of detected actions during the course of the input video stream. The results of this study can help enhance the ability of intelligent surveillance systems by providing a new means for monitoring abnormal behaviors and deeply understanding the underlying intentions.