Vision–based framework for automatic interpretation of construction workers' hand gestures

Construction robots have been recently developed to improve construction productivity and safety. One of the critical steps to make the robots work with human workers as teams is to provide a user-friendly interface to support their mutual interactions on construction sites. Compared with existing interfaces, hand gestures are easy to use, natural, and intuitive. This paper proposed a novel vision-based framework to capture and interpret the worker's hand gestures as a human-robot interface in construction. The framework consists of three components: worker detection and tracking, recognition queues formulation, and hand gesture recognition. Its effectiveness on the hand gesture recognition was tested with field experiments and achieved the overall precision and recall of 87.0% and 66.7%. Also, a laboratory study was conducted to illustrate the use of the framework to interact with a robotic dump truck. Future work will integrate the proposed framework into robotic construction machines.