Deep-learning-based vision for earth-moving automation

Earth-moving machines are heavy-duty vehicles designed for construction operations involving earthworks. The tasks performed by such machines typically involve navigation and interaction with materials such as soil, gravel, and blasted rock. Skilled operators use a combination of visual, sound, tactile and possibly motion feedback to perform tasks efficiently. We survey the literature in this research area and analyse the relative importance of different sensor system modalities focusing on deep-learning-based vision and automation for the short-cycle loading task. This is a common and repetitive task that is attractive to automate. The analysis indicates that computer vision, in combination with onboard sensors, is more critical than coordinate-based positioning. Furthermore, we find that data-driven approaches, in general, have high potential in terms of productivity, adaptability, versatility and wear and tear with respect to automation system solutions. The main knowledge gaps identified relate to loading non-fine heterogeneous material and navigation during loading and unloading.