Teleoperation stands for operating a system, machine, or robot from a distance. Teleoperation has been widely adopted as a promising way to enhance worker safety in extreme and hazardous construction workplaces. Over the years, studies have proposed various approaches to teleoperate construction equipment for excavation, which could bring significant advantages such as declining injury rate and dealing with dangerous on-site tasks. This paper identifies challenges, tasks, and opportunities in teleoperation of excavator through quantitative and qualitative analyses. Prior studies from the past two decades were rigorously reviewed and analyzed through the bibliometric analysis and a systematic review for in-depth discussion. The outcomes provide the future direction of teleoperation in construction workplaces in the following aspects: human operator, interface, operation, and environment. The outcomes indicate that human-centered research that understands and develops systems and technologies from the human point of view is necessary since seamless human-robot interaction is required for teleoperation.