Real-time intelligent control of a vision-servoed fruit-picking robot

A programming language was developed to provide a modular approach for specifying intelligence of a vision-servoed fruit-picking robot. The programming language consisted of models, states, and global parameter definitions. Models mapped sensor and system data to binary values according to criteria established by a programmer. Global parameters were used for tuning modeling criteria on-line and for establishing communication between an operator and program execution. States specified robot actions and used model results to determine when actions had been completed or if problems existed that prevented completion of specified actions. A state network was created by linking states together with decision statements. The intelligence to perform a robot task was programmed in the state network. A state network was developed to provide a vision-servoed robot with the intelligence needed to automatically pick fruit. The picking intelligence recognized and solved many problems associated with grove operation of the robot. Grove-related problems that were addressed in the state network included attempting to pick fruit that were out of the robot work space, detecting excessive fruit motion that prevented a successful pick cycle, detecting when a fruit vanished from view during a pick cycle, picking fruit that were partially occluded by limbs and leaves, and detecting collisions between the robot and tree. Over 150 hours of development time were spent with the robot in grove settings. Fruit were successfully picked under a variety of production conditions. During a 35-minute period of fruit picking, 321 completed pick cycles were performed. This averaged to approximately 6.5 seconds per pick cycle. It was estimated that 75$ of the fruit that the robot attempted to pick was actually removed.