Deep Dexterous Grasping of Novel Objects From a Single View

Dexterous grasping of a novel object given a single view is an open problem. This paper makes several contributions to its solution. First, we present a simulator for generating and testing dexterous grasps. Second, we present a dataset, generated by this simulator, of 2.4 million simulated dexterous grasps of variations of 294 base objects drawn from 20 categories. Third, we combine an existing approach to learn a grasp generation model with three different learned evaluative models employing ResNet-50 or VGG16 as their visual backbone. Fourth, we train, and evaluate 17 variants of the resulting generative-evaluative architectures on the simulated dataset, showing improvement from 69.53% grasp success rate to 90.49%. Fifth, we present a real robot implementation and evaluate the four most promising variants, executing 196 real robot grasps in total. We show that our best architectural variant achieves a grasp success rate of 87.8% on real novel objects seen from a single view, improving on a baseline of 57.1%. Finally, we explore the inner workings of our best evaluative model and perform an extensive analysis of its results on the simulated dataset.