Estimating Fingertip Forces, Torques, and Local Curvatures from Fingernail Images

SUMMARY The study of dexterous manipulation has provided important insights into human sensorimotor control as well as inspiration for manipulation strategies in robotic hands. Previous work focused on experimental environment with restrictions. Here, we describe a method using the deformation and color distribution of the fingernail and its surrounding skin to estimate the fingertip forces, torques, and contact surface curvatures for various objects, including the shape and material of the contact surfaces and the weight of the objects. The proposed method circumvents limitations associated with sensorized objects, gloves, or fixed contact surface type. In addition, compared with previous single finger estimation in an experimental environment, we extend the approach to multiple finger force estimation, which can be used for applications such as human grasping analysis. Four algorithms are used, c.q., Gaussian process, convolutional neural networks, neural networks with fast dropout, and recurrent neural networks with fast dropout, to model a mapping from images to the corresponding labels. The results further show that the proposed method has high accuracy to predict force, torque, and contact surface.