Inferring accurate and precise material models necessary for high-fidelity predictions has been a central challenge in constitutive modeling. Both traditional regression methods and modern machine-learning approaches require specialized data labels, which often cannot be sufficiently obtained from experiments. This data demand makes many sophisticated models impractical for real-world problems. Improvements in digital image correlation techniques have enabled accurate measurements of displacement data, providing an alternative kinematics-based approach for model identification. However, for materials undergoing plastic deformation, fracture, and damage, the corresponding inverse problem could be inherently challenging due to the dependence on loading history. We overcome this by formulating an inverse problem to discover interpretable plasticity models parameterized by neural networks (NN) from kinematic observations, leveraging a differentiable simulator with a smooth constitutive update that enables backpropagation for the NN training. The ability to use kinematic observations to infer complex material models may pave the way for a massive generation of material models that can be game-changing for emerging applications such as the design of metamaterials, response surface analyses, and the design of experiments.