Gait impairments are among the most prevalent and disabling symptoms in Parkinson's Disease (PD), featuring complex and highly heterogeneous manifestations. Developing methods for routinely and quantitatively assessing gait impairments is vital for understanding personalized disease progression and facilitating tailored treatments; nonetheless, achieving accurate and comprehensive assessments is considerably challenging. Here, we propose a deep learning-based framework to assess gait impairments using smartphone-recorded videos. This framework achieved a high proficiency for predicting PD severity, with a micro-average area under the receiver operating characteristic curve (AUC) of 0.87 and an F1 score of 0.806, comparable to the average performance of three clinical specialists. In addition, it effectively discerned the comprehensive efficacy of medications on gait impairments with a precision of 73.68%. In particular, it demonstrated the ability to discriminate medication-induced fine-granular gait changes beyond the resolution of the Unified Parkinson’s Disease Rating Scale (UPDRS). Furthermore, our interpretable framework enabled the extraction of traditional clinically used motion markers and the discovery of novel digital biomarkers more sensitive to disease progression and medication response than traditional ones. The findings underscore its great potential for efficiently assessing disease progression in clinical and home settings and evaluating disease-modifying effects in clinical trials to promote personalized therapies.