Paramagnetic rim lesions (PRLs) are an imaging biomarker of chronic inflammation in MS that are associated with more aggressive disease. However, the precise tissue characteristics and extent of their damage, particularly with regard to connected axonal tracts, are incompletely understood. Quantitative diffusion tissue measurements and fiber tractography can provide a more complete picture of these phenomena. One hundred fifteen people with MS were enrolled in this study. Quantitative susceptibility mapping and DWI were acquired on a 3T MRI scanner. PRLs were identified in 49 (43%) subjects. Diffusion tractography was then used to identify nearby PRL-connected versus non-PRL connected tracts and PRL-connected versus nonconnected surrounding tracts. DWI metrics, including fractional anisotropy (FA), quantitative anisotropy (QA), mean diffusivity, axial diffusivity, radial diffusivity, isotropy, and restricted diffusion imaging, were compared between these tracts and within PRLs and non-PRL lesions themselves. Tissue within PRLs had significantly lower FA than tissue within non-PRL T2 lesions (P = .04). Tracts connected to PRLs exhibited significantly lower FA (P < .001), higher restricted diffusion imaging (P = .02, and higher Iso values (P = .007) than tracts connected to non-PRL T2 lesions. Only QA was different between tracts connected to PRLs and nonconnected surrounding tracts (P = .003). PRLs are more destructive both within themselves and to surrounding tissue. This damage appears more spatially than axonally mediated.