Abstract Objectives To investigate the long-term trajectory of radiographic progression in difficult-to-treat rheumatoid arthritis (D2T RA) and poly-refractory RA (pr-RA) patients and to evaluate the impact of ultrasound-based persistent inflammatory refractory RA (PIRRA) and non-inflammatory refractory RA (NIRRA) classification on predicting rapid radiographic progression (RRP, ≥5 mSvdH units/year). Methods Radiographic damage was assessed using the modified Sharp/van der Heijde (mSvdH) score in EULAR-defined D2T RA patients. PIRRA and NIRRA subgroups were classified based on a single ultrasound time point assessing grayscale and power Doppler synovitis. The impact of time-integrated CRP and swollen joint counts (SJC) on radiographic progression was examined. Results Among 254 D2T RA patients, 114 had serial radiographs with a mean follow-up of 9 years. The mean annual mSvdH progression was 2.8 units with both time-integrated CRP (P < 0.001) and the PIRRA patients (n = 43) having significantly greater annual radiographic progression (3.3 units in PIRRA vs 2.4 units in NIRRA, P = 0.025). In multivariable analysis, older age (P = 0.017) and swollen joint count (P = 0.009) were independently associated with RRP. Additionally, RRP was observed in 50% of pr-RA patients (n = 14) vs 19.4% in other D2T RA cases (P = 0.048). Conclusion Although pr-RA cases are an uncommon subgroup, half of them demonstrated RRP, emphasizing the need for more aggressive treatment approaches. In contrast, many D2T RA patients exhibited comparatively slow radiographic progression indicating that many D2T RA cases are at least partially treated. These findings underscore the heterogeneity within D2T RA and highlight the need for additional strategies for the pr-RA subgroup.