Background: Antibody-drug conjugates (ADCs) represent a novel therapeutic class that combines an antibody against a tumor-associated antigen (TAA), a payload, and a linker that binds these two components. Serious adverse events (SAEs), particularly those of grade 3 (G3) or higher, frequently contribute to the abandonment of ADCs during clinical development. Methods: In this study, we analyzed the toxicity profiles of all approved ADCs, aiming to uncover correlations between their safety profiles and the specific characteristics of their components. Results: In our analysis, dose reductions, dose delays, treatment discontinuations, and ≥G3 toxicities were not significantly different across payload types. Similarly, no association was found between the payload mechanism of action and ≥G3 toxicities, including anemia, neutropenia, febrile neutropenia, thrombocytopenia, and diarrhea. By exploring the specific toxicities of ADCs observed by organ, we identified that most were related to the payload mechanism of action, like the ≥G3 diarrhea observed in 10% of patients treated with sacituzumab govitecan (the payload SN-38 is the active metabolite of irinotecan), and very few were related to the presence of the TAA in normal tissue (presence of Nectin-4 in skin and ≥G3 rash toxicity in 14% of patients treated with enfortumab vedotin). In line with this, no major differences in ≥G3 toxicities were identified in studies with different levels of the TAA (trastuzumab deruxtecan in Destiny Breast Studies with different HER2 expression levels). Conclusions: Our analysis reveals that most ADC toxicities are driven by the payload's effects on non-transformed tissues; however, a detailed analysis of each ADC component should be taken into consideration.