3-Amide-β-carbolines block the cell cycle by targeting CDK2 and DNA in tumor cells potentially as anti-mitotic agents

β-Carboline alkaloids are natural and synthetic products with outstanding antitumor activity. C3 substituted and dimerized β-carbolines exert excellent antitumor activity. In the present research, 37 β-carboline derivatives were synthesized and characterized. Their cytotoxicity, cell cycle, apoptosis, and CDK2- and DNA-binding affinity were evaluated. β-Carboline monomer M3 and dimer D4 showed selective activity and higher cytotoxicity in tumor cells than in normal cells. Structure-activity relationships (SAR) indicated that the amide group at C3 enhanced the antitumor activity. M3 blocked the A549 (IC50 = 1.44 ± 1.10 μM) cell cycle in the S phase and inhibited A549 cell migration, while D4 blocked the HepG2 (IC50 = 2.84 ± 0.73 μM) cell cycle in the G0/G1 phase, both of which ultimately induced apoptosis. Furthermore, associations of M3 and D4 with CDK2 and DNA were proven by network pharmacology analysis, molecular docking, and western blotting. The expression level of CDK2 was downregulated in M3-treated A549 cells and D4-treated HepG2 cells. Moreover, M3 and D4 interact with DNA and CDK2 at sub-micromolar concentrations in endothermic interactions caused by entropy-driven adsorption processes, which means that the favorable entropy change (ΔS > 0) overcomes the unfavorable enthalpy change (ΔH > 0) and drives the spontaneous reaction (ΔG < 0). Overall, these results clarified the antitumor mechanisms of M3 and D4 through disrupting the cell cycle by binding DNA and CDK2, which demonstrated the potential of M3 and D4 as novel antiproliferative drugs targeting mitosis.