ABSTRACT In this paper, the epoxy (EP) resin blended with the polyurethane (PU) prepolymer and carboxylated carbon nanotube (CNT) was cured with a dicyandiamide curing agent. The prepared thermoset specimens were analyzed by Fourier‐transform infrared spectroscopy, dynamic mechanical analysis, scanning electron microscopy, and tensile and impact testing. Based on the molecular dynamics simulation method, the binding energy and elastic constants of the EP composites were calculated. The results of the experiment and simulation showed that PU addition can increase the tensile strength and impact strength as well as reduce the elasticity modulus for the EP composites. CNT had the potential to enhance tensile strength and elasticity modulus, an adverse effect on improving impact strength due to the agglomerations. The synergistic effect of PU and CNT on the modified EP was verified to be able to simultaneously increase the tensile strength, elastic modulus, and impact strength. The addition of PU and CNT was further optimized by the response surface methodology coupled with central composite design to acquire the high mechanical performance of the EP composite. The optimization results show that the addition of 5.70 wt% PU and 0.23 wt% CNT provided the greatest synergistic effect on improving the mechanical performance of EP composites, and the tensile strength, Young's modulus, and impact strength of the EP composite increased by 19.92%, 0.14%, and 22.93%, respectively, compared to the pure EP.