Abstract To further improve the thermal efficiency of the coal‐fired boiler, lower environment restraints, and minimize key components, the supercritical CO2 Brayton cycle was considered to replace the conventional steam Rankine cycle. In this paper, a three‐dimensional mathematical model of the thermal behaviors between the supercritical CO2 tube and flue gas was established using the Lam–Bremhorst k–ε model, the gas real model, and the P‐1 radiation model. The distributions of the velocity and the temperature between the supercritical CO2 tube and flue gas were investigated numerically. Furthermore, the effects of the supercritical CO2 inlet temperature on the thermal behaviors were also examined. The results show the surface heat transfer coefficient goes up with the supercritical CO2 inlet temperature due to the rise of the thermal conductivity. More significant thermal behaviors and detailed physical explanations were elaborated, which would offer a novel insight to understand, design, and optimize the supercritical CO2 Brayton cycle for engineering applications.