Abstract The present study optimized the structure of the capillary wick to enhance the heat transfer performance of a 2 m long loop heat pipe (LHP) using ammonia as the working fluid for infrared camera cooling. The composite wick was designed and sintered to have the outer layer of wick being nickel to provide high capillary forces and high thermal conductivity, and have the inner layer being stainless steel to reduce flow resistance and heat leakage. A three-dimensional computational fluid dynamics (CFD) model was constructed to simulate the heat and mass transfer process in a cylindrical evaporator. A capillary pressure model that was built using user-defined functions was used to describe the wicking process. To improve simulation accuracy, the wick parameters including porosity, pore radius, and permeability have been measured and used. Test results indicated that the LHP with the optimized composite wick had a total thermal resistance of 0.198 K/W and an evaporator temperature of 311.90 K at a heat load of 120 W and an antigravity orientation with the evaporator elevated by 50 cm. Compared with that of the monoporous wick, the thermal resistance has been reduced by about 13%, and the heat transfer coefficient of the evaporator has been enhanced by 94%. Through the comparative analysis of temperature, flowrate, and heat distribution, the composite wick structure achieves enhancing liquid replenishment and reducing local heat accumulation, which can enhance the evaporation efficiency and optimize the LHP thermal resistance.