Abstract Traps play a key role in the performance of infrared (IR) organic photodetectors (OPDs). However, the correlation between trap states and the device performance remains unclear, and the methods to modify trap states are very limited. In this work, inverted OPD devices based on PTB7‐Th and COTIC‐4Cl are fabricated by employing a layer‐by‐layer (LbL) solution processing method. LbL solution processing enhances the crystallinity and reduces the density of trap states, thereby mitigating carrier recombination and improving photoresponsivity. Moreover, a wide‐bandgap polymer, poly(N‐vinylcarbazole) (PVK), is introduced to further tailor the trap distribution, shifting it from shallower to deeper energy levels. Such changes in trap distribution suppress charge carrier injection under reverse bias resulting in a lower dark current. Finally, a high specific detectivity ( D * ) of over 10 12 Jones at 1050 nm is achieved under a bias of −0.2 V with a dark current of 6.94 × 10 −9 A cm −2 . These results elucidate the important relationship of trap energy distribution with the device performance and provide a versatile additive incorporation method to tailor energetic position of trap states, advancing the development of high‐performance SWIR OPDs.