Abstract Mixed layer instability (MLI) is generally regarded as a key mechanism for generating submesoscale processes (submesoscales) in the upper ocean. However, this mechanism alone is insufficient to explain recent observations of submesoscales that penetrate far beneath the mixed layer. Through analyzing the 1/48° LLC4320 simulation data in the North Pacific subtropical countercurrent region, this study finds that the e -folding attenuation depth of most submesoscales is much deeper than the mixed layer, and their energy injection scale is larger than that predicted by MLI. These deep-penetrating submesoscales can be explained by recently proposed mixed transitional layer instability (MTI), the baroclinic instability occurring in the mixed layer and transitional layer (a moderately stratified layer between the mixed layer and thermocline). The MTI mechanism is illustrated using a three-layer quasigeostrophic model, which includes a weakly stratified mixed layer on the top, a well-stratified deep layer on the bottom, and a moderately stratified transitional layer in the middle. Instability analysis based on realistic stratification and mean flow shear reveals three baroclinically unstable modes: MLI, MTI, and mesoscale baroclinic modes. The unstable waves of MTI have a mean wavelength of 58 km (radius of ∼18 km), which is a few times larger and smaller than those of MLI and mesoscale baroclinic instability, respectively. The vertical structure of MTI unstable waves is significantly deeper than that of MLI and resembles the structure of submesoscales simulated in LLC4320. As such, this study demonstrates that MTI is an important mechanism for generating deep-penetrating submesoscales in subtropical upper oceans. SIGNIFICANCE STATEMENT Mixed layer instability (MLI) is regarded as a key generation mechanism of submesoscales, but it is insufficient to explain the observed submesoscales that usually penetrate far beneath the mixed layer. Based on the submesoscale phenomenon analysis and linear instability theory, we show that the deep-penetrating submesoscales can be explained by a special type of baroclinic instability occurring in the mixed and transitional layers [mixed transitional layer instability (MTI)]. Compared to submesoscales generated by MLI, those of MTI have larger horizontal and vertical scales and more resemble the realistic simulation results in the subtropical upper ocean. The submesoscales generated by MTI may exert stronger influences on vertical tracer transport, for example, taking nutrients from nutricline to the euphotic layer more efficiently in the oligotrophic subtropical gyre.