With the rapid development of negative thermal expansion (NTE) materials, there is an increasing need to explore the underlying mechanisms behind NTE, particularly the phonon vibration mechanisms, which have garnered considerable attention from researchers. The isotropic NTE compound tetracyanoborates $AB{(\mathrm{CN})}_{4}$ ($A=\mathrm{Cu}$, Li, Ag), exhibits an intriguing phenomenon: a significant enhancement in NTE upon the substitution of $A$-site atoms. This study utilizes first-principles calculations based on density functional theory to investigate how $A$-site atom substitution affects thermal expansion mechanisms. Crystal structure analysis indicates that the substitution of $A$-site atoms changes the ionic character and bond lengths of $A$-N bonds, thereby influencing structural flexibility. Moreover, a strong linear relationship is observed between the coefficients of thermal expansion (CTE) and the average atomic volume (AAV). Phonon vibration analysis, based on the Gr\"uneisen parameters, reveals that low-frequency optical phonon modes involve coupled rotations of the $A{\mathrm{N}}_{4}$ and $B{\mathrm{C}}_{4}$ tetrahedra, along with nonrigid movement of the $A{\mathrm{N}}_{4}$ tetrahedra, leading to a reduction in lattice volume causing NTE. In addition, the low-frequency transverse acoustic (TA) mode with negative Gr\"uneisen parameter induces contraction in specific directions, further promoting NTE. Thus, the synergistic effects of low-frequency optical and acoustic modes emerge as the primary drivers behind the NTE phenomenon. This study offers insight into the phonon vibration mechanisms in tetracyanoborates, contributing to a deeper understanding of the NTE mechanism.