Se-doped $\mathrm{B}{\mathrm{i}}_{2}\mathrm{T}{\mathrm{e}}_{3}$ is a commercial $n$-type thermoelectric material near room temperature, which is valued for its high ZT and low lattice thermal conductivity. Nowadays, this material has attracted widespread interest as a candidate for advancing sustainable energy solutions. However, the understanding of the origins of this low lattice thermal conductivity remains inadequate. We present a temperature-dependent Raman spectroscopy study exploring the effects of Se atom substitution on phonon behavior and anharmonicity in the Se-doped $\mathrm{B}{\mathrm{i}}_{2}\mathrm{T}{\mathrm{e}}_{3}$ system, and discuss their influence on the low lattice thermal conductivity. Our analysis of the temperature-dependent Raman shift and linewidth of Raman peaks reveals significant third-order anharmonicity across all observed phonon modes. Conversely, tiny fourth and higher-order anharmonicity occurs only in the out-of-plane ${A}_{1\mathrm{g}}$ vibration modes, regardless of the Se substitution level. Furthermore, as the Se substitution exceeds one-third, Se begins to replace the Te1 site in $\mathrm{B}{\mathrm{i}}_{2}\mathrm{T}{\mathrm{e}}_{3}$, which significantly broadens the linewidth of the ${E}_{\mathrm{g}}^{2}$ mode. This broadening indicates a rapid reduction in the lifetime of ${E}_{\mathrm{g}}^{2}$ mode owing to increased alloy scattering. These findings provide direct evidence of phonon anharmonicity in Se-doped $\mathrm{B}{\mathrm{i}}_{2}\mathrm{T}{\mathrm{e}}_{3}$, highlighting its crucial role in reducing lattice thermal conductivity. Our experimental results suggest that appropriate concentration of substituted material and specific site substitution can enhance phonon scattering processes, leading to improved thermoelectric performance.