Second language (L2) utterance fluency is crucial for speaking-proficiency assessment. The measurement of L2 utterance fluency relies heavily on silent pause identification. However, empirical studies establishing specific silent pause thresholds for L2 monologic speaking are scarce, and even fewer exist for L2 dialogic speaking. This study thus aims to explore the optimal silent pause threshold for measuring L2 utterance fluency in both speaking contexts. Following De Jong and Bosker, the study assumes that silent pause measures calculated at optimal thresholds can best predict L2 proficiency and perceived fluency. Multiple linear regression models were conducted using four silent pause measures (mid-Analysis of Speech Unit [ASU] silent pause rate, end-ASU silent pause rate, mid-ASU silent pause duration, end-ASU silent pause duration) at various thresholds (100–1000 ms) to predict participants’ L2 proficiency and perceived fluency scores. For the monologic task, silent pause measures calculated at a threshold of 200 ms best predicted both L2 proficiency and perceived fluency scores. For the dialogic task, silent pause measures calculated at a 200-ms threshold had the best predictive power for L2 proficiency score, whereas the optimal threshold for predicting the perceived fluency score was 350 ms.