HKSR: A Research of Code-Switching Hong Kong Cantonese Speech Recognition Based on Multi-task Rescoring Strategy

Hong Kong Cantonese blends Cantonese and English due to its specific historical background and regional circumstance. This paper studies code-switching in Hong Kong-style Cantonese and proposes a Hong Kong Cantonese Speech Recognition (HKSR) method based on a multi-task rescoring strategy. Firstly, this work innovatively develops the Cantonese-English difference modelling unit to narrow modeling discrepancies between Cantonese and English, and simultaneously alleviate insufficient vocabulary issue caused by the lack of English in the data. Secondly, to better distinguish Cantonese from English, we construct a language identification(LID) subtask. Finally, to jointly train the LID and Automatic Speech Recognition(ASR), this paper develops a multi-task bilingual rescoring module based on U2 end-to-end model. We also investigate the impact of five different rescoring strategies, including multi-task bilingual rescoring, on Hong Kong Cantonese speech recognition. The experimental results demonstrate that HKSR combined with the multi-task bilingual rescoring strategy improves accuracy by 10%-49%