Metamorphic Testing of Machine Translation Models using Back Translation

Machine translation software has been widely adopted in recent years. The recent advance in deep learning research has massively improved the accuracy and fluency of the translated output. However, incorrect translations may still occur, which causes misunderstandings, and even more detrimental consequences when applying these systems for crucial applications, such as translating legal and medical documents. This calls for methods that can test the correctness of machine translation software efficiently and effectively. In this paper, we propose a method that uses back-translation as a reference for machine translation testing, minimizing the knowledge and use of the NLP tools in the target language so that the same workflow can be applied to test systems translating English to multiple languages. We build a metamorphic testing method using our proposed concept called contextual referentially transparent input (CRTI). A CRTI is a piece of text that should have a similar meaning under a certain context in any given language. Our method detects inconsistency between a CRTI in the original sentence and the back-translation to report translation errors. To evaluate our method, we translate 200 sentences using Google Translate. Our method reports 57 suspicious issues with a precision of 74% in Chinese translation and 22 suspicious issues with a precision of 82% in Vietnamese translation.