SegCTC: Offline Handwritten Chinese Text Recognition via Better Fusion Between Explicit and Implicit Segmentation

Handwritten Chinese text recognition (HCTR) is still a challenging and unsolved problem. The existing recognition methods are mainly categorized into two: explicit vs implicit segmentation-based methods. Explicit segmentation recognition methods use explicit character location information to train the recognizers. However, the widely used weakly supervised training strategy based on pseudo-label makes it difficult to get effective supervised training for difficult character samples. In contrast, the implicit segmentation recognition method use all transcript annotations for supervised training, but it is prone to misalignment problem due to the lack of explicit supervised information of character positions. To take advantage of the complementary nature of explicit and implicit segmentation approaches, we propose a new method, SegCTC, which better integrates these two approaches into a unified to be a more powerful recognizer. Specifically, we designed a hybrid Segmentation-based and Segmentation-free Feature Fusion Module (S $$^2$$ FFM) to better fuse the features of both explicit and implicit segmentation-based recognition branches. Moreover, a co-transcription strategy is also proposed to better combine the predictions from different branches. Experiments on four widely used benchmarks including CASIA-HWDB, ICDAR2013, SCUT-HCCDoc and MTHv2 show that our method achieves state-of-the-art performance for the HCTR task under different scenarios.