HFT6D: Multimodal 6D object pose estimation based on hierarchical feature transformer

Visual information is usually multimodal, including texture, color (2D information), and space (3D information). However, there are two problems in establishing multimodal 6D object pose estimation: (1) substantial differences between RGB images and depth data; (2) systematic noise in the depth images and lack contextual information in the association process. To solve the above problems, this paper proposes an end-to-end hierarchical feature transformer (HFT6D) containing four independent stages of crossmodal transformer. The novel hierarchical feature architecture suppresses the effect of noise by modeling the spatial correspondence between two different modalities. The core module of HFT6D is the bi-directional crossmodal attention, which aligns the appearance and geometric representation by recalibrating RGB-D data. In addition, our proposed HFT6D is real-time and achieves robustness against occluded scenes. Comprehensive experiments on two benchmark datasets show that HFT6D achieves state-of-the-art performance in terms of accuracy and speed.