Toward Efficient Test Time Adaptation With Hierarchical Distribution Alignment

A model trained in a source domain often experiences a decline in effectiveness when deployed in a different target domain, primarily due to the discrepancies between the source and target domain characteristics. Test time adaptation (TTA) provides a practical solution for addressing the domain gap by adapting the models during the test phase. Existing TTA approaches mainly focus on aligning image features into a unified feature space. However, they generally only manage to achieve broad, coarse-grained alignment across domains while overlooking the more detailed, fine-grained feature clusters within each category. Furthermore, these methods are susceptible to settling at local optima because significant details can be lost when image features are abstracted into distribution parameters. To surpass these challenges, we introduce a novel approach that ensures hierarchical cross-domain alignment at three distinct levels: category-level, subcategory-level, and sample-level. Simple category-level alignment is inadequate due to the presence of various subcategories within each category, which possess distinct semantic properties identified through unsupervised clustering in our approach. Advancing further, we enhance our method by creating synthesized features from the initially extracted category-specific features, aiming for precise sample-level alignment. During our optimization process, we redefine TTA as essentially a feature matching problem, concentrating on the calculation of feature matching probabilities. Through hierarchical distribution alignment across these levels, our method maintains the semantic consistency of cross-domain image features from a broad to a detailed scale. Unlike prior test-time adaptation methods such as Tent, our method leverages source data only once after pre-training to fit feature distributions. During the testing phase, source data is completely discarded, and the model relies solely on test sample features. This design ensures privacy preservation and makes the method well-suited for privacy-sensitive applications. Our experimental evaluations on recognized datasets demonstrate that our approach significantly surpasses other established TTA methods in performance. Our code is accessible at https://github.com/yaboliudotug/HDA-TTA.