Beyond Homogeneous Adaptation: Decoupled Control for Sample- and Layer-Aware Test-Time Robustness in Temporally Correlated Streams

Test-Time Adaptation (TTA) enables pre-trained models to adapt online to distribution shifts during inference, but its self-training dynamics are fragile under temporally correlated streams. Once biased target evidence is used for adaptation, the resulting errors can accumulate over time, distort target representations, and persist unevenly across network layers. Existing methods typically mitigate this problem with confidence filtering or global parameter regularization, but these homogeneous controls overlook two forms of heterogeneity: target samples differ in whether they provide trustworthy adaptation evidence, and layers differ in whether they can safely absorb the induced updates. We propose DCF, a Decoupled Control Framework that stabilizes TTA by jointly controllingwhich samples drive adaptation and which layers retain the resulting update. At the sample level, DCF uses Probe-supported Sample Routing (PSR) to identify probe-supported trusted evidence via entropy and a Fourier stress response, while Routed-away Geometry Repair (RGR) reuses routed-away samples for optimal-transport-based geometry repair instead of discarding them. At the parameter level, the sample-side update is treated as only a candidate update: Curvature-aware Layer Retention (CLR) then evaluates its layer-wise source compatibility, retaining useful plasticity and anchoring high-mismatch layers to the source model. This route–adapt–retain design prevents unreliable evidence from dominating adaptation while avoiding indiscriminate rollback of useful plasticity. Extensive experiments on temporally correlated corruptions, natural shifts, and long-horizon streams demonstrate that DCF consistently improves test-time robustness and avoids the late-stage collapse of existing TTA methods. Code is available at https://github.com/ioslide/DCF.