STTRL-DVO: Transformer-based Reinforcement Learning for Robust Dynamic Target Tracking in Cluttered Environments

Magnetic microrobots capable of autonomous operation hold significant promise for critical cell and small creature manipulation tasks, including trapping, transportation, sorting, etc. Although conventional microrobot navigation methods have shown notable performance, they often lack adaptability to novel environments. To address these limitations, we propose a learning-based framework for real-world microrobot navigation and dynamic target tracking. Our approach employs spatial-temporal transformer reinforcement learning (STTRL) with a deterministic velocity obstacle (DVO) that processes historical navigation states and virtual light detection and ranging (LiDAR) scans to predict optimal actions. The key innovation lies in the model's ability to extract and utilize contextual information from observation histories, enabling adaptive behavior even in previously unseen environments. Through large-scale model-free reinforcement learning trained in randomized simulation environments, our method achieves remarkable real-world performance with zero-shot transfer capability. Experimental results demonstrate superior navigation agility with an 89.8% success rate in the base environment, representing a 7.4% improvement over state-of-the-art (SOTA) algorithms. Furthermore, the method exhibits robust generalization in diverse unseen environments, validating its adaptability to different environmental characteristics.