Learning-Based Auto-Focus and 3D Pose Identification of Moving Micro-and Nanowires in Fluid Suspensions

Precise manipulation of micro-and nano-objects through visual feedback is challenging because of the difficulty of observing their motion along the line-of-sight of microscopes. This paper presents an efficient learning-based auto-focus (AF) and visual posture estimation scheme for tracking the three-dimensional (3D) poses of multiple moving micro-and nanowires in fluid suspensions under bright-field microscopes. The proposed AF and 3D pose estimation methods integrate convolutional neural networks (CNNs) to precisely identify the focal distances and inclination angles of multiple moving wires through a single region-of-interest (ROI) image for each wire. Furthermore, we demonstrate the versatility of the proposed AF method by adapting it for wires of other materials through transfer learning (TF), using a limited dataset. Extensive experimental results validate the high accuracy and efficiency of AF and 3D pose estimation compared to traditional methods. This work lays the foundation for the automated control of micro-and nano-objects in 3D microfluidic environments. Note to Practitioners —Autonomous manipulation of multiple micro-and nanoscale objects is of major interest for various research applications. However, precise manipulation of micro-and nanowires through visual feedback is challenging because of the difficulty in focusing microscopes on their 3D pose or orientation to the x, y, and z axes as they move in the depth direction. Traditionally, achieving AF involves using one or more image metrics to evaluate image quality by performing a series of mechanical movements to locate the peak sharpness using optimization methods. In contrast, this paper proposes a learning-based approach that uses CNNs to estimate the positions of the focal planes and identify the 3D positions and orientations of multiple moving micro-and nanowires from a single ROI image. Extensive experiments demonstrate that the learning-based methods exhibit higher stability, extended duration, and faster converging speed of tracking and 3D pose estimation compared to traditional methods. This work facilitates researchers' ability to observe and manipulate micro-and nano-objects under microscopes in 3D microfluidic environments.