A Visual Sensor-based Approach for Robotic Pick-and-Place Operations

An autonomous robotic grasping system aims to use sensor information to allow real-time object detection and manipulation. This research focuses on integrating computer vision with robotic control to accurately pick objects using the Interbotix X-Series ViperX 300 robot arm. Equipped with a RealSense D435i depth camera and a YOLOv11-based object detection model, the system enables adaptive picking without relying on predefined object locations. Autonomous grasping is relevant in various applications, especially in unstructured environments where objects are not placed in predefined locations. This project uses a combination of real-time object detection, depth-based 3D localization, and dynamic robot path planning. The RealSense camera provides color images and depth data. A custom-trained YOLOv11 model recognizes objects in color images. Those image positions are then translated to real-world 3D positions using the depth data. The ViperX 300 robotic arm then executes grasping maneuvers based on this object position, as well as object geometry and orientation. The system continuously adapts to object movement, ensuring a higher success rate in picking and placing compared to traditional approaches reliant on fixed coordinates. This research is related to many applications, including industrial automation, where it enhances pick-and-place operations in warehouses and factories. In assistive robotics, autonomous robots can help individuals with mobility impairments by retrieving objects. Space robotics also benefits from this technology, enabling robotic arms on spacecraft to interact with tools and objects in microgravity environments. Additionally, in agriculture, robots can automate selective harvesting based on real-time object detection. The ability to dynamically identify and grasp objects in varying conditions is significant in robotic manipulation. The development process involved multiple stages, beginning with dataset collection and model training. A custom dataset was created using labeled images of target objects, with data augmentation techniques applied to improve generalization. The YOLOv11 model was then trained to recognize these objects with high accuracy. Simultaneously, the RealSense D435i depth camera was calibrated to ensure precise 3D localization, allowing the system to determine object position effectively. A necessary step in this process was performing a transformation to convert coordinates from the camera frame to the robot's coordinate system, ensuring accurate object positioning for grasp execution. The robotic control system was implemented using ROS 2 Humble, enabling real-time communication between perception and motion planning modules. Grasping strategies were optimized based on object characteristics, incorporating real-time adjustments to improve success rates. Testing and validation demonstrated significant improvements in grasping accuracy and adaptability. By integrating segmentation masks, object detection precision was enhanced, reducing errors in depth estimation and grasp planning. Instance segmentation goes beyond standard object detection by not only identifying objects but also generating masks that outline their exact shape. These segmentation masks provide more precise localization by isolating objects from the background, helping the system determine their exact edges and contours. This improved precision reduces miscalculations in depth estimation and allows for more accurate grasp planning, as the robotic arm can align its end-effector with the actual shape of the object rather than relying solely on bounding box approximations. The system successfully adjusted to object movement, demonstrating the ability to react dynamically to changes in position and orientation. Efficiency gains were observed, with the system achieving a higher grasp success rate and reduced execution time compared to conventional methods. These findings highlight the potential of combining deep learning-based vision systems with robotic manipulators to create more autonomous and reliable grasping solutions. Future improvements will focus on increasing dataset diversity, refining grasp planning algorithms, and expanding functionality to multi-object interactions, further advancing robotic autonomy in real-world applications.