A robotic arm designed for automated fruit harvesting began systematically damaging tree trunks on a smart farm. The robot's stereo vision system, responsible for calculating the exact distance to branches, critically failed. During the technical inspection, it was identified that the disparity map generated by the cameras contained optical artifacts. The goal of the forensic analysis was to determine whether the reflection of sunlight on leaves had deceived the depth detection algorithm.
Forensic analysis of the disparity map with MATLAB 🛠️
The first step of the inspection involved extracting raw frames from the robot's stereo cameras. Using MATLAB and its Computer Vision Toolbox, the scene's disparity map was reconstructed. The results showed regions with anomalous disparity values, especially in areas where leaves exhibited intense brightness. The solar reflection generated saturated pixels that the stereo correspondence algorithm interpreted as objects at a much shallower depth than reality. This caused the robotic arm, when calculating the grasping trajectory, to program a gripper movement too close to the trunk, impacting the bark. With SolidWorks, the arm's kinematics were modeled, and the geometry was exported to MeshLab to clean the mesh at the impact points, confirming the systematic collision at the erroneous coordinates.
Practical solutions and simulation in Unity 🎯
To prevent future incidents, two correction lines are proposed. The first is the installation of polarizing filters on the stereo camera lenses to reduce specular reflections from the sun. The second, more advanced, involves implementing a monocular depth neural network as a backup, trained to ignore brightness artifacts. To validate these solutions, the complete scenario was recreated in Unity, simulating dynamic sunlight and the robotic arm's behavior. The simulation demonstrated that, with the virtual polarizing filter activated, the disparity map remained stable, and the robot harvested the fruit without damaging the trunk. This case study reinforces the need to integrate robust vision systems against adverse environmental conditions in agricultural robotics.
How can a 3D inspection identify whether the depth error in a harvesting robot is due to a defective LiDAR sensor calibration or an erroneous interpretation of the trunk geometry in the computer vision model
(PS: Simulating robots is fun, until they decide not to follow your orders.)