An Open Source Computer Vision and Machine Learning Framework for Affordable Life Science Robotic Automation

A team of researchers has introduced a significant advancement in laboratory automation with a new open-source framework that combines computer vision and machine learning to control robotic systems. The work, led by Zachary Logan, Andrew Dudash, and Daniel Negrón, presents a complete software stack designed to perform delicate life science tasks such as microbial colony picking and liquid handling. At its core, the system uses a custom-trained U-Net model for semantic segmentation to accurately identify and locate colonies on a petri dish. For robotic control, it employs a Mixture Density Network (MDN), a type of neural network adept at predicting complex, multi-modal outputs, to calculate the inverse kinematics—translating a desired end-effector position into the precise joint angles needed for a 5-degree-of-freedom robot arm.

The framework was rigorously tested using a modified robot arm fitted with a custom liquid handling end-effector. The experimental results demonstrate its high precision and repeatability, with a mean positional error of less than 1 millimeter and joint angle prediction errors under 4 degrees. For the computer vision component, the model achieved an Intersection over Union (IoU) score of 0.537 and a Dice coefficient of 0.596 for colony detection, validating its practical utility. By being open-source, this framework directly challenges proprietary, high-cost laboratory automation systems, potentially democratizing access for academic labs, startups, and educational institutions. It provides a blueprint for building affordable, intelligent robotic assistants capable of executing repetitive and precise laboratory protocols.