Self-Assessment of Robotic Laboratory and Equipment Readiness Using Large Language Models and Robotic Data Capture

This study explores the potential of automating robotic laboratory readiness assessment by integrating Large Language Models (LLMs) with robotic data acquisition. It investigates the capability of LLMs to detect equipment motion and operational status using visual and auditory information. Despite the challenges LLMs face in spatial analysis, this study also investigates LLM grounding methods to ensure accurate workspace assessment. By inspecting a robotic cooking setup with camera-equipped robotic arm, LLMs can detect the motion of custom equipment via color-coded marks, and identify the operational status of kitchen appliances from a single image without any physical augmentations. Additionally, device operation perceived through the emission of loud noises can be assessed by post-processing sound recordings and analyzing loudness and sound frequency metrics presented in a visual plot form. For simple spatial tasks like saucepan positioning, LLM provides accurate assessments when grounded with a single image, while complex workspace safety assessment task requires extensive knowledge of past experiences. By reviewing status of each checklist item, the LLM can decide whether experiment needs to be halted or requires human intervention, offering a set of troubleshooting steps. These findings demonstrate feasibility of the self-assessment approach for robotic laboratory systems, paving the way for future deployments.