Drug Discovery Agent: An Automated Vision Detection System for Drug-Cell Interactions

We introduce a vision agentic model for detecting drug-cell interactions in microscopy that can operate in real-time to reduce the time needed for drug discovery. Instead of requiring task-specific training or fine-tuning, or adaptation of the training data to usage, our approach is a prompt-driven AI agent to detect and classify phenotypic changes (think of machine vision, tie it in with drug discovery) in cells to new drugs. This addresses a key limitation of state-of-the-art (SOTA) deep learning models like YOLO v8/v12, SAM 2, Vision Transformers (ViTs), CLIP, and ConvNeXt, that typically require significant amounts of labeled data for rehabilitation of previously trained models for new experiments. We empirically demonstrate that our model achieves at least comparable or superior accuracy to SOTA supervised models, while processing at real-time-based speeds, to evaluate on the BBBC021 and BBBC022 high-content imaging datasets, and on a collection of live-cell YouTube videos. We also demonstrate the proposed agentic detector as a model works more effectively than conventional models because it can efficiently generalize to new cell types to be treated with no retraining of the model nor original data collection. We also show significant proactive capabilities from efficiency (inferring at dozens of frames per second) and robustness to dataset shifts. The results reveal that our approach not only achieves SOTA accuracy in drug mechanism-of-action discovery, but provides unprecedented flexibility and speed, which may signal a new paradigm of AI-driven phenotypic screening in drug discovery. The code is available at https://github.com/adibgpt/Drug-Discovery-Agent.