Enhancing SAM's Biomedical Image Analysis through Prompt-based Learning

The Segment Anything Model (SAM), a foundational model trained on an extensive collection of images, presents many opportunities for diverse applications. For instance, we employed SAM in our biological pathway curation pipeline that synergizes image understanding and text mining techniques for deciphering gene relationships. SAM has proven highly efficient in recognizing pathway entities and their interconnections. However, SAM does not work well when applied to low-contrastive images directly. To counter this, we investigated prompt-based learning with SAM, specifically for identifying proteins from cryo-Electron Microscopy (cryo-EM) images. We trained a U-Net-based filter to adapt these grayscale cryo-EM images into RGB images suitable for SAM's input. We also trained continuous prompts and achieved state-of-the-art (SOTA) performance, even with a limited quantity of labeled data. The outcomes of our studies underscore the potential utilities of prompt-based learning on SAM for efficient biomedical image analyses.