SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Driving

Retrieving rare and safety-critical driving scenarios from large-scale datasets is essential for building robust autonomous driving (AD) systems. As dataset sizes continue to grow, the key challenge shifts from collecting more data to efficiently identifying the most relevant samples.We introduce SearchAD, a large-scale rare image retrieval dataset for AD containing over 423k frames drawn from 11 established datasets. SearchAD provides high-quality manual annotations of more than 386k bounding boxes covering 64 rare categories. It specifically targets the “needle-in-a-haystack” problem of locating extremely rare classes, with some appearing fewer than 50 times across the entire dataset. Unlike existing benchmarks, which focused on instance-level retrieval, SearchAD emphasizes semantic image retrieval with a well-defined data split, enabling text-to-image and image-to-image retrieval, few-shot learning, and fine-tuning of multi-modal retrieval models.Comprehensive zero-shot evaluations show that text-based methods outperform image-based ones due to stronger inherent semantic grounding. Models that directly align spatial visual features with language achieve the best relative results, yet none demonstrate satisfactory retrieval capability in absolute terms. With a held-out test set on a public benchmark server, SearchAD establishes the first large-scale benchmark for retrieval-driven data curation and long-tail perception research in AD.