Skip to yearly menu bar Skip to main content


Polos: Multimodal Metric Learning from Human Feedback for Image Captioning

Yuiga Wada · Kanta Kaneda · Daichi Saito · Komei Sugiura

Arch 4A-E Poster #377
award Highlight
[ ] [ Project Page ]
Thu 20 Jun 10:30 a.m. PDT — noon PDT

Abstract: Establishing an automatic evaluation metric that closely aligns with human judgments is essential for effectively developing image captioning models. Recent data-driven metrics have demonstrated a stronger correlation with human judgments than classic metrics such as CIDEr; however they lack sufficient capabilities to handle hallucinations and generalize across diverse images and texts partially because they compute scalar similarities merely using embeddings learned from tasks unrelated to image captioning evaluation.In this study, we propose Polos, a supervised automatic evaluation metric for image captioning models.Polos computes scores from multimodal inputs, using a parallel feature extraction mechanism that leverages embeddings trained through large-scale contrastive learning.To train Polos, we introduce Multimodal Metric Learning from Human Feedback (M$^2$LHF), a framework for developing metrics based on human feedback. We constructed the Polaris dataset, which comprises 131K human judgments from 550 evaluators, which is approximately ten times larger than standard datasets. Our approach achieved state-of-the-art performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and the Polaris dataset, thereby demonstrating its effectiveness and robustness.

Live content is unavailable. Log in and register to view live content