Oral

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

Matt Deitke ⋅ Christopher Clark ⋅ Sangho Lee ⋅ Rohun Tripathi ⋅ Yue Yang ⋅ Jae Sung Park ⋅ Reza Salehi ⋅ Niklas Muennighoff ⋅ Kyle Lo ⋅ Luca Soldaini ⋅ Jiasen Lu ⋅ Taira Anderson ⋅ Erin Bransom ⋅ Kiana Ehsani ⋅ Huong Ngo ⋅ Yen-Sung Chen ⋅ Ajay Patel ⋅ Mark Yatskar ⋅ Chris Callison-Burch ⋅ Andrew Head ⋅ Rose Hendrix ⋅ Favyen Bastani ⋅ Eli VanderBilt ⋅ Nathan Lambert ⋅ Yvonne Chou ⋅ Arnavi Chheda-Kothary ⋅ Jenna Sparks ⋅ Sam Skjonsberg ⋅ Michael Schmitz ⋅ Aaron Sarnat ⋅ Byron Bischoff ⋅ Pete Walsh ⋅ Christopher Newell ⋅ Piper Wolters ⋅ Tanmay Gupta ⋅ Kuo-Hao Zeng ⋅ Jon Borchardt ⋅ Dirk Groeneveld ⋅ Crystal Nam ⋅ Sophie Lebrecht ⋅ Caitlin Wittlif ⋅ Carissa Schoenick ⋅ Oscar Michel ⋅ Ranjay Krishna ⋅ Luca Weihs ⋅ Noah A. Smith ⋅ Hannaneh Hajishirzi ⋅ Ross Girshick ⋅ Ali Farhadi ⋅ Aniruddha Kembhavi

Award Candidate

2025 Oral

Paper PDF

Abstract

Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present \textbf{Molmo}, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets, including a dataset of highly detailed image captions for pre-training called \textbf{PixMo}, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code will all be released.

Chat is not available.