Skip to yearly menu bar Skip to main content


GROUNDHOG: Grounding Large Language Models to Holistic Segmentation

Yichi Zhang · Ziqiao Ma · Xiaofeng Gao · Suhaila Shakiah · Qiaozi Gao · Joyce Chai

Arch 4A-E Poster #442
[ ] [ Project Page ]
Thu 20 Jun 10:30 a.m. PDT — noon PDT


Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM developed by Grounding Large Language Models to holistic Segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated a grounded visual instruction tuning dataset - Multi-Modal Multi-Grained Grounding (M3G2) - by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning. GROUNDHOG demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.

Live content is unavailable. Log in and register to view live content