Skip to yearly menu bar Skip to main content


Prompt Highlighter: Interactive Control for Multi-Modal LLMs

Yuechen Zhang · Shengju Qian · Bohao Peng · Shu Liu · Jiaya Jia

Arch 4A-E Poster #345
[ ] [ Project Page ]
Thu 20 Jun 10:30 a.m. PDT — noon PDT


This study targets a critical aspect of multi-modal LLMs' (LLMs\&VLMs) inference: explicit controllable text generation. Multi-modal LLMs empower multi-modality understanding with the capability of semantic generation yet bring less explainability and heavier reliance on prompt contents due to their autoregressive generative nature. While manipulating prompt formats could improve outputs, designing specific and precise prompts per task can be challenging and ineffective. To tackle this issue, we introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. Motivated by the classifier-free diffusion guidance, we form regular and unconditional context pairs based on highlighted tokens, demonstrating that the autoregressive generation in models can be guided in a classifier-free way. Notably, we find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs. Our approach is compatible with current LLMs and VLMs, achieving impressive customized generation results without training. Experiments confirm its effectiveness in focusing on input contexts and generating reliable content. Without tuning on LLaVA-v1.5, our method secured 70.7 in the MMBench test and 1552.5 in MME-perception. Code is available at

Live content is unavailable. Log in and register to view live content