Science-T2I: Addressing Scientific Illusions in Image Synthesis

This paper presents a novel approach to integrating scientific knowledge into generative models, enhancing their realism and consistency in image synthesis. We present SciScore, an end-to-end reward model that refines the assessment of generated images based on scientific knowledge, which is achieved by augmenting both the scientific comprehension and visual capabilities of pre-trained CLIP model. We also introduce SciBench, an expert-annotated adversarial dataset comprising 30k image pairs with 9k prompts, covering wide distinct scientific knowledge categories. Leveraging SciBench, we propose a two-stage training framework, comprising a supervised fine-tuning phase and a masked online fine-tuning phase, to incorporate scientific knowledge into existing generative models. Through comprehensive experiments, we demonstrate the effectiveness of our framework in establishing new standards for evaluating the scientific realism of generated content. Specifically, SciScore attains performance comparable to human-level, demonstrating a 5% improvement similar to evaluations conducted by experienced human experts. Furthermore, by applying our proposed fine-tuning method to FLUX, we achieve a performance enhancement exceeding 50% based on SciScore.