Abstract: Most full-reference image quality assessment (FR-IQA) models assume that the reference image is of perfect quality. However, this assumption is flawed because many reference images in existing IQA datasets are of subpar quality. Moreover, recent generative image enhancement methods are capable of producing images of higher quality than their original counterparts. These factors challenge the effectiveness and applicability of current FR-IQA models. To address this limitation, we build a large-scale IQA database, namely DiffIQA, which contains approximately 180,000 images generated by a diffusion-based image enhancer with adjustable hyper-parameters. Each image is annotated by human subjects as either worse, similar, or better quality compared to its reference. Building on this, we present a generalized FR-IQA model, namely $\textbf{A}$daptive $\textbf{FI}$delity-$\textbf{N}$aturalness $\textbf{E}$valuator (A-FINE), to accurately assess and adaptively combine the fidelity and naturalness of the test image. A-FINE aligns well with standard FR-IQA when the reference image is much more natural than the test image. We demonstrate by extensive experiments that A-FINE surpasses existing FR-IQA models on well-established IQA datasets and our newly created DiffIQA. To further validate A-FINE, we additionally construct a super-resolution IQA benchmark (SRIQA-Bench), encompassing test images derived from ten state-of-the-art SR methods with reliable human quality annotations. Tests on SRIQA-Bench re-affirm the advantages of A-FINE. Our data, code and models will be made publicly available.