SVG-IR: Spatially-Varying Gaussian Splatting for Inverse Rendering

Reconstructing 3D assets from images, known as inverse rendering (IR), remains a challenging task due to its ill-posed nature and the complexities of appearance and lighting. 3D Gaussian Splatting (3DGS) has demonstrated impressive capabilities for novel view synthesis (NVS) tasks. It has also been introduced into relighting by decoupling radiance into Bidirectional Reflectance Distribution Function (BRDF) parameters and environmental lighting. Unfortunately, these methods often produce inferior relighting quality, exhibiting visible artifacts and unnatural indirect illumination. The main reason is the limited capability of each Gaussian, which has constant material parameters and normal, alongside the absence of physical constraints for indirect lighting. In this paper, we present a novel framework called Curved Gaussian Inverse Rendering (CG-IR), aimed at enhancing both NVS and relighting quality. To this end, we propose a new representation—Curved Gaussian (CG)—that generalizes per-Gaussian constant material parameters to allow for spatially varying parameters, indicating that different regions of each Gaussian can have various normals and material properties. This enhanced representation is complemented by a CG splatting scheme akin to vertex/fragment shading in traditional graphics pipelines. Furthermore, we integrate a physically-based indirect lighting model, enabling more realistic relighting. The proposed CG-IR framework significantly improves rendering quality, outperforming state-of-the-art NeRF-based methods by 2.5 dB in peak signal-to-noise ratio (PSNR) and surpassing existing Gaussian-based techniques by 3.5 dB in relighting tasks, all while maintaining a real-time rendering speed. We will release the code upon acceptance.