PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy

Abstract

Endoluminal endoscopic imaging—used in diagnosing colorectal cancer and other internal diseases—stands to benefit greatly from accurate 3D reconstructions and novel-view synthesis. However, current methods struggle with artifacts due to constrained camera trajectories and lighting effects that depend heavily on view, often overfitting and failing when viewing from novel angles.

PR-ENDO presents a new reconstruction framework specifically for endoscopic scenes, leveraging the fact that in such imaging the light source is usually co-located with the camera. The core idea is to separate light effects from intrinsic tissue properties, using an enhanced physically-based relighting model built on 3D Gaussian Splatting (3DGS). In addition to the usual geometry (shape) and color (albedo), PR-ENDO models materials with parameters like roughness and Fresnel coefficients, and learns a diffuse correction module via a small multi-layer perceptron (MLP) to better approximate how light interacts with tissue surfaces. This helps mitigate artefacts like specular highlights and inconsistent shading that tend to mislead simpler models.

To train the system, the authors incorporate several loss terms: those enforcing consistency of geometry, diffuse lighting behaviour, and separation of lighting from tissue texture (so albedo is not “baked in”). They optionally use depth maps or SLAM-based reconstructions for initial geometry regularization. The paper demonstrates that PR-ENDO outperforms baseline methods in both public and private datasets (including the C3VD dataset), improving novel view rendering and reducing common artifacts.

Importantly, PR-ENDO allows for tissue modifications and realistic relighting under novel lighting/viewpoint conditions, making it better suited for simulation, training, and potentially real-clinical usage. While some limitations remain — such as dependence on reasonably good depth estimation and quasi-rigid tissue motion — PR-ENDO represents a strong step toward more physically faithful, relightable endoscopic 3D reconstructions.

Paper: Click here to read