How we approach it
Object-decoupled Gaussian Splatting
Vanilla Gaussian Splatting captures how a scene looked, but the resulting splats have no notion of objects — every Gaussian belongs to one anonymous global point cloud. Our work treats object decoupling as a research problem in its own right: splitting Gaussian-Splatted scenes into a static background and a set of individually addressable object splat sets that can be transformed, re-lit, and physically simulated as if they were rigid bodies.
We pair this with a small but deliberate engineering choice — keeping each object's splats in their own group and exposing those groups as first-class entities to the renderer. That makes the rest of the stack possible: per-object physics, per-object fine-tuning, and predictive completion of surfaces no camera ever observed.
Optimization that holds up under decoupling
A Gaussian Splatting scene in which some objects are designated movable rigid bodies cannot be trained with the off-the-shelf optimizer. Object and background splats compete near boundaries; fine-tuning a localized region tends to degrade global PSNR; and the renderer alone provides no signal for distinguishing genuine surface coverage from floaters or holes. The lab's optimization work targets these failure modes with specialized strategies: alternating two-phase training that separates global structural updates from local rigid-body refinement, hull-based selective fine-tuning that restricts local updates to splats inside an object's geometric hull, and explicit object–background separation losses (repulsion and adoption regularization) that recover clean shared boundaries.
Mobile XR deployment
Most Gaussian Splatting research stops at desktop visualization. Putting decoupled splats on a head — with rigid-body physics, controller interaction, and headset frame rates on a mobile-class device — is a different engineering problem than rendering on a workstation GPU. Our runtime is built on the open-source aras-p Gaussian Splatting framework for Unity, extended with a native CUDA plugin architecture that exposes per-object GPU transforms to the engine's scene graph.
Physics integration runs through Meta Quest 3 / OpenXR; controller-triggered grab, drop, and throw all flow through Unity's existing rigid-body system. We also explore foveated rendering as a frame-budget lever — concentrating splat-shading effort where the user is actually looking and letting peripheral regions render at lower fidelity. The whole stack is built on actively-maintained open-source components (COLMAP, SAM2, LaMa, gsplat, aras-p), keeping the workflow reproducible and lettings each contribution slot back into the wider Gaussian Splatting ecosystem.
