New AI Can Pick Out Any Object in a 3D Scene

Imagine pointing your phone around a living room and getting back a walkable 3D copy of it. That's the promise of a technique called 3D Gaussian Splatting — think of it as building a scene out of millions of tiny coloured dots instead of flat polygons. It's already used in virtual property tours, film and game effects, and robot training. The catch: a 3D copy is only useful if the computer knows what's inside it. This dot is the sofa. That dot is the window.

That job is called segmentation — teaching a computer to draw a line around each object. Doing it at different zoom levels is hard. A chair looks different up close than from across the room, and most existing tools learn the scene's shape and its object labels separately. The geometry ends up ignoring meaning, and the labels lean on incomplete clues borrowed from 2D image-recognition models.

The team's fix, PePESeg3D, feeds "perception priors" — pre-trained knowledge about depth and object shapes — into both stages at once. Depth estimates and rough object outlines make the 3D structure itself meaningful, and dense colour-plus-depth cues fill in the gaps where the 2D labelling was patchy. Tested on three standard research datasets (SPIn-NeRF, LERF-Mask and NVOS), it beat earlier methods at both labelling objects and rebuilding the scene. The work was accepted to BMVC 2026, a computer vision conference.

The catch: this is research code, not a consumer app, and it still depends on those imperfect 2D models for its starting hints. Don't expect to tap and delete your furniture in a phone scan tomorrow. But the direction is clear. When computers genuinely understand what's inside a 3D scene, editing video, redesigning a kitchen, or training a warehouse robot gets dramatically easier — and cheaper.