FOUND

Ontological Admission Control for Incremental 3D Scene Graphs

Anonymous submission (paper under double-blind review)

Abstract

Semantic maps are built by accumulation: whatever the perception stack proposes is written into the map, and the only operation able to decline a proposal is data association, which declines solely for duplication. Running an open-vocabulary scene-graph pipeline on a simulated apartment, we observe a map of 2710 entries for 119 objects, including chairs of 4×10−4 m3; none of these entries is a detection failure in the detector's own terms. We present FOUND, an admission stage placed at the boundary of an incremental 3D scene graph. Each proposal is aligned to a class of the robot's ontology and checked against class-level expectations of physical extent and room compatibility; the expectations are populated from measured product corpora, asserted in the ontology's own vocabulary, and carry provenance per statement. The verdict is graded: a proposal is admitted, held as an unconfirmed entity outside the tracked map, declined with only the decision recorded, or returned as a gap when no grounds exist. We describe the integration with an incremental 3D scene graph exposed through a REST API and report a first evaluation on Matterport scenes, with instance-level ground truth used only to score verdicts and never to make them.

Video

Architecture

FOUND architecture
Perception proposes object candidates; association and tracking decide which change each one implies. Before any candidate reaches the graph, the admission stage aligns its label to an ontology class and checks it against the expectations that class indexes. Only admitted candidates enter the map: held ones wait in a pool outside it, and declined ones leave a decision behind and nothing else.

What it does

A semantic map is built by accumulation: whatever perception proposes is written in. Data association is the only operation that can decline a proposal, and it declines for one reason only, that the proposal is already present. Existing maps can refuse a proposal, but not on grounds the robot's own ontology can state and justify, not by declining to rule when it cannot, and not by taking back what it already admitted.

FOUND adds that operation, at the boundary of an incremental 3D scene graph (a component of LOST-3DSG: LOST builds the semantic map; FOUND is the stage that decides what is allowed into it). Each proposal is checked against what the ontology says its class can dimensionally be, and the verdict is graded:

VerdictMeaning
admitconsistent with what is known; written into the tracked map
holdkept as an unconfirmed entity outside the tracked map
declineviolates an expectation outright; only the decision is recorded, never the object
no groundsthe gate lacks the class or the measurements; the gap itself is returned

no grounds is what separates this from thresholding. A gate with an answer for every label would be a table of tuned constants.

Align first, then measure

A detector emits a string, and a string is not a class. "couch", "sofa" and "settee" name one kind of thing and an open-vocabulary detector will produce all three, so the instance is aligned to a class of the robot's ontology before any dimensional question is meaningful.

The dimensions are then looked up, not authored. They come from measured product corpora (Amazon Berkeley Objects listings: 147k product records, 41,489 of them carrying three usable physical dimensions), aggregated into a 1st-to-99th percentile envelope per class and asserted in the ontology's own vocabulary with provenance per statement. The chair envelope rests on 1,654 measured chairs.

This is the difference from asking a language model how big a chair is. An elicited prior cannot be checked; an envelope over measured objects is a fact, and the sample count travels with it so a thin envelope can be distrusted.

Envelopes are order-independent. Object pose is only partially recovered by the perception stack, so a rotated chair swaps its horizontal axes; records and proposals are both sorted into (smallest, middle, largest) before comparison. Comparing named axes would reject correctly-sized but rotated objects.

Abstention means the instance could not be aligned, or that its envelope rests on too few samples to trust. It is a statement about the evidence, not about a missing rule.

How it is built

The scene graph is organised in layers over a metric substrate produced by RTAB-Map: agent poses, objects, free-space regions from an approximated Generalized Voronoi Diagram, rooms, and the building at the root. Graph operations are ROS services exposed through a REST API, so perception, graph management and admission can each run on the robot or on a remote machine.

StageWhat runs
Labels and attributesa multimodal vision-language model, per frame and open-vocabulary
GroundingYOLO-World-L, from the labels the model proposed
Masks and extentsSAM 2.1, with PCA over the segmented points for orientation
Associationall-MiniLM-L6-v2 sentence embeddings; description 0.50, colour 0.30, material 0.15, label 0.05, renormalised when an attribute is missing on either side
Alignmentlabel to ontology class, abstaining rather than forcing a placement
Size checkone interval per sorted axis; at least 30 measured samples before it is used numerically, otherwise a coarse band
Room checkclass-to-room-type prior from an annotated RGB-D corpus, advisory: a conflict can only move a verdict to borderline
Recordevery decision stored under its own identifier with the conditions that held, the bound cited and the corpus behind it

Nothing the gate uses is a constant in code. The criteria in force, their parameters and the corpus each was derived from are declared as a policy, and a copy of that policy is filed with every decision, so asking why an object was refused and under which version of which expectation is a query rather than an exercise in reading the source.

One run, seen as a graph

The admission record of a single twenty-minute run on a Habitat-Matterport scene. Every proposal the gate judged is a node, joined to the ontology class its label was aligned to. Only admitted objects are joined to the room, because only they reached the map. Click a node to see the verdict and the bound that produced it.

Bird's-eye view of the traversed floor: light grey is where the robot can walk, dark is everything else. Each dot is a proposal, placed where it was seen and coloured by verdict, so furniture against a wall sits off the walkable area by construction. Selecting here selects it in the graph, and the other way round.

admitted held declined no grounds ontology class room
Per distinct object
Objects judged54
Let in47.4%
Set aside as borderline611.1%
Refused3463.0%
No grounds to judge1018.5%
Real objects refused8

Eight refusals removed a real object, and that is the cost the design has to answer for. All eight were judged against their own class rather than a broader ancestor, so they are failures of measurement rather than of alignment: the same bed measured between 0.698 and 1.059 m on its smallest axis across five viewpoints, with the limit that decides its fate falling inside that spread. One case shows the graded verdict earning its place, a bed 2.7 cm below its lower limit set aside rather than refused, kept off the map and kept available.

Extended results: ontology properties and merging

Measured in October 2026 under the adopted ontology set (SOMA, SOMA-HOME, DUL, DogOnt and Wikidata items, stitched by equivalence and subclass mappings; no class is minted). The figures elsewhere on this page were measured before that change and are not comparable with the figures in this section.

The admission stage now does more than judge a proposal. After alignment, FOUND collects the properties that the stitched ontologies declare for the class, ranks them by relevance on the class–property lattice, and keeps the most relevant ones until their cumulative relevance reaches 0.8, so the number of properties per object depends on relevance. A vision-language model fills each selected property from the object's image crops at first observation, at each re-observation and after each merge. It abstains below a confidence of 0.65, and the value of an object property must be an existing node. Every fill is written to RDF with its provenance.

The filled properties feed a second FOUND component: a merge scorer. When geometry proposes that two objects are the same, the scorer compares them with a situated similarity operator. Asserted disjointness vetoes the merge; otherwise the shared properties, the box dimensions and the room are weighted by diagnosticity, relevance and the evidence that each object carries. A missing property removes weight instead of counting as a mismatch, and the scorer abstains when coverage is below 0.5. The decision band (same at 0.7 or above, different at 0.3 or below) is declared, not calibrated.

Architecture: the scene-graph stack (a-c) and FOUND (d)
(a)–(c): the scene-graph stack. (d): FOUND, outside the stack, connected through its proposal and merge hooks. It admits proposals, fills the properties that the stitched ontologies declare, scores merges, and stores every verdict, fill and merge decision with its provenance.

Three-arm ablation

Both storeys of one simulated house (HM3D scene 00861), one fixed tour per storey, three arms: (1) the original scene-graph pipeline, whose describer attaches colour, material and description; (2) FOUND properties instead of the describer attributes, with merges decided by geometry only; (3) FOUND properties, with each merge decided by the FOUND merge scorer. Every verdict was logged at first observation. The logged verdicts are scored offline against the scene's ground-truth object inventory, which FOUND never reads.

ArmStoreyObjectsProposalsCoveredPrecision beforePrecision afterWrong rejectionsMergesFills
1 baseline−1.59 m2613472560.2380.2393086–
+1.21 m3404573460.3350.30859116–
2 FOUND properties, geometric merges−1.59 m1132602070.2510.28618147520
+1.21 m84109890.4160.5001524185
3 FOUND properties and scorer−1.59 m1181581240.2900.4411038568
+1.21 m2252922480.3630.4103223499

Covered: proposals with a grounded verdict (admit, hold or decline). Precision before: real covered proposals over covered proposals. Precision after: real admitted proposals over admitted proposals. Wrong rejections: declined proposals that were real.

Pooled over both storeys, precision after filtering is 0.28 (58 of 209 admitted) for the baseline, 0.31 (30 of 96) for arm 2 and 0.42 (58 of 139) for arm 3. The baseline gate declines half of the real covered proposals (89 of 177); with FOUND properties and the scorer, one third (42 of 126). FOUND filled 1,067 properties in arm 3, from SOMA (46%), Wikidata (36%), DogOnt (10%) and DUL (9%). The scorer returned same 61 times and abstained in most comparisons, mainly because one of the two objects had no published fill yet.

Read with these limits. One scene and one run per arm, so there are no confidence intervals. The admission verdict is fixed at first observation, so the arms differ through the proposals that each association policy produces, not through properties filled later. Arm 2 also relaxes the stack's evidence threshold for association, which changes tracking as well and leaves it with fewer objects; its 0.500 on the upper storey rests on 12 admitted proposals. The baseline ran two days earlier with different transport settings. A first attempt at arm 3 stopped on a full command queue and is excluded.

The runs, frame by frame

Arm 3 at run time
Arm 3 (FOUND properties and scorer) during the tour of the upper storey: detections, FOUND property fills as they are committed, the robot path with proposals coloured by verdict, and the latest objects in the graph with their aligned classes and filled properties (admitted solid, held dashed).
Arm 1 at run time
Arm 1 (baseline) on the lower storey, for comparison: the same views, with the verdicts FOUND logged but no property filling.