Semantic maps are built by accumulation: whatever the perception stack proposes is written into the map, and the only operation able to decline a proposal is data association, which declines solely for duplication. Running an open-vocabulary scene-graph pipeline on a simulated apartment, we observe a map of 2710 entries for 119 objects, including chairs of 4×10−4 m3; none of these entries is a detection failure in the detector's own terms. We present FOUND, an admission stage placed at the boundary of an incremental 3D scene graph. Each proposal is aligned to a class of the robot's ontology and checked against class-level expectations of physical extent and room compatibility; the expectations are populated from measured product corpora, asserted in the ontology's own vocabulary, and carry provenance per statement. The verdict is graded: a proposal is admitted, held as an unconfirmed entity outside the tracked map, declined with only the decision recorded, or returned as a gap when no grounds exist. We describe the integration with an incremental 3D scene graph exposed through a REST API and report a first evaluation on Matterport scenes, with instance-level ground truth used only to score verdicts and never to make them.
A semantic map is built by accumulation: whatever perception proposes is written in. Data association is the only operation that can decline a proposal, and it declines for one reason only, that the proposal is already present. Existing maps can refuse a proposal, but not on grounds the robot's own ontology can state and justify, not by declining to rule when it cannot, and not by taking back what it already admitted.
FOUND adds that operation, at the boundary of an incremental 3D scene graph (a component of LOST-3DSG: LOST builds the semantic map; FOUND is the stage that decides what is allowed into it). Each proposal is checked against what the ontology says its class can dimensionally be, and the verdict is graded:
| Verdict | Meaning |
|---|---|
admit | consistent with what is known; written into the tracked map |
hold | kept as an unconfirmed entity outside the tracked map |
decline | violates an expectation outright; only the decision is recorded, never the object |
no grounds | the gate lacks the class or the measurements; the gap itself is returned |
no grounds is what separates this from thresholding. A gate with an answer for every label would be a table of tuned constants.
A detector emits a string, and a string is not a class. "couch", "sofa" and "settee" name one kind of thing and an open-vocabulary detector will produce all three, so the instance is aligned to a class of the robot's ontology before any dimensional question is meaningful.
The dimensions are then looked up, not authored. They come from measured product corpora (Amazon Berkeley Objects listings: 147k product records, 41,489 of them carrying three usable physical dimensions), aggregated into a 1st-to-99th percentile envelope per class and asserted in the ontology's own vocabulary with provenance per statement. The chair envelope rests on 1,654 measured chairs.
This is the difference from asking a language model how big a chair is. An elicited prior cannot be checked; an envelope over measured objects is a fact, and the sample count travels with it so a thin envelope can be distrusted.
Envelopes are order-independent. Object pose is only partially recovered by the perception stack, so a rotated chair swaps its horizontal axes; records and proposals are both sorted into (smallest, middle, largest) before comparison. Comparing named axes would reject correctly-sized but rotated objects.
Abstention means the instance could not be aligned, or that its envelope rests on too few samples to trust. It is a statement about the evidence, not about a missing rule.
The scene graph is organised in layers over a metric substrate produced by RTAB-Map: agent poses, objects, free-space regions from an approximated Generalized Voronoi Diagram, rooms, and the building at the root. Graph operations are ROS services exposed through a REST API, so perception, graph management and admission can each run on the robot or on a remote machine.
| Stage | What runs |
|---|---|
| Labels and attributes | a multimodal vision-language model, per frame and open-vocabulary |
| Grounding | YOLO-World-L, from the labels the model proposed |
| Masks and extents | SAM 2.1, with PCA over the segmented points for orientation |
| Association | all-MiniLM-L6-v2 sentence embeddings; description 0.50, colour 0.30, material 0.15, label 0.05, renormalised when an attribute is missing on either side |
| Alignment | label to ontology class, abstaining rather than forcing a placement |
| Size check | one interval per sorted axis; at least 30 measured samples before it is used numerically, otherwise a coarse band |
| Room check | class-to-room-type prior from an annotated RGB-D corpus, advisory: a conflict can only move a verdict to borderline |
| Record | every decision stored under its own identifier with the conditions that held, the bound cited and the corpus behind it |
Nothing the gate uses is a constant in code. The criteria in force, their parameters and the corpus each was derived from are declared as a policy, and a copy of that policy is filed with every decision, so asking why an object was refused and under which version of which expectation is a query rather than an exercise in reading the source.
The admission record of a single twenty-minute run on a Habitat-Matterport scene. Every proposal the gate judged is a node, joined to the ontology class its label was aligned to. Only admitted objects are joined to the room, because only they reached the map. Click a node to see the verdict and the bound that produced it.
Bird's-eye view of the traversed floor: light grey is where the robot can walk, dark is everything else. Each dot is a proposal, placed where it was seen and coloured by verdict, so furniture against a wall sits off the walkable area by construction. Selecting here selects it in the graph, and the other way round.
| Per distinct object | ||
|---|---|---|
| Objects judged | 54 | |
| Let in | 4 | 7.4% |
| Set aside as borderline | 6 | 11.1% |
| Refused | 34 | 63.0% |
| No grounds to judge | 10 | 18.5% |
| Real objects refused | 8 |
Eight refusals removed a real object, and that is the cost the design has to answer for. All eight were judged against their own class rather than a broader ancestor, so they are failures of measurement rather than of alignment: the same bed measured between 0.698 and 1.059 m on its smallest axis across five viewpoints, with the limit that decides its fate falling inside that spread. One case shows the graded verdict earning its place, a bed 2.7 cm below its lower limit set aside rather than refused, kept off the map and kept available.
Measured in October 2026 under the adopted ontology set (SOMA, SOMA-HOME, DUL, DogOnt and Wikidata items, stitched by equivalence and subclass mappings; no class is minted). The figures elsewhere on this page were measured before that change and are not comparable with the figures in this section.
The admission stage now does more than judge a proposal. After alignment, FOUND collects the properties that the stitched ontologies declare for the class, ranks them by relevance on the class–property lattice, and keeps the most relevant ones until their cumulative relevance reaches 0.8, so the number of properties per object depends on relevance. A vision-language model fills each selected property from the object's image crops at first observation, at each re-observation and after each merge. It abstains below a confidence of 0.65, and the value of an object property must be an existing node. Every fill is written to RDF with its provenance.
The filled properties feed a second FOUND component: a merge scorer. When geometry proposes that two objects are the same, the scorer compares them with a situated similarity operator. Asserted disjointness vetoes the merge; otherwise the shared properties, the box dimensions and the room are weighted by diagnosticity, relevance and the evidence that each object carries. A missing property removes weight instead of counting as a mismatch, and the scorer abstains when coverage is below 0.5. The decision band (same at 0.7 or above, different at 0.3 or below) is declared, not calibrated.
Both storeys of one simulated house (HM3D scene 00861), one fixed tour per storey, three arms: (1) the original scene-graph pipeline, whose describer attaches colour, material and description; (2) FOUND properties instead of the describer attributes, with merges decided by geometry only; (3) FOUND properties, with each merge decided by the FOUND merge scorer. Every verdict was logged at first observation. The logged verdicts are scored offline against the scene's ground-truth object inventory, which FOUND never reads.
| Arm | Storey | Objects | Proposals | Covered | Precision before | Precision after | Wrong rejections | Merges | Fills |
|---|---|---|---|---|---|---|---|---|---|
| 1 baseline | −1.59 m | 261 | 347 | 256 | 0.238 | 0.239 | 30 | 86 | – |
| +1.21 m | 340 | 457 | 346 | 0.335 | 0.308 | 59 | 116 | – | |
| 2 FOUND properties, geometric merges | −1.59 m | 113 | 260 | 207 | 0.251 | 0.286 | 18 | 147 | 520 |
| +1.21 m | 84 | 109 | 89 | 0.416 | 0.500 | 15 | 24 | 185 | |
| 3 FOUND properties and scorer | −1.59 m | 118 | 158 | 124 | 0.290 | 0.441 | 10 | 38 | 568 |
| +1.21 m | 225 | 292 | 248 | 0.363 | 0.410 | 32 | 23 | 499 |
Covered: proposals with a grounded verdict (admit, hold or decline). Precision before: real covered proposals over covered proposals. Precision after: real admitted proposals over admitted proposals. Wrong rejections: declined proposals that were real.
Pooled over both storeys, precision after filtering is 0.28 (58 of 209 admitted) for the baseline, 0.31 (30 of 96) for arm 2 and 0.42 (58 of 139) for arm 3. The baseline gate declines half of the real covered proposals (89 of 177); with FOUND properties and the scorer, one third (42 of 126). FOUND filled 1,067 properties in arm 3, from SOMA (46%), Wikidata (36%), DogOnt (10%) and DUL (9%). The scorer returned same 61 times and abstained in most comparisons, mainly because one of the two objects had no published fill yet.
Read with these limits. One scene and one run per arm, so there are no confidence intervals. The admission verdict is fixed at first observation, so the arms differ through the proposals that each association policy produces, not through properties filled later. Arm 2 also relaxes the stack's evidence threshold for association, which changes tracking as well and leaves it with fewer objects; its 0.500 on the upper storey rests on 12 admitted proposals. The baseline ran two days earlier with different transport settings. A first attempt at arm 3 stopped on a full command queue and is excluded.