Skip to main content

World understanding

xb.world exposes enabled physical-world evidence. Configure each branch before initialization, then convert fresh evidence into explicit application state. Every branch can be unsupported, warming up, empty, stale, or failed.

Choose the evidence​

EvidenceEnableTypical reaction
Coarse floors, walls, and tablesoptions.enablePlaneDetection()One-time placement or surface-specific state
Platform room geometryoptions.world.enableMeshDetection()Visualization, collision, or room-aware behavior
Live per-pixel distance and depth meshoptions.enableDepth()Reticle projection, occlusion, or collision
Named physical objectsoptions.enableObjectDetection() plus backend prerequisitesLabels, grounded actions, or scene state
Human body jointsoptions.enableHumanDetection()Pose-aware application behavior
Face landmarks and expressionsoptions.enableFaceDetection()Avatar or expression behavior
Person segmentation maskoptions.enableSegmentation()Camera effects or person/background separation

Use Depth and Occlusion for depth details. Use Placement scripts to distinguish one-time room placement from continuous follow, face, orbit, and visibility behavior.

Plane detection​

const options = new xb.Options();
options.enablePlaneDetection();
await xb.init(options);

Planes are coarse semantic surfaces supplied by WebXR or the simulator. The plane collection can remain empty on unsupported devices.

For one-time placement, let XR Blocks inspect available planes and meshes:

const placed = await xb.world.placeOnHorizontalSurface(model, {seconds: 15});
if (!placed) showPlacementUnavailable();

The result is a boolean. Treat false as a normal no-suitable-surface result. The helper places once; it does not keep the object anchored or facing the viewer. Read Placement scripts for continuous behavior and manipulation rebasing.

Scene mesh detection​

const options = new xb.Options();
options.world.enableMeshDetection();
options.world.meshes.showDebugVisualizations = true;
await xb.init(options);

Scene meshes are platform-reconstructed room geometry. Read current meshes from xb.world.meshes.xrMeshToThreeMesh. The map can be empty and native support is experimental. The simulator can inject its environment meshes through the same runtime branch.

When Rapier is configured, enabled scene meshes can create environment colliders. Mesh detection does not enable physics by itself.

Object detection​

Object detection captures the environment camera, finds named items in 2D, and uses depth to ground them in world space. Choose the backend explicitly.

Backends​

Under the hood, object detection uses pluggable backends selected via options.world.objects.backend:

  • GeminiDetectorBackend: Queries Gemini models using visual camera frames (requires AI API keys, see the Generative AI guide).
  • MediaPipeDetectorBackend: Runs local MediaPipe object detection model.

Gemini setup:

const options = new xb.Options();
options.enableAI();
options.enableCamera('environment');
options.enableDepth();
options.enableObjectDetection();
options.world.objects.backendConfig.activeBackend = 'gemini';

This branch sends captured image data to Google and requires prototype or production credentials as described in Generative AI.

On-device MediaPipe setup:

const options = new xb.Options();
options.enableCamera('environment');
options.enableDepth();
options.enableObjectDetection();
options.world.objects.backendConfig.activeBackend = 'mediapipe';

Run one detection:

const objects = await xb.world.objects.runDetection();
for (const object of objects) {
  console.log(object.name, object.position);
}

Concurrent calls share the active detection. A new run clears previous detected-object scene nodes before publishing current results. Treat [] as a normal warm-up, empty, unavailable, or failed result and update application state accordingly.

For continuous ownership, pair the same client object:

const client = {};
xb.world.objects.start(client);
// Later:
xb.world.objects.stop(client);

Use options.world.objects.pollingIntervalMs to control cadence. The desktop simulator can supply deterministic object ground truth when options.world.objects.simulatorOverride = true and its environment defines detectable objects.

Human pose detection​

enableHumanDetection() declares camera permission and enables the environment camera and depth prerequisites:

const options = new xb.Options();
options.enableHumanDetection();
await xb.init(options);

const poses = await xb.world.humans.runDetection();
const leftWrist = poses[0]?.getJointPosition(xb.PoseJointName.LeftWrist);

Human detection uses on-device MediaPipe and projects supported joints into world space with depth. Guard empty results and missing joints. Use start(client) and stop(client) for continuous ownership.

Inference runs in a web worker by default so a detection pass does not stall the render loop. Workers have no DOM canvas for MediaPipe to draw on, so the worker uses the CPU delegate; set options.world.humans.backendConfig.mediapipe.useWorker = false to run on the main thread with the GPU delegate instead. When the browser has no worker support the SDK falls back to the main thread on its own.

Face landmark detection​

enableFaceDetection() declares camera permission and enables camera and depth:

const options = new xb.Options();
options.enableFaceDetection();
await xb.init(options);

const faces = await xb.world.faces.runDetection();
const nose = faces[0]?.getLandmarkPosition(xb.FaceLandmarkName.NoseTip);
const jawOpen = faces[0]?.getBlendshape('jawOpen');

Each result can contain world-space landmarks, optional blendshapes, and an optional facial transformation. Guard every optional output. Use start(client) and stop(client) for continuous ownership.

Semantic segmentation​

Segmentation is an on-device 2D camera operation and does not require depth:

const options = new xb.Options();
options.enableSegmentation();
await xb.init(options);

const mask = await xb.world.segmentation.runSegmentation();
if (mask) console.log(mask.width, mask.height, mask.data);

Read latestMask for the newest completed inference. null is a normal not-ready or unavailable state. Category values are exported as xb.SegmentCategory.

Freshness, permission, and cleanup​

  • Declare camera and other permissions before entering immersive XR.
  • Lock explicit requests when duplicate work has no meaning.
  • Use the matching start(client) and stop(client) pair for continuous detectors.
  • Clear old visible results when a new detection is empty or stale.
  • Distinguish synthetic simulator evidence from native device evidence.
  • Stop application-owned polling and media resources in dispose().

Seeing real people on the desktop simulator​

Face detection, human pose detection, and segmentation all look for people, but enableFaceDetection(), enableHumanDetection(), and enableSegmentation() each turn on the environment camera, which is right on a headset: you are detecting the people in front of you.

On desktop that same setting hands MediaPipe the simulator's rendered room, which never contains a person, so detection silently returns nothing. Ask for the user-facing camera instead:

options.enableHumanDetection();
// After the helper, which calls enableCamera() internally and would otherwise
// overwrite this.
options.enableCamera('user');

Do not reach for options.simulator.deviceCamera.enabled = false here. It does route frames from the real webcam, but it also stops the simulator camera being registered, and on desktop that is what supplies the camera pose. Without a pose, getCameraParametersSnapshot() returns null and detection is skipped before MediaPipe runs, leaving you with a working camera and no results. enableCamera('user') keeps the pose and changes only the stream.

Neither affects a real headset, where the simulator never starts.

For working samples, see samples/depthmesh/main.js,