Skip to main content

Desktop simulator

The simulator runs XR Blocks applications in a normal desktop browser. It uses the same scripts, targeting, selection, and world APIs as a real XR session. It does not emulate the browser WebXR API.

Start the simulator​

Show the simulator option in the XR button:

const options = new xb.Options();
options.xrButton.showEnterSimulatorButton = true;
await xb.init(options);

Autostart it in code or by URL:

options.formFactor = 'desktop';
http://127.0.0.1:8080/templates/14_simulator_setup/?formFactor=desktop

The simulator runtime is loaded only when startup is requested. On a desktop autostart path, await xb.init(options) waits for that load. Applications can also call:

await xb.init(options);
const simulator = await xb.core.startSimulator();

xb.core.simulator is undefined before startup completes. Use onSimulatorStarted() when a script must react to simulator startup.

The Start Simulator button keeps the entry UI visible while the runtime loads and initializes, with both entry buttons disabled during startup. If startup fails, it displays the error and lets the user retry. The entry UI is removed only after startup succeeds.

Browser interface​

The settings, instructions, and hand-pose panels register automatically when Lit is available. No simulator add-on import is required. CDN applications need these mappings:

{
  "imports": {
    "lit": "https://esm.sh/lit@3.3.1",
    "lit/": "https://esm.sh/lit@3.3.1/"
  }
}

Disable panels before initialization when an application needs a clean canvas:

options.simulator.simulatorSettingsPanel.enabled = false;
options.simulator.instructions.enabled = false;
options.simulator.handPosePanel.enabled = false;

Control modes​

XR Blocks supplies five simulator modes:

  • SimulatorMode.USER: user movement and mouse selection.
  • SimulatorMode.POSE: navigation plus virtual hand-pose controls.
  • SimulatorMode.CONTROLLER: drive an active virtual hand or controller.
  • SimulatorMode.POINTER_LOCK: first-person pointer-lock navigation.
  • SimulatorMode.EDITOR: simulator scene editing.
options.simulator.defaultMode = xb.SimulatorMode.CONTROLLER;
options.simulator.modeToggle.enabled = true;

When mode toggling is enabled, the configured toggle key cycles through options.simulator.modeToggle.toggleOrder.

Keyboard navigation and simulator shortcuts are suspended while an input, textarea, select, or editable region has focus, including text controls inside open shadow roots. Focusing a text control clears held movement keys; leaving it does not resume movement until another navigation key is pressed. Text editing and its normal key events are not blocked, and IME composition does not trigger simulator shortcuts.

Automation preset​

Use the shared preset for automated or external browser runs:

const options = new xb.Options().enableAutomationMode({
  hideSimulatorUi: true,
  defaultMode: xb.SimulatorMode.POSE,
  enableHands: true,
  enableCamera: true,
});
await xb.init(options);

?xrAutomation=1 applies the default preset. ?debug=1 separately exposes window.xb and the initialization promise window.xbReady.

Scene environments​

Each environment points to one JSON manifest:

options.simulator.environments = [
  {name: 'Evaluation Room', manifestPath: './evaluation-room.json'},
];
options.simulator.activeEnvironmentIndex = 0;
evaluation-room.json
{
  "scenePath": "./room.glb",
  "scenePlanesPath": "./room-planes.json",
  "navMeshPath": "./room-navmesh.glb",
  "position": [0, 0, 0],
  "locations": {
    "wall-primary": {
      "description": "Clear wall point for ray interaction.",
      "position": [0, 1.4, -2]
    }
  },
  "objects": [
    {
      "id": "chair",
      "assetPath": "./chair.glb",
      "position": [0.5, 0, -1.5],
      "physics": "fixed",
      "detectObject": true,
      "label": "chair"
    }
  ]
}

Use scenePath or videoPath, not both. Paths can be manifest-relative, root-relative, or absolute URLs. Object physics can be false, fixed, or dynamic.

Load another environment at runtime with:

await xb.core.simulator.setEnvironment('Ad Hoc Room', './ad-hoc-room.json');

Read the optional named world-space locations from the active environment:

const locations = xb.core.simulator.getLocations();
const wall = locations['wall-primary'];
console.log(wall.description, wall.position);
options.simulator.navMesh.enabled = true;
options.simulator.navMesh.showDebugVisualizations = true;

The navmesh constrains and grounds the simulated user. It does not constrain hands or controllers. CDN applications that enable it need the three-pathfinding mapping shown in rolldown.config.js and templates/14_simulator_setup/index.html.

Runtime physical objects​

const [chair] = await xb.core.simulator.objects.addObjects([
  {
    assetPath: './chair.glb',
    physics: 'fixed',
    detectObject: true,
    label: 'chair',
  },
]);

xb.core.simulator.objects.get([chair.id]);
xb.core.simulator.objects.removeObjects([chair.id]);
xb.core.simulator.objects.clear();

Use simulator ground truth through the normal world detector API:

options.enableObjectDetection();
options.world.objects.simulatorOverride = true;

const objects = await xb.world.objects.runDetection();

The override applies only in the simulator. Real XR sessions use the configured detector backend.

Hand reach and physical hands​

options.simulator.reachDistance.enabled = true;
options.simulator.reachDistance.radius = 0.75;
options.simulator.reachAngle.enabled = true;
options.simulator.reachAngle.angle = Math.PI;

options.simulator.handPhysics.enabled = true;

Physical hands require Rapier. leftHandOrigin and rightHandOrigin are camera-local shoulder anchors for reach limits and the physics tether. Set options.simulator.physics.enabled = false to disable simulator-owned collision without disabling the application's Rapier world.

See templates/14_simulator_setup, demos/sim_hand_poses, and the API reference for SimulatorOptions.