Skip to main content

Desktop simulator

The simulator runs XR Blocks applications in a normal desktop browser. It uses the same scripts, targeting, selection, and world APIs as a real XR session. It does not emulate the browser WebXR API.

Start the simulator​

Show the simulator option in the XR button:

const options = new xb.Options();
options.xrButton.showEnterSimulatorButton = true;
await xb.init(options);

Autostart it in code or by URL:

options.formFactor = 'desktop';
http://127.0.0.1:8080/templates/14_simulator_setup/?formFactor=desktop

The simulator runtime is loaded only when startup is requested. On a desktop autostart path, await xb.init(options) waits for that load. Applications can also call:

await xb.init(options);
const simulator = await xb.core.startSimulator();

xb.core.simulator is undefined before startup completes. Use onSimulatorStarted() when a script must react to simulator startup.

The Start Simulator button keeps the entry UI visible while the runtime loads and initializes, with both entry buttons disabled during startup. If startup fails, it displays the error and lets the user retry. The entry UI is removed only after startup succeeds.

Browser interface​

The settings, instructions, and hand-pose panels register automatically when Lit is available. No simulator add-on import is required. CDN applications need these mappings:

{
"imports": {
"lit": "https://esm.sh/lit@3.3.1",
"lit/": "https://esm.sh/lit@3.3.1/"
}
}

Disable panels before initialization when an application needs a clean canvas:

options.simulator.simulatorSettingsPanel.enabled = false;
options.simulator.instructions.enabled = false;
options.simulator.handPosePanel.enabled = false;

Glasses display presets​

The simulator can narrow its perspective camera to a numeric diagonal field of view in degrees. For example, Spectacles '27 is about 51° and XREAL Aura about 70°; values are approximate and device specifications can change:

options.simulator.displayDiagonalFov = 51; // Spectacles '27, approx.
// Or: options.simulator.displayDiagonalFov = 70; // XREAL Aura, approx.
options.simulator.blendingMode = 'screen';

Leave displayDiagonalFov unset to keep the camera's field of view unchanged. screen blending approximates black-as-transparent compositing over the simulator environment, which is useful for checking bright-on-dark additive UI. It is not a display or optics emulator; verify contrast, text size, and target size on-device.

Control modes​

XR Blocks supplies five simulator modes:

  • SimulatorMode.USER: user movement and mouse selection.
  • SimulatorMode.POSE: navigation plus virtual hand-pose controls.
  • SimulatorMode.CONTROLLER: drive an active virtual hand or controller.
  • SimulatorMode.POINTER_LOCK: first-person pointer-lock navigation.
  • SimulatorMode.EDITOR: simulator scene editing.
options.simulator.defaultMode = xb.SimulatorMode.CONTROLLER;
options.simulator.modeToggle.enabled = true;

When mode toggling is enabled, the configured toggle key cycles through options.simulator.modeToggle.toggleOrder.

Keyboard navigation and simulator shortcuts are suspended while an input, textarea, select, or editable region has focus, including text controls inside open shadow roots. Focusing a text control clears held movement keys; leaving it does not resume movement until another navigation key is pressed. Text editing and its normal key events are not blocked, and IME composition does not trigger simulator shortcuts.

Automation preset​

Use the shared preset for automated or external browser runs:

const options = new xb.Options().enableAutomationMode({
hideSimulatorUi: true,
defaultMode: xb.SimulatorMode.POSE,
enableHands: true,
enableCamera: true,
});
await xb.init(options);

?xrAutomation=1 applies the default preset. ?debug=1 separately exposes window.xb and the initialization promise window.xbReady.

Scene environments​

Each environment points to one JSON manifest:

options.simulator.environments = [
{name: 'Evaluation Room', manifestPath: './evaluation-room.json'},
];
options.simulator.activeEnvironmentIndex = 0;
evaluation-room.json
{
"scenePath": "./room.glb",
"scenePlanesPath": "./room-planes.json",
"navMeshPath": "./room-navmesh.glb",
"position": [0, 0, 0],
"locations": {
"wall-primary": {
"description": "Clear wall point for ray interaction.",
"position": [0, 1.4, -2]
}
},
"objects": [
{
"id": "chair",
"assetPath": "./chair.glb",
"position": [0.5, 0, -1.5],
"physics": "fixed",
"detectObject": true,
"label": "chair"
}
]
}

Use scenePath or videoPath, not both. Paths can be manifest-relative, root-relative, or absolute URLs. Object physics can be false, fixed, or dynamic.

Load another environment at runtime with:

await xb.core.simulator.setEnvironment('Ad Hoc Room', './ad-hoc-room.json');

Read the optional named world-space locations from the active environment:

const locations = xb.core.simulator.getLocations();
const wall = locations['wall-primary'];
console.log(wall.description, wall.position);
options.simulator.navMesh.enabled = true;
options.simulator.navMesh.showDebugVisualizations = true;

The navmesh constrains and grounds the simulated user. It does not constrain hands or controllers. CDN applications that enable it need the three-pathfinding mapping shown in rolldown.config.js and templates/14_simulator_setup/index.html.

Runtime physical objects​

const [chair] = await xb.core.simulator.objects.addObjects([
{
assetPath: './chair.glb',
physics: 'fixed',
detectObject: true,
label: 'chair',
},
]);

xb.core.simulator.objects.get([chair.id]);
xb.core.simulator.objects.removeObjects([chair.id]);
xb.core.simulator.objects.clear();

Use simulator ground truth through the normal world detector API:

options.enableObjectDetection();
options.world.objects.simulatorOverride = true;

const objects = await xb.world.objects.runDetection();

The override applies only in the simulator. Real XR sessions use the configured detector backend.

Hand reach and physical hands​

options.simulator.reachDistance.enabled = true;
options.simulator.reachDistance.radius = 0.75;
options.simulator.reachAngle.enabled = true;
options.simulator.reachAngle.angle = Math.PI;

options.simulator.handPhysics.enabled = true;

Physical hands require Rapier. leftHandOrigin and rightHandOrigin are camera-local shoulder anchors for reach limits and the physics tether. Set options.simulator.physics.enabled = false to disable simulator-owned collision without disabling the application's Rapier world.

See templates/14_simulator_setup, demos/sim_hand_poses, and the API reference for SimulatorOptions.

Authoring hand rotations​

setLeftHandRotations() and setRightHandRotations() accept joint-name maps of [x, y, z] Euler angles in radians, composed in XYZ order. The demo sliders show degrees and convert them to radians. All three axes remain available.

The controls follow anatomical movement directions:

JointPositive XPositive YZ
Finger MCPPalmar flexionAbduction away from the middle fingerAxial rotation
Finger PIP/DIPPalmar flexionOff-axis adjustmentAxial rotation
Thumb CMC (thumb-metacarpal)Flexion across the palmPalmar abduction, out of the palm plane toward the finger-pad sideAxial rotation
Thumb MCP/IPFlexion toward the thumb's own padOff-axis adjustmentAxial rotation
WristPalmar flexionRadial deviation toward the thumbAxial rotation

Negative X extends; negative Y reverses the listed movement. On the middle finger, positive Y means radial deviation toward the index finger. Thumb MCP/IP flexion follows the thumb's pad/nail orientation as the base rotates.

Thumb opposition is a combined movement involving CMC flexion, palmar abduction, and axial rotation. These movement definitions follow OUHSC's thumb biomechanics description. The thumb-pad orientation uses the rig's WebXR joint-frame convention.

Recording and replaying sessions​

The session-replay addon records head, controller, select, and hand-joint input in a headset or the simulator, and replays it in the simulator:

import {
SessionPlayer,
SessionRecorder,
} from 'xrblocks/addons/session-replay/index.js';

const player = new SessionPlayer();
const recorder = new SessionRecorder();
xb.add(player);
xb.add(recorder);

recorder.start();
// ...
const recording = recorder.stop();
await player.play(recording);

Recordings are JSON, so a session captured on a headset can be saved and replayed on a desktop. Replay drives input, not scene state. play() first loads the simulator room the recording was made in (pass {environment: false} to stay in the current room), and your app restores its own objects from recorder.start({state}) before play(). Headset recordings, including AR ones, do not capture the real room, so replay uses the simulator room's planes and depth. Add the player before other scripts so they see replayed poses in the same frame. Replay drives the simulator only; in a real XR session play() rejects. See demos/session_replay and the addon README for the format and limitations.