Skip to main content

Interaction and manipulation

XR Blocks uses one interaction pipeline for mouse, gaze, hand rays, tracked controllers, direct touch, reticles, built-in UI, and automatic manipulation. For each input source, the pipeline resolves one public hit surface and one logical target. Private renderer meshes are normalized to their public owner.

input source
-> one ordered hit list
-> physical blocking surface
-> logical target and ancestor path
-> hover, capture, selection, semantic control, or manipulation
-> one shared event vocabulary

Application code does not add a UI raycaster or sort a second set of UI hits.

Event fields​

Object callbacks receive resolved event data:

FieldMeaningLifetime
sourceLogical mouse, gaze, hand, controller-ray, direct-touch, or simulator source; source.controller identifies its controllerStable for the callback
targetLogical object that owns the selected behaviorCaptured for the action; optional for a global event without a target
surfacePublic object representing the resolved hit surface; private renderer meshes are normalized to their public ownerCaptured for targeted actions; optional for a global event
currentTargetScript currently receiving the bubbled callbackChanges at each ancestor callback
intersectionCloned current ray intersection on the captured surfacePresent only while the ray still hits that surface; often absent after release outside
touchPositionWorld-space direct-contact positionPresent on touch and grab events instead of a ray intersection

target and surface can be different. An explicit hit registration can map a private physical mesh to a public surface while application behavior belongs to another logical ancestor. The private mesh is not part of the public event.

class SelectableCube extends xb.MeshScript {
constructor() {
super(
new THREE.BoxGeometry(0.2, 0.2, 0.2),
new THREE.MeshStandardMaterial({color: 0xfbbc04})
);
}

onObjectSelectStart(event) {
console.log({
sourceType: event.source.type,
controller: event.source.controller,
target: event.target,
surface: event.surface,
receiver: event.currentTarget,
point: event.intersection?.point,
});
this.material.color.set(0x4285f4);
event.stopPropagation();
}

onObjectSelectEnd(event) {
console.log(event.completed, event.reason);
this.material.color.set(0xfbbc04);
event.stopPropagation();
}
}

Use the resolved hit​

Inside an interaction callback, use the event's target, surface, and intersection. Repeating a raycast inside the callback can observe a different frame, ignore capture, or select a different physical surface.

Use current target queries only when code outside an event flow needs the current state:

const hit = xb.user.getRayIntersection(0);
const objectHit = xb.user.getIntersectionAt(object, 0);

xb.user.isPointingAt(object);
xb.user.isSelectingAt(object);
xb.user.isManipulating(object);

Reticles present targeting​

A reticle is a visual presentation of the resolved hit. It is not a data owner and does not define another targeting API.

const options = new xb.Options();
options.enableReticles();
options.enableDepth();
options.reticles.projectOnDepthMesh = true;

Read the hit from the interaction event or xb.user.getRayIntersection(). The projection option changes where the reticle is drawn when depth is available; it does not create or store a separate intersection contract.

Like Android XR's cursor, a reticle is a small disc with a dark border that grows with its distance from the viewer, so it stays easy to see far away without covering nearby UI. While selecting, it shrinks and its fill turns opaque. Set options.reticles.hoverRing = true to add a ring around the reticle over interactive targets. Set options.reticles.scaleWithDistance = false to keep a fixed world size, or options.reticles.enabled = false to hide reticles.

Addon input sources​

Addon controllers can implement the public Controller / ControllerEventMap types and register through input.enableController(controller). Set controller.userData.connected and selected, and provide updatePose() when the pose needs sampling each frame. Late-registered sources receive an engine-owned reticle when reticles are enabled, including the current renderer's reticle configuration. A supplied reticle remains caller-owned.

Use input.removeController(controller) for permanent teardown. It removes the source, event forwarding, frame state and attached reticle, disposing only an engine-owned reticle. Only addon controllers registered through enableController can be removed; built-in XR/simulator sources and repeated removals are no-ops. Removal sends selectend and disconnected once, including for sources without a disconnect() hook, and clears handedness aliases that reference that source. Addon IDs are monotonically allocated; use input.get(id) rather than array indices to look them up after removals. Its active interaction is cancelled as source-lost on the next frame; interaction.removeSource(controller) can cancel it immediately when needed.

interaction.queueScaleIntent(controller, factor) represents explicit pinch/zoom. It raycasts even in select raycast mode and can scale the manipulable ancestor of a UI control, honoring disabled boundaries and scale limits. Wheel intents retain scroll-container ownership and their existing fallback behavior. Explicit scale events preserve the actual input modality; only wheel scaling reports simulator as its source type.

Propagation and default behavior​

These controls solve different problems:

ControlMeaningWhere available
event.stopPropagation()Stop the targeted event from continuing to later ancestor scriptsObject select, hover, touch, grab, and manipulation callback dispatch
event.preventDefault()Keep callbacks but suppress the framework's proposed default actionTouch start and automatic manipulation events

The separate event.stopGlobalPropagation() control suppresses global selection callbacks without stopping ancestor callbacks or the default action.

For example, a child can handle touch without starting the default selection:

onObjectTouchStart(event) {
event.preventDefault(); // no default touch selection or manipulation
this.showContactFeedback();
event.stopPropagation(); // parent scripts do not also handle this touch-start event
}

Calling stopPropagation() does not suppress a default transform. Calling preventDefault() does not stop ancestor callbacks. Use each only for its own purpose.

event.stopGlobalPropagation() is available on selection events. Call it in onObjectSelectStart to suppress global onSelectStart, onSelecting, onLongSelect, onSelect, and onSelectEnd for that captured selection.

onObjectSelectStart(event) {
event.stopGlobalPropagation();
this.showSelectionFeedback();
}

Calling stopGlobalPropagation() on an end or long-select event only suppresses callbacks that have not run; it cannot undo earlier global callbacks.

Global propagation does not affect the selectionend event that Interaction dispatches on the source controller when its selection ends. Input-device code can listen there for its own feedback; event.completed is true for a successful selection, as in onSelectEnd.

Buttons, toggles, dropdowns and their options, sliders, text inputs, and scroll views stop ancestor and global selection propagation so operating UI does not also trigger scene actions. When overriding their selection handlers, call the corresponding superclass handler to preserve this behavior:

onObjectSelectStart(event) {
super.onObjectSelectStart(event);
this.beginAction();
}

onObjectSelectEnd(event) {
super.onObjectSelectEnd(event);
this.endAction();
}

Selection completion and capture​

onSelectEnd and onObjectSelectEnd include:

  • completed: true only when a valid selection completes;
  • reason: released, released-outside, source-lost, pointer-cancel, removed, hidden, or disabled.

Use onSelect for a global completed selection. Use onLongSelect or onObjectLongSelect for a held selection. Configure the delay with options.interaction.longSelectDuration. Manipulation captures do not also emit long-select behavior.

Touch, selection, grab, and manipulation are separate​

Direct-hand interaction uses related but distinct lifecycles:

StateStarts whenContinues withEnds whenDefault relationship
TouchIndex tip enters a targetonObjectTouchingTip leavesStarts selection unless touch start prevents it
SelectionSource presses or direct touch startsonSelectingRelease, contact end, or cancellationOwns completion and end reason
GrabA touching hand pinchesonObjectGrabbingPinch ends or contact is lostCan start direct manipulation
ManipulationA selected source claims an enabled manipulation owneronObjectManipulate with updateRelease, cancel, owner invalidation, or action changeApplies a proposed transform unless prevented

Typical direct sequence:

touch start
-> select start
-> touch/selecting updates
-> grab start
-> manipulation start/update
-> grab end
-> manipulation end
-> touch end
-> select end

Releasing a pinch can end grab and manipulation while touch and selection continue. Call event.preventDefault() in touch start when contact must not start the default selection.

Direct touch requires a tracked hand and index-tip pose. When either loses tracking, its last position is ignored and any active contact ends as source-lost, without completing a click. This releases touch suppression so a tracked controller ray can target again. Touch resumes when valid hand poses return.

Automatic manipulation​

Configure manipulation on the owner object. Do not create a manager:

object.xb = {
manipulation: {
actions: {
translate: {faceCamera: true},
rotate: {axis: 'y', space: 'world'},
scale: {minScale: 0.5, maxScale: 2},
},
},
};

For a plain object, manipulation: true enables translate and scale and uses translate as the surface action. On a UICard, manipulation: true also makes translation face the camera, scale with distance from the camera, smooth the pointer ray, and follow thumbstick push/pull within Android XR's distance limits, and it enables corner resize with easing and overshoot. A card's own translate and resize options merge over those defaults. The card can display an edge. ModelViewer enables move, Y-axis rotate, and scale with its own private interaction proxies.

Face-camera translation uses mode: 'capsule' by default. It keeps an object upright within 0.25 meters above or below the camera, then tilts it toward the viewer. Set capsuleHalfHeight to change that region, or select cylindrical or spherical mode explicitly.

Set translate: {scaleWithDistance: true} to scale an owner with its distance from the camera while it moves, like Android XR panels, so it looks larger when pulled closer and smaller when pushed away. A ray grab point stays on the same spot of the owner as it scales. The Scale action's minScale and maxScale clamp that scale.

A ray move keeps the point you grabbed under the ray, even when you swing an owner around you by its edge. Set translate: {raySmoothing: true} to smooth the ray. A small movement as the pinch starts does not jolt the owner. After that the owner eases toward the pointer instead of following it exactly, and catches up quickly when the hand reverses.

Set translate: {pushPull: true} to push an owner away with thumbstick forward and pull it closer with thumbstick back while a controller ray translates it. Pass {speed} to tune it: the distance changes exponentially by e^speed per second at full deflection, 1.5 by default. Hands have no thumbstick, so they change depth by moving the hand.

Set translate: {minDistance, maxDistance} in meters to keep every move, including push/pull, within that distance of the viewer. An owner that starts outside the limits can still move, just not farther outside them. UICard with manipulation: true uses Android XR's 0.75 to 5 meters.

resize applies only to UICard owners. It changes the card's size from a dragged corner and keeps anchor: 'center' (default) or the 'opposite' corner in place, within minSize and maxSize in meters. Set preserveAspectRatio: true to keep the card's proportions. Resize never starts from a card surface; it needs the card edge corners, from a ray or a direct-touch pinch, or a resize handle. A resize that changes size converts automatic width and height dimensions to fixed sizes using the current layout as its starting point.

Use a handle when one surface must select a specific action:

rotateHandle.xb = {manipulationHandle: {action: 'rotate'}};
object.add(rotateHandle);

onObjectManipulate(event) observes start, update, end, and cancel. The event includes action, owner, primary source, and all active sources. Action-specific events also contain the proposed position, rotation, scale, or card width and height values. Call event.preventDefault() during start to replace the automatic transform for that phase.

Concurrency and two-source scale​

Manipulation state is stored per owner, not in one global drag slot:

  • different sources can manipulate different objects at the same time;
  • one source has only one active role;
  • a second source can join one active scale-enabled owner for two-source scale;
  • the first source remains the primary owner of that session;
  • releasing the auxiliary source ends scale and resumes the primary configured action when one exists;
  • hiding, removing, reparenting, disabling, or invalidating an active owner cancels its session.

Keep application state per object or owning script. Do not store the active object in one application-wide draggedObject field if simultaneous manipulation is allowed.

Placement scripts during manipulation​

XR Blocks suspends direct TransformScript children of a manipulation owner. It resumes and rebases them after end or cancel. This prevents a follow, face, or orbit update from fighting the user's transform.

Read Placement scripts for supported combinations, suspension, manual rebasing, and one-time surface placement.

Visual, depth, and pointer participation​

Opacity, depth behavior, and interaction are independent. For custom Three.js objects, use material.depthTest and material.depthWrite for depth and object.xb.pointerEvents for hit participation. visible = false removes an object branch from rendering and interaction. A transparent object can still write depth and block a pointer.

Built-in UI registers its private physical surfaces with this same interaction resolver. Application code configures public UI components and does not access or sort the private render meshes.

Executable foundation​

See templates/02_object_interaction for ray selection, direct touch, and automatic manipulation. See demos/interaction_playground/main.js for placement scripts, UI, handles, concurrent object behavior, and two-source scale in one scene.