Interaction and manipulation
XR Blocks uses one interaction pipeline for mouse, gaze, hand rays, tracked controllers, direct touch, reticles, built-in UI, and automatic manipulation. For each input source, the pipeline resolves one public hit surface and one logical target. Private renderer meshes are normalized to their public owner.
input source
-> one ordered hit list
-> physical blocking surface
-> logical target and ancestor path
-> hover, capture, selection, semantic control, or manipulation
-> one shared event vocabulary
Application code does not add a UI raycaster or sort a second set of UI hits.
Event fields
Object callbacks receive resolved event data:
| Field | Meaning | Lifetime |
|---|---|---|
source | Logical mouse, gaze, hand, controller-ray, direct-touch, or simulator source; source.controller identifies its controller | Stable for the callback |
target | Logical object that owns the selected behavior | Captured for the action; optional for a global event without a target |
surface | Public object representing the resolved hit surface; private renderer meshes are normalized to their public owner | Captured for targeted actions; optional for a global event |
currentTarget | Script currently receiving the bubbled callback | Changes at each ancestor callback |
intersection | Cloned current ray intersection on the captured surface | Present only while the ray still hits that surface; often absent after release outside |
touchPosition | World-space direct-contact position | Present on touch and grab events instead of a ray intersection |
target and surface can be different. An explicit hit registration can map a
private physical mesh to a public surface while application behavior belongs to
another logical ancestor. The private mesh is not part of the public event.
class SelectableCube extends xb.MeshScript {
constructor() {
super(
new THREE.BoxGeometry(0.2, 0.2, 0.2),
new THREE.MeshStandardMaterial({color: 0xfbbc04})
);
}
onObjectSelectStart(event) {
console.log({
sourceType: event.source.type,
controller: event.source.controller,
target: event.target,
surface: event.surface,
receiver: event.currentTarget,
point: event.intersection?.point,
});
this.material.color.set(0x4285f4);
event.stopPropagation();
}
onObjectSelectEnd(event) {
console.log(event.completed, event.reason);
this.material.color.set(0xfbbc04);
event.stopPropagation();
}
}
Use the resolved hit
Inside an interaction callback, use the event's target, surface, and
intersection. Repeating a raycast inside the callback can observe a different
frame, ignore capture, or select a different physical surface.
Use current target queries only when code outside an event flow needs the current state:
const hit = xb.user.getRayIntersection(0);
const objectHit = xb.user.getIntersectionAt(object, 0);
xb.user.isPointingAt(object);
xb.user.isSelectingAt(object);
xb.user.isManipulating(object);
Reticles present targeting
A reticle is a visual presentation of the resolved hit. It is not a data owner and does not define another targeting API.
const options = new xb.Options();
options.enableReticles();
options.enableDepth();
options.reticles.projectOnDepthMesh = true;
Read the hit from the interaction event or xb.user.getRayIntersection(). The
projection option changes where the reticle is drawn when depth is available;
it does not create or store a separate intersection contract.
Like Android XR's cursor, a reticle is a small disc with a dark border that
grows with its distance from the viewer, so it stays easy to see far away
without covering nearby UI. While selecting, it shrinks and its fill turns
opaque. Set options.reticles.hoverRing = true to add a ring around the
reticle over interactive targets. Set
options.reticles.scaleWithDistance = false to keep a fixed world size, or
options.reticles.enabled = false to hide reticles.
Addon input sources
Addon controllers can implement the public Controller / ControllerEventMap
types and register through input.enableController(controller). Set
controller.userData.connected and selected, and provide updatePose() when
the pose needs sampling each frame. Late-registered sources receive an
engine-owned reticle when reticles are enabled, including the current renderer's
reticle configuration. A supplied reticle remains caller-owned.
Use input.removeController(controller) for permanent teardown. It removes the
source, event forwarding, frame state and attached reticle, disposing only an
engine-owned reticle. Only addon controllers registered through enableController
can be removed; built-in XR/simulator sources and repeated removals are no-ops.
Removal sends selectend and disconnected once, including for sources without
a disconnect() hook, and clears handedness aliases that reference that source.
Addon IDs are monotonically allocated; use input.get(id) rather than array
indices to look them up after removals. Its active interaction is cancelled as
source-lost on the next frame;
interaction.removeSource(controller) can cancel it immediately when needed.
interaction.queueScaleIntent(controller, factor) represents explicit pinch/zoom.
It raycasts even in select raycast mode and can scale the manipulable ancestor
of a UI control, honoring disabled boundaries and scale limits. Wheel intents
retain scroll-container ownership and their existing fallback behavior. Explicit
scale events preserve the actual input modality; only wheel scaling reports
simulator as its source type.
Propagation and default behavior
These controls solve different problems:
| Control | Meaning | Where available |
|---|---|---|
event.stopPropagation() | Stop the targeted event from continuing to later ancestor scripts | Object select, hover, touch, grab, and manipulation callback dispatch |
event.preventDefault() | Keep callbacks but suppress the framework's proposed default action | Touch start and automatic manipulation events |
The separate event.stopGlobalPropagation() control suppresses global selection
callbacks without stopping ancestor callbacks or the default action.
For example, a child can handle touch without starting the default selection:
onObjectTouchStart(event) {
event.preventDefault(); // no default touch selection or manipulation
this.showContactFeedback();
event.stopPropagation(); // parent scripts do not also handle this touch-start event
}
Calling stopPropagation() does not suppress a default transform. Calling
preventDefault() does not stop ancestor callbacks. Use each only for its own
purpose.
event.stopGlobalPropagation() is available on selection events. Call it in
onObjectSelectStart to suppress global onSelectStart, onSelecting,
onLongSelect, onSelect, and onSelectEnd for that captured selection.
onObjectSelectStart(event) {
event.stopGlobalPropagation();
this.showSelectionFeedback();
}
Calling stopGlobalPropagation() on an end or long-select event only suppresses
callbacks that have not run; it cannot undo earlier global callbacks.
Global propagation does not affect the selectionend event that Interaction
dispatches on the source controller when its selection ends. Input-device code
can listen there for its own feedback; event.completed is true for a
successful selection, as in onSelectEnd.
Buttons, toggles, dropdowns and their options, sliders, text inputs, and scroll views stop ancestor and global selection propagation so operating UI does not also trigger scene actions. When overriding their selection handlers, call the corresponding superclass handler to preserve this behavior:
onObjectSelectStart(event) {
super.onObjectSelectStart(event);
this.beginAction();
}
onObjectSelectEnd(event) {
super.onObjectSelectEnd(event);
this.endAction();
}
Selection completion and capture
onSelectEnd and onObjectSelectEnd include:
completed:trueonly when a valid selection completes;reason:released,released-outside,source-lost,pointer-cancel,removed,hidden, ordisabled.
Use onSelect for a global completed selection. Use onLongSelect or
onObjectLongSelect for a held selection. Configure the delay with
options.interaction.longSelectDuration. Manipulation captures do not also
emit long-select behavior.
Touch, selection, grab, and manipulation are separate
Direct-hand interaction uses related but distinct lifecycles:
| State | Starts when | Continues with | Ends when | Default relationship |
|---|---|---|---|---|
| Touch | Index tip enters a target | onObjectTouching | Tip leaves | Starts selection unless touch start prevents it |
| Selection | Source presses or direct touch starts | onSelecting | Release, contact end, or cancellation | Owns completion and end reason |
| Grab | A touching hand pinches | onObjectGrabbing | Pinch ends or contact is lost | Can start direct manipulation |
| Manipulation | A selected source claims an enabled manipulation owner | onObjectManipulate with update | Release, cancel, owner invalidation, or action change | Applies a proposed transform unless prevented |
Typical direct sequence:
touch start
-> select start
-> touch/selecting updates
-> grab start
-> manipulation start/update
-> grab end
-> manipulation end
-> touch end
-> select end
Releasing a pinch can end grab and manipulation while touch and selection
continue. Call event.preventDefault() in touch start when contact must not
start the default selection.
Direct touch requires a tracked hand and index-tip pose. When either loses tracking, its last position is ignored and any active contact ends as source-lost, without completing a click. This releases touch suppression so a tracked controller ray can target again. Touch resumes when valid hand poses return.
Automatic manipulation
Configure manipulation on the owner object. Do not create a manager:
object.xb = {
manipulation: {
actions: {
translate: {faceCamera: true},
rotate: {axis: 'y', space: 'world'},
scale: {minScale: 0.5, maxScale: 2},
},
},
};
For a plain object, manipulation: true enables translate and scale and uses
translate as the surface action. On a UICard, manipulation: true also
makes translation face the camera, scale with distance from the camera, smooth
the pointer ray, and follow thumbstick push/pull within Android XR's distance
limits, and it enables corner resize with easing and overshoot. A card's
own translate and resize options merge over those defaults. The card can
display an edge. ModelViewer enables move, Y-axis rotate, and scale with its
own private interaction proxies.
Face-camera translation uses mode: 'capsule' by default. It keeps an object
upright within 0.25 meters above or below the camera, then tilts it toward the
viewer. Set capsuleHalfHeight to change that region, or select cylindrical
or spherical mode explicitly.
Set translate: {scaleWithDistance: true} to scale an owner with its distance
from the camera while it moves, like Android XR panels, so it looks larger when
pulled closer and smaller when pushed away. A ray grab point stays on the same
spot of the owner as it scales. The Scale action's minScale and maxScale
clamp that scale.
A ray move keeps the point you grabbed under the ray, even when you swing an
owner around you by its edge. Set translate: {raySmoothing: true} to smooth
the ray. A small movement as the pinch starts does not jolt the owner. After
that the owner eases toward the pointer instead of following it exactly, and
catches up quickly when the hand reverses.
Set translate: {pushPull: true} to push an owner away with thumbstick forward
and pull it closer with thumbstick back while a controller ray translates it.
Pass {speed} to tune it: the distance changes exponentially by e^speed
per second at full deflection, 1.5 by default. Hands have no thumbstick, so they change depth by
moving the hand.
Set translate: {minDistance, maxDistance} in meters to keep every move,
including push/pull, within that distance of the viewer. An owner that starts
outside the limits can still move, just not farther outside them. UICard
with manipulation: true uses Android XR's 0.75 to 5 meters.
resize applies only to UICard owners. It changes the card's size from a
dragged corner and keeps anchor: 'center' (default) or the 'opposite' corner
in place, within minSize and maxSize in meters. Set
preserveAspectRatio: true to keep the card's proportions. Resize never starts
from a card surface; it needs the card edge corners, from a ray or a
direct-touch pinch, or a resize handle. A resize that changes size converts
automatic width and height dimensions to fixed sizes using the current layout
as its starting point.
Use a handle when one surface must select a specific action:
rotateHandle.xb = {manipulationHandle: {action: 'rotate'}};
object.add(rotateHandle);
onObjectManipulate(event) observes start, update, end, and cancel.
The event includes action, owner, primary source, and all active
sources. Action-specific events also contain the proposed position, rotation,
scale, or card width and height values. Call event.preventDefault() during start to replace the
automatic transform for that phase.
Concurrency and two-source scale
Manipulation state is stored per owner, not in one global drag slot:
- different sources can manipulate different objects at the same time;
- one source has only one active role;
- a second source can join one active scale-enabled owner for two-source scale;
- the first source remains the primary owner of that session;
- releasing the auxiliary source ends scale and resumes the primary configured action when one exists;
- hiding, removing, reparenting, disabling, or invalidating an active owner cancels its session.
Keep application state per object or owning script. Do not store the active
object in one application-wide draggedObject field if simultaneous
manipulation is allowed.
Placement scripts during manipulation
XR Blocks suspends direct TransformScript children of a manipulation owner.
It resumes and rebases them after end or cancel. This prevents a follow, face,
or orbit update from fighting the user's transform.
Read Placement scripts for supported combinations, suspension, manual rebasing, and one-time surface placement.
Visual, depth, and pointer participation
Opacity, depth behavior, and interaction are independent. For custom Three.js
objects, use material.depthTest and material.depthWrite for depth and
object.xb.pointerEvents for hit participation. visible = false removes an
object branch from rendering and interaction. A transparent object can still
write depth and block a pointer.
Built-in UI registers its private physical surfaces with this same interaction resolver. Application code configures public UI components and does not access or sort the private render meshes.
Executable foundation
See templates/02_object_interaction for ray selection, direct touch, and
automatic manipulation. See demos/interaction_playground/main.js for
placement scripts, UI, handles, concurrent object behavior, and two-source
scale in one scene.