Skip to main content

Class: ObjectDetector

Defined in: src/world/objects/ObjectDetector.ts:90

Detects objects in the user's environment using a specified backend. It queries an AI model with the device camera feed and returns located objects with 2D and 3D positioning data.

Extends​

Constructors​

Constructor​

new ObjectDetector(): ObjectDetector

Defined in: node_modules/@types/three/src/core/Object3D.d.ts:105

This creates a new Object3D object.

Returns​

ObjectDetector

Inherited from​

Script.constructor

Properties​

detectedObjects​

detectedObjects: DetectedObject<unknown>[] = []

Defined in: src/world/objects/ObjectDetector.ts:127

The latest detected objects.


isXRScript​

isXRScript: boolean = true

Defined in: src/core/Script.ts:100

Inherited from​

Script.isXRScript


xb?​

optional xb: XBObjectOptions

Defined in: src/interaction/InteractionTypes.ts:34

Inherited from​

Script.xb


dependencies​

static dependencies: object

Defined in: src/world/objects/ObjectDetector.ts:91

camera​

camera: typeof Camera = THREE.Camera

depth​

depth: typeof Depth = Depth

deviceCamera​

deviceCamera: typeof XRDeviceCamera = XRDeviceCamera

options​

options: typeof WorldOptions = WorldOptions

rendererHolder​

rendererHolder: typeof RendererHolder = RendererHolder


optionalDependencies​

static optionalDependencies: object

Defined in: src/world/objects/ObjectDetector.ts:99

ai​

ai: typeof AI = AI

aiOptions​

aiOptions: typeof AIOptions = AIOptions

Accessors​

targetDevice​

Get Signature​

get targetDevice(): string

Defined in: src/world/objects/ObjectDetector.ts:146

Target device profile used to look up RGB camera intrinsics and pose for converting detection bounding boxes into world space. Follows the device camera's targetDevice unless assigned here. Assign undefined to follow it again; unknown profile keys throw.

Returns​

string

Set Signature​

set targetDevice(targetDevice): void

Defined in: src/world/objects/ObjectDetector.ts:154

Parameters​
targetDevice​

string | undefined

Returns​

void

Methods​

clear()​

clear(): ObjectDetector

Defined in: src/world/objects/ObjectDetector.ts:537

Removes all currently detected objects from the scene and internal tracking.

Returns​

ObjectDetector

Overrides​

Script.clear


dispose()​

dispose(): void

Defined in: src/world/objects/ObjectDetector.ts:569

Called when the script is removed from the scene. Opposite of init.

Returns​

void

Overrides​

Script.dispose


get()​

get<T>(label): DetectedObject<T>[]

Defined in: src/world/objects/ObjectDetector.ts:523

Retrieves a list of currently detected objects.

Type Parameters​

T​

T = null

Parameters​

label​

null = null

The semantic label to filter by (e.g., 'chair'). If null, all objects are returned.

Returns​

DetectedObject<T>[]

An array of Object instances.


init()​

init(__namedParameters): void

Defined in: src/world/objects/ObjectDetector.ts:163

Initializes the ObjectDetector.

Parameters​

__namedParameters​
ai?​

AI

aiOptions?​

AIOptions

camera​

PerspectiveCamera

depth​

Depth

deviceCamera​

XRDeviceCamera

options​

WorldOptions

rendererHolder​

RendererHolder

Returns​

void

Overrides​

Script.init


initPhysics()​

initPhysics(_physics): void | Promise<void>

Defined in: src/core/Script.ts:119

Enables depth-aware interactions with physics. See /samples/advanced/ballpit

Parameters​

_physics​

Physics

Returns​

void | Promise<void>

Inherited from​

Script.initPhysics


onHoverEnter()​

onHoverEnter(_event): void

Defined in: src/core/Script.ts:222

Called when a source starts hovering over this object.

Parameters​

_event​

HoverEvent

The hover source, target, surface, and intersection. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onHoverEnter


onHoverExit()​

onHoverExit(_event): void

Defined in: src/core/Script.ts:228

Called when a source stops hovering over this object.

Parameters​

_event​

HoverEvent

The hover source, target, surface, and intersection. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onHoverExit


onHovering()​

onHovering(_event): void

Defined in: src/core/Script.ts:234

Called while a source hovers over this object.

Parameters​

_event​

HoverEvent

The hover source, target, surface, and intersection. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onHovering


onKeyDown()​

onKeyDown(_event): void

Defined in: src/core/Script.ts:159

Called on keyboard keypress.

Parameters​

_event​

KeyEvent

Event containing .code to read the keyboard key.

Returns​

void

Inherited from​

Script.onKeyDown


onKeyUp()​

onKeyUp(_event): void

Defined in: src/core/Script.ts:160

Parameters​

_event​

KeyEvent

Returns​

void

Inherited from​

Script.onKeyUp


onLongSelect()​

onLongSelect(_event): void

Defined in: src/core/Script.ts:153

Called when an object selection reaches the long-select delay.

Parameters​

_event​

LongSelectEvent

Returns​

void

Inherited from​

Script.onLongSelect


onObjectGrabbing()​

onObjectGrabbing(_event): void

Defined in: src/core/Script.ts:260

Called every frame a hand is grabbing this object.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabbing


onObjectGrabEnd()​

onObjectGrabEnd(_event): void

Defined in: src/core/Script.ts:265

Called when a hand stops grabbing this object. This ends built-in direct-touch manipulation without ending contact.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabEnd


onObjectGrabStart()​

onObjectGrabStart(_event): void

Defined in: src/core/Script.ts:256

Called when a hand starts grabbing this object (touching + pinching). A grab starts built-in direct-touch manipulation when enabled.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabStart


onObjectLongSelect()​

onObjectLongSelect(_event): void

Defined in: src/core/Script.ts:210

Called once when a captured selection is held for the long-select delay. Manipulation captures do not emit this callback.

Parameters​

_event​

LongSelectEvent

The controller and completed hold duration. Call event.stopPropagation() to stop bubbling to ancestor Scripts. Call event.stopGlobalPropagation() to suppress global selection callbacks for the rest of this capture

Returns​

void

Inherited from​

Script.onObjectLongSelect


onObjectManipulate()​

onObjectManipulate(_event): void

Defined in: src/core/Script.ts:216

Called for each phase of an automatic object manipulation. Call event.stopPropagation() to stop bubbling. Calling preventDefault() on a start event suppresses the automatic action.

Parameters​

_event​

ManipulationEvent

Returns​

void

Inherited from​

Script.onObjectManipulate


onObjectSelectEnd()​

onObjectSelectEnd(_event): void

Defined in: src/core/Script.ts:201

Called when a source stops selecting the object this Script represents.

Parameters​

_event​

SelectEndEvent

The completed state and end reason. Call event.stopPropagation() to stop bubbling to ancestor Scripts. Call event.stopGlobalPropagation() to suppress global selection callbacks for the rest of this capture

Returns​

void

Inherited from​

Script.onObjectSelectEnd


onObjectSelectStart()​

onObjectSelectStart(_event): void

Defined in: src/core/Script.ts:193

Called when a source starts selecting the object this Script represents.

Parameters​

_event​

SelectEvent

event.target is the logical object and event.source.controller identifies the controller. Call event.stopPropagation() to stop bubbling to ancestor Scripts. Call event.stopGlobalPropagation() to suppress global selection callbacks for the rest of this capture

Returns​

void

Inherited from​

Script.onObjectSelectStart


onObjectTouchEnd()​

onObjectTouchEnd(_event): void

Defined in: src/core/Script.ts:251

Called when a hand's index finger stops touching this object. This ends the default selection lifecycle after the touch callback.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouchEnd


onObjectTouching()​

onObjectTouching(_event): void

Defined in: src/core/Script.ts:246

Called every frame that a hand's index finger is touching this object. The object remains selected during these frames unless touch selection was prevented when contact started.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouching


onObjectTouchStart()​

onObjectTouchStart(_event): void

Defined in: src/core/Script.ts:240

Called when a hand's index finger starts touching this object. Direct touch starts the object's selection lifecycle by default. Call event.preventDefault() to handle contact without selecting.

Parameters​

_event​

ObjectTouchStartEvent

Returns​

void

Inherited from​

Script.onObjectTouchStart


onSelect()​

onSelect(_event): void

Defined in: src/core/Script.ts:145

Called whenever pinch / mouse click successfully completes, globally.

Parameters​

_event​

SelectEvent

The interaction source and completed target.

Returns​

void

Inherited from​

Script.onSelect


onSelectEnd()​

onSelectEnd(_event): void

Defined in: src/core/Script.ts:139

Called whenever pinch / mouse click discontinues, globally.

Parameters​

_event​

SelectEndEvent

The completed state and end reason.

Returns​

void

Inherited from​

Script.onSelectEnd


onSelecting()​

onSelecting(_event): void

Defined in: src/core/Script.ts:150

Called whenever pinch / mouse click is happening, globally.

Parameters​

_event​

SelectEvent

Returns​

void

Inherited from​

Script.onSelecting


onSelectStart()​

onSelectStart(_event): void

Defined in: src/core/Script.ts:133

Called whenever pinch / mouse click starts, globally.

Parameters​

_event​

SelectEvent

The interaction source and optional captured target.

Returns​

void

Inherited from​

Script.onSelectStart


onSimulatorStarted()​

onSimulatorStarted(): void

Defined in: src/core/Script.ts:125

Returns​

void

Inherited from​

Script.onSimulatorStarted


onSqueeze()​

onSqueeze(_event): void

Defined in: src/core/Script.ts:182

Called whenever gamepad trigger successfully completes, globally.

Parameters​

_event​

SelectEvent

event.source.controller identifies the controller.

Returns​

void

Inherited from​

Script.onSqueeze


onSqueezeEnd()​

onSqueezeEnd(_event): void

Defined in: src/core/Script.ts:171

Called whenever gamepad trigger stops, globally.

Parameters​

_event​

SelectEvent

event.source.controller identifies the controller.

Returns​

void

Inherited from​

Script.onSqueezeEnd


onSqueezeStart()​

onSqueezeStart(_event): void

Defined in: src/core/Script.ts:166

Called whenever gamepad trigger starts, globally.

Parameters​

_event​

SelectEvent

event.source.controller identifies the controller.

Returns​

void

Inherited from​

Script.onSqueezeStart


onSqueezing()​

onSqueezing(_event): void

Defined in: src/core/Script.ts:176

Called whenever gamepad is being triggered, globally.

Parameters​

_event​

SelectEvent

Returns​

void

Inherited from​

Script.onSqueezing


onXRSessionEnded()​

onXRSessionEnded(): void

Defined in: src/core/Script.ts:123

Returns​

void

Inherited from​

Script.onXRSessionEnded


onXRSessionStarted()​

onXRSessionStarted(_session?): void

Defined in: src/core/Script.ts:122

Parameters​

_session?​

XRSession

Returns​

void

Inherited from​

Script.onXRSessionStarted


physicsStep()​

physicsStep(): void

Defined in: src/core/Script.ts:120

Returns​

void

Inherited from​

Script.physicsStep


runDetection()​

runDetection<T>(options): Promise<DetectedObject<T>[]>

Defined in: src/world/objects/ObjectDetector.ts:281

Runs object detection or returns the ongoing detection promise.

  • If continuous detection is started (has active clients), returns the promise for the next detection result.
  • If continuous detection is not started, performs a one-off detection and returns the result. If a one-off detection is already in progress, returns the promise for that ongoing detection.
  • Supplying a snapshot or backend queues a separate one-off run after pending detections, without changing the continuous detector's options. Supplied snapshots retain the camera pose and depth at submission time. Simulator ground truth, when installed, still takes precedence.

Type Parameters​

T​

T = null

Parameters​

options​

ObjectDetectionOptions = {}

Optional inputs for this run only.

Returns​

Promise<DetectedObject<T>[]>

A promise that resolves with an array of detected DetectedObject instances.

Throws​

If an explicit backend or snapshot is invalid, or the detector is disposed before a queued request starts.


setSimulatorSource()​

setSimulatorSource(source?): ObjectDetector

Defined in: src/world/objects/ObjectDetector.ts:356

Installs or removes the desktop simulator's ground-truth detector.

Parameters​

source?​

SimulatorObjectDetectionSource

Returns​

ObjectDetector


showDebugVisualizations()​

showDebugVisualizations(visible): void

Defined in: src/world/objects/ObjectDetector.ts:563

Toggles the visibility of all debug visualizations for detected objects.

Parameters​

visible​

boolean = true

Whether the visualizations should be visible.

Returns​

void


start()​

start(client): void

Defined in: src/world/objects/ObjectDetector.ts:203

Starts continuous object detection for the given client. Detection starts on the next update after initialization.

Parameters​

client​

object

The client object requesting object detection.

Returns​

void


stop()​

stop(client): void

Defined in: src/world/objects/ObjectDetector.ts:215

Stops continuous object detection for the given client. If this was the last client, stops the background detection loop.

Parameters​

client​

object

The client object that no longer needs object detection.

Returns​

void


update()​

update(): void

Defined in: src/world/objects/ObjectDetector.ts:223

Called per frame by the engine. If there are active clients, ensures the continuous object detection is running.

Returns​

void

Overrides​

Script.update