Class: ObjectDetector
Defined in: src/world/objects/ObjectDetector.ts:90
Detects objects in the user's environment using a specified backend. It queries an AI model with the device camera feed and returns located objects with 2D and 3D positioning data.
Extends
Constructors
Constructor
new ObjectDetector():
ObjectDetector
Defined in: node_modules/@types/three/src/core/Object3D.d.ts:105
This creates a new Object3D object.
Returns
ObjectDetector
Inherited from
Properties
detectedObjects
detectedObjects:
DetectedObject<unknown>[] =[]
Defined in: src/world/objects/ObjectDetector.ts:127
The latest detected objects.
isXRScript
isXRScript:
boolean=true
Defined in: src/core/Script.ts:100
Inherited from
xb?
optionalxb:XBObjectOptions
Defined in: src/interaction/InteractionTypes.ts:34
Inherited from
dependencies
staticdependencies:object
Defined in: src/world/objects/ObjectDetector.ts:91
camera
camera: typeof
Camera=THREE.Camera
depth
depth: typeof
Depth=Depth
deviceCamera
deviceCamera: typeof
XRDeviceCamera=XRDeviceCamera
options
options: typeof
WorldOptions=WorldOptions
rendererHolder
rendererHolder: typeof
RendererHolder=RendererHolder
optionalDependencies
staticoptionalDependencies:object
Defined in: src/world/objects/ObjectDetector.ts:99
ai
ai: typeof
AI=AI
aiOptions
aiOptions: typeof
AIOptions=AIOptions
Accessors
targetDevice
Get Signature
get targetDevice():
string
Defined in: src/world/objects/ObjectDetector.ts:146
Target device profile used to look up RGB camera intrinsics and pose
for converting detection bounding boxes into world space. Follows the
device camera's targetDevice unless assigned here. Assign undefined to
follow it again; unknown profile keys throw.
Returns
string
Set Signature
set targetDevice(
targetDevice):void
Defined in: src/world/objects/ObjectDetector.ts:154
Parameters
targetDevice
string | undefined
Returns
void
Methods
clear()
clear():
ObjectDetector
Defined in: src/world/objects/ObjectDetector.ts:537
Removes all currently detected objects from the scene and internal tracking.
Returns
ObjectDetector
Overrides
Script.clear
dispose()
dispose():
void
Defined in: src/world/objects/ObjectDetector.ts:569
Called when the script is removed from the scene. Opposite of init.
Returns
void
Overrides
get()
get<
T>(label):DetectedObject<T>[]
Defined in: src/world/objects/ObjectDetector.ts:523
Retrieves a list of currently detected objects.
Type Parameters
T
T = null
Parameters
label
null = null
The semantic label to filter by (e.g., 'chair'). If null, all objects are returned.
Returns
DetectedObject<T>[]
An array of Object instances.
init()
init(
__namedParameters):void
Defined in: src/world/objects/ObjectDetector.ts:163
Initializes the ObjectDetector.
Parameters
__namedParameters
ai?
aiOptions?
camera
depth
deviceCamera
options
rendererHolder
RendererHolder
Returns
void
Overrides
initPhysics()
initPhysics(
_physics):void|Promise<void>
Defined in: src/core/Script.ts:119
Enables depth-aware interactions with physics. See /samples/advanced/ballpit
Parameters
_physics
Returns
void | Promise<void>
Inherited from
onHoverEnter()
onHoverEnter(
_event):void
Defined in: src/core/Script.ts:222
Called when a source starts hovering over this object.
Parameters
_event
The hover source, target, surface, and intersection.
Call event.stopPropagation() to stop bubbling to ancestor Scripts.
Returns
void
Inherited from
onHoverExit()
onHoverExit(
_event):void
Defined in: src/core/Script.ts:228
Called when a source stops hovering over this object.
Parameters
_event
The hover source, target, surface, and intersection.
Call event.stopPropagation() to stop bubbling to ancestor Scripts.
Returns
void
Inherited from
onHovering()
onHovering(
_event):void
Defined in: src/core/Script.ts:234
Called while a source hovers over this object.
Parameters
_event
The hover source, target, surface, and intersection.
Call event.stopPropagation() to stop bubbling to ancestor Scripts.
Returns
void
Inherited from
onKeyDown()
onKeyDown(
_event):void
Defined in: src/core/Script.ts:159
Called on keyboard keypress.
Parameters
_event
Event containing .code to read the keyboard key.
Returns
void
Inherited from
onKeyUp()
onKeyUp(
_event):void
Defined in: src/core/Script.ts:160
Parameters
_event
Returns
void
Inherited from
onLongSelect()
onLongSelect(
_event):void
Defined in: src/core/Script.ts:153
Called when an object selection reaches the long-select delay.
Parameters
_event
Returns
void
Inherited from
onObjectGrabbing()
onObjectGrabbing(
_event):void
Defined in: src/core/Script.ts:260
Called every frame a hand is grabbing this object.
Parameters
_event
Returns
void
Inherited from
onObjectGrabEnd()
onObjectGrabEnd(
_event):void
Defined in: src/core/Script.ts:265
Called when a hand stops grabbing this object. This ends built-in direct-touch manipulation without ending contact.
Parameters
_event
Returns
void
Inherited from
onObjectGrabStart()
onObjectGrabStart(
_event):void
Defined in: src/core/Script.ts:256
Called when a hand starts grabbing this object (touching + pinching). A grab starts built-in direct-touch manipulation when enabled.
Parameters
_event
Returns
void
Inherited from
onObjectLongSelect()
onObjectLongSelect(
_event):void
Defined in: src/core/Script.ts:210
Called once when a captured selection is held for the long-select delay. Manipulation captures do not emit this callback.
Parameters
_event
The controller and completed hold duration.
Call event.stopPropagation() to stop bubbling to ancestor Scripts.
Call event.stopGlobalPropagation() to suppress global selection callbacks
for the rest of this capture
Returns
void
Inherited from
onObjectManipulate()
onObjectManipulate(
_event):void
Defined in: src/core/Script.ts:216
Called for each phase of an automatic object manipulation. Call
event.stopPropagation() to stop bubbling. Calling preventDefault()
on a start event suppresses the automatic action.
Parameters
_event
Returns
void
Inherited from
onObjectSelectEnd()
onObjectSelectEnd(
_event):void
Defined in: src/core/Script.ts:201
Called when a source stops selecting the object this Script represents.
Parameters
_event
The completed state and end reason.
Call event.stopPropagation() to stop bubbling to ancestor Scripts.
Call event.stopGlobalPropagation() to suppress global selection callbacks
for the rest of this capture
Returns
void
Inherited from
onObjectSelectStart()
onObjectSelectStart(
_event):void
Defined in: src/core/Script.ts:193
Called when a source starts selecting the object this Script represents.
Parameters
_event
event.target is the logical object and
event.source.controller identifies the controller.
Call event.stopPropagation() to stop bubbling to ancestor Scripts.
Call event.stopGlobalPropagation() to suppress global selection callbacks
for the rest of this capture
Returns
void
Inherited from
onObjectTouchEnd()
onObjectTouchEnd(
_event):void
Defined in: src/core/Script.ts:251
Called when a hand's index finger stops touching this object. This ends the default selection lifecycle after the touch callback.
Parameters
_event
Returns
void
Inherited from
onObjectTouching()
onObjectTouching(
_event):void
Defined in: src/core/Script.ts:246
Called every frame that a hand's index finger is touching this object. The object remains selected during these frames unless touch selection was prevented when contact started.
Parameters
_event
Returns
void
Inherited from
onObjectTouchStart()
onObjectTouchStart(
_event):void
Defined in: src/core/Script.ts:240
Called when a hand's index finger starts touching this object.
Direct touch starts the object's selection lifecycle by default. Call
event.preventDefault() to handle contact without selecting.
Parameters
_event
Returns
void
Inherited from
onSelect()
onSelect(
_event):void
Defined in: src/core/Script.ts:145
Called whenever pinch / mouse click successfully completes, globally.
Parameters
_event
The interaction source and completed target.
Returns
void
Inherited from
onSelectEnd()
onSelectEnd(
_event):void
Defined in: src/core/Script.ts:139
Called whenever pinch / mouse click discontinues, globally.
Parameters
_event
The completed state and end reason.
Returns
void
Inherited from
onSelecting()
onSelecting(
_event):void
Defined in: src/core/Script.ts:150
Called whenever pinch / mouse click is happening, globally.
Parameters
_event
Returns
void
Inherited from
onSelectStart()
onSelectStart(
_event):void
Defined in: src/core/Script.ts:133
Called whenever pinch / mouse click starts, globally.
Parameters
_event
The interaction source and optional captured target.
Returns
void
Inherited from
onSimulatorStarted()
onSimulatorStarted():
void
Defined in: src/core/Script.ts:125
Returns
void
Inherited from
onSqueeze()
onSqueeze(
_event):void
Defined in: src/core/Script.ts:182
Called whenever gamepad trigger successfully completes, globally.
Parameters
_event
event.source.controller identifies the controller.
Returns
void
Inherited from
onSqueezeEnd()
onSqueezeEnd(
_event):void
Defined in: src/core/Script.ts:171
Called whenever gamepad trigger stops, globally.
Parameters
_event
event.source.controller identifies the controller.
Returns
void
Inherited from
onSqueezeStart()
onSqueezeStart(
_event):void
Defined in: src/core/Script.ts:166
Called whenever gamepad trigger starts, globally.
Parameters
_event
event.source.controller identifies the controller.
Returns
void
Inherited from
onSqueezing()
onSqueezing(
_event):void
Defined in: src/core/Script.ts:176
Called whenever gamepad is being triggered, globally.
Parameters
_event
Returns
void
Inherited from
onXRSessionEnded()
onXRSessionEnded():
void
Defined in: src/core/Script.ts:123
Returns
void
Inherited from
onXRSessionStarted()
onXRSessionStarted(
_session?):void
Defined in: src/core/Script.ts:122
Parameters
_session?
Returns
void
Inherited from
physicsStep()
physicsStep():
void
Defined in: src/core/Script.ts:120
Returns
void
Inherited from
runDetection()
runDetection<
T>(options):Promise<DetectedObject<T>[]>
Defined in: src/world/objects/ObjectDetector.ts:281
Runs object detection or returns the ongoing detection promise.
- If continuous detection is started (has active clients), returns the promise for the next detection result.
- If continuous detection is not started, performs a one-off detection and returns the result. If a one-off detection is already in progress, returns the promise for that ongoing detection.
- Supplying a snapshot or backend queues a separate one-off run after pending detections, without changing the continuous detector's options. Supplied snapshots retain the camera pose and depth at submission time. Simulator ground truth, when installed, still takes precedence.
Type Parameters
T
T = null
Parameters
options
Optional inputs for this run only.
Returns
Promise<DetectedObject<T>[]>
A promise that resolves with an
array of detected DetectedObject instances.
Throws
If an explicit backend or snapshot is invalid, or the detector is disposed before a queued request starts.
setSimulatorSource()
setSimulatorSource(
source?):ObjectDetector
Defined in: src/world/objects/ObjectDetector.ts:356
Installs or removes the desktop simulator's ground-truth detector.
Parameters
source?
SimulatorObjectDetectionSource
Returns
ObjectDetector
showDebugVisualizations()
showDebugVisualizations(
visible):void
Defined in: src/world/objects/ObjectDetector.ts:563
Toggles the visibility of all debug visualizations for detected objects.
Parameters
visible
boolean = true
Whether the visualizations should be visible.
Returns
void
start()
start(
client):void
Defined in: src/world/objects/ObjectDetector.ts:203
Starts continuous object detection for the given client. Detection starts on the next update after initialization.
Parameters
client
object
The client object requesting object detection.
Returns
void
stop()
stop(
client):void
Defined in: src/world/objects/ObjectDetector.ts:215
Stops continuous object detection for the given client. If this was the last client, stops the background detection loop.
Parameters
client
object
The client object that no longer needs object detection.
Returns
void
update()
update():
void
Defined in: src/world/objects/ObjectDetector.ts:223
Called per frame by the engine. If there are active clients, ensures the continuous object detection is running.
Returns
void