Skip to main content

Class: AI

Defined in: src/ai/AI.ts:56

AI Interface to wrap different AI models (primarily Gemini) Handles both traditional query-based AI interactions and real-time live sessions

Features:

  • Text and multimodal queries
  • Real-time audio/video AI sessions (Gemini Live)
  • Advanced API key management with multiple sources
  • Session locking to prevent concurrent operations

The URL param and key.json shortcut is only for demonstration and prototyping practice and we strongly suggest not using it for production or deployment purposes. One should set up a proper server to converse with AI servers in deployment.

API Key Management Features:

  1. Multiple Key Sources (Priority Order):
    • Model option
    • Generic and model-specific URL parameters
    • Current-page memory
    • keys.json file
  2. keys.json Support:
    • Structure: {"gemini": {"apiKey": "YOUR_KEY_HERE"}}
    • Automatically loads if present

Extends​

Constructors​

Constructor​

new AI(): AI

Defined in: node_modules/@types/three/src/core/Object3D.d.ts:98

This creates a new Object3D object.

Returns​

AI

Inherited from​

Script.constructor

Properties​

editorIcon​

editorIcon: string = 'network_intelligence'

Defined in: src/ai/AI.ts:58

isXRScript​

isXRScript: boolean = true

Defined in: src/core/Script.ts:99

Inherited from​

Script.isXRScript

keysCache?​

optional keysCache: KeysJson

Defined in: src/ai/AI.ts:62

lock​

lock: boolean = false

Defined in: src/ai/AI.ts:60

model?​

optional model: ModelClass

Defined in: src/ai/AI.ts:59

options​

options: AIOptions

Defined in: src/ai/AI.ts:61

xb?​

optional xb: XBObjectOptions

Defined in: src/interaction/InteractionTypes.ts:32

Inherited from​

Script.xb

dependencies​

static dependencies: object

Defined in: src/ai/AI.ts:57

aiOptions​

aiOptions: typeof AIOptions = AIOptions

Methods​

dispose()​

dispose(): void

Defined in: src/core/Script.ts:263

Called when the script is removed from the scene. Opposite of init.

Returns​

void

Inherited from​

Script.dispose

generate()​

generate(prompt, type, systemInstruction, model): Promise<string | undefined>

Defined in: src/ai/AI.ts:280

Parameters​

prompt​

string | string[]

type​

"image" = 'image'

systemInstruction​

string = 'Generate an image'

model​

undefined = undefined

Returns​

Promise<string | undefined>

getLiveSessionStatus()​

getLiveSessionStatus(): object

Defined in: src/ai/AI.ts:265

Returns​

object

hasSession​

hasSession: boolean

isActive​

isActive: boolean

isAvailable​

isAvailable: boolean | typeof Modality | undefined

hasApiKey()​

hasApiKey(): Promise<boolean | "" | null>

Defined in: src/ai/AI.ts:312

Check if the current model has an API key available from any source

Returns​

Promise<boolean | "" | null>

True if API key is available

init()​

init(__namedParameters): Promise<void>

Defined in: src/ai/AI.ts:83

Initializes an instance with XR controllers, grips, hands, and default options. We allow all scripts to quickly access its user (e.g., user.isSelecting(), user.hands), world (e.g., physical depth mesh, lighting estimation, and recognized objects), and scene (the root of three.js's scene graph). If this returns a promise, we will wait for it.

Parameters​

__namedParameters​
aiOptions​

AIOptions

Returns​

Promise<void>

Overrides​

Script.init

initializeModel()​

initializeModel(ModelClass, modelOptions): Promise<void>

Defined in: src/ai/AI.ts:106

Parameters​

ModelClass​

typeof Gemini | typeof OpenAI

modelOptions​

ModelOptions

Returns​

Promise<void>

initPhysics()​

initPhysics(_physics): void | Promise<void>

Defined in: src/core/Script.ts:118

Enables depth-aware interactions with physics. See /samples/advanced/ballpit

Parameters​

_physics​

Physics

Returns​

void | Promise<void>

Inherited from​

Script.initPhysics

isAvailable()​

isAvailable(): boolean | undefined

Defined in: src/ai/AI.ts:196

Returns​

boolean | undefined

isLiveAvailable()​

isLiveAvailable(): false | typeof Modality | undefined

Defined in: src/ai/AI.ts:272

Returns​

false | typeof Modality | undefined

isValidApiKey()​

isValidApiKey(key): boolean | ""

Defined in: src/ai/AI.ts:192

Parameters​

key​

string

Returns​

boolean | ""

loadKeysFromFile()​

loadKeysFromFile(): Promise<KeysJson | null>

Defined in: src/ai/AI.ts:68

Load API keys from keys.json file if available Parsed keys object or null if not found

Returns​

Promise<KeysJson | null>

onHoverEnter()​

onHoverEnter(_event): void

Defined in: src/core/Script.ts:215

Called when a source starts hovering over this object.

Parameters​

_event​

HoverEvent

The hover source, target, surface, and intersection. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onHoverEnter

onHoverExit()​

onHoverExit(_event): void

Defined in: src/core/Script.ts:221

Called when a source stops hovering over this object.

Parameters​

_event​

HoverEvent

The hover source, target, surface, and intersection. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onHoverExit

onHovering()​

onHovering(_event): void

Defined in: src/core/Script.ts:227

Called while a source hovers over this object.

Parameters​

_event​

HoverEvent

The hover source, target, surface, and intersection. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onHovering

onKeyDown()​

onKeyDown(_event): void

Defined in: src/core/Script.ts:158

Called on keyboard keypress.

Parameters​

_event​

KeyEvent

Event containing .code to read the keyboard key.

Returns​

void

Inherited from​

Script.onKeyDown

onKeyUp()​

onKeyUp(_event): void

Defined in: src/core/Script.ts:159

Parameters​

_event​

KeyEvent

Returns​

void

Inherited from​

Script.onKeyUp

onLongSelect()​

onLongSelect(_event): void

Defined in: src/core/Script.ts:152

Called when an object selection reaches the long-select delay.

Parameters​

_event​

LongSelectEvent

Returns​

void

Inherited from​

Script.onLongSelect

onObjectGrabbing()​

onObjectGrabbing(_event): void

Defined in: src/core/Script.ts:253

Called every frame a hand is grabbing this object.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabbing

onObjectGrabEnd()​

onObjectGrabEnd(_event): void

Defined in: src/core/Script.ts:258

Called when a hand stops grabbing this object. This ends built-in direct-touch manipulation without ending contact.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabEnd

onObjectGrabStart()​

onObjectGrabStart(_event): void

Defined in: src/core/Script.ts:249

Called when a hand starts grabbing this object (touching + pinching). A grab starts built-in direct-touch manipulation when enabled.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabStart

onObjectLongSelect()​

onObjectLongSelect(_event): void

Defined in: src/core/Script.ts:203

Called once when a captured selection is held for the long-select delay. Manipulation captures do not emit this callback.

Parameters​

_event​

LongSelectEvent

The controller and completed hold duration. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onObjectLongSelect

onObjectManipulate()​

onObjectManipulate(_event): void

Defined in: src/core/Script.ts:209

Called for each phase of an automatic object manipulation. Call event.stopPropagation() to stop bubbling. Calling preventDefault() on a start event suppresses the automatic action.

Parameters​

_event​

ManipulationEvent

Returns​

void

Inherited from​

Script.onObjectManipulate

onObjectSelectEnd()​

onObjectSelectEnd(_event): void

Defined in: src/core/Script.ts:196

Called when a source stops selecting the object this Script represents.

Parameters​

_event​

SelectEndEvent

The completed state and end reason. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onObjectSelectEnd

onObjectSelectStart()​

onObjectSelectStart(_event): void

Defined in: src/core/Script.ts:190

Called when a source starts selecting the object this Script represents.

Parameters​

_event​

SelectEvent

event.target is the logical object and event.source.controller identifies the controller. Call event.stopPropagation() to stop bubbling to ancestor Scripts.

Returns​

void

Inherited from​

Script.onObjectSelectStart

onObjectTouchEnd()​

onObjectTouchEnd(_event): void

Defined in: src/core/Script.ts:244

Called when a hand's index finger stops touching this object. This ends the default selection lifecycle after the touch callback.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouchEnd

onObjectTouching()​

onObjectTouching(_event): void

Defined in: src/core/Script.ts:239

Called every frame that a hand's index finger is touching this object. The object remains selected during these frames unless touch selection was prevented when contact started.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouching

onObjectTouchStart()​

onObjectTouchStart(_event): void

Defined in: src/core/Script.ts:233

Called when a hand's index finger starts touching this object. Direct touch starts the object's selection lifecycle by default. Call event.preventDefault() to handle contact without selecting.

Parameters​

_event​

ObjectTouchStartEvent

Returns​

void

Inherited from​

Script.onObjectTouchStart

onSelect()​

onSelect(_event): void

Defined in: src/core/Script.ts:144

Called whenever pinch / mouse click successfully completes, globally.

Parameters​

_event​

SelectEvent

The interaction source and completed target.

Returns​

void

Inherited from​

Script.onSelect

onSelectEnd()​

onSelectEnd(_event): void

Defined in: src/core/Script.ts:138

Called whenever pinch / mouse click discontinues, globally.

Parameters​

_event​

SelectEndEvent

The completed state and end reason.

Returns​

void

Inherited from​

Script.onSelectEnd

onSelecting()​

onSelecting(_event): void

Defined in: src/core/Script.ts:149

Called whenever pinch / mouse click is happening, globally.

Parameters​

_event​

SelectEvent

Returns​

void

Inherited from​

Script.onSelecting

onSelectStart()​

onSelectStart(_event): void

Defined in: src/core/Script.ts:132

Called whenever pinch / mouse click starts, globally.

Parameters​

_event​

SelectEvent

The interaction source and optional captured target.

Returns​

void

Inherited from​

Script.onSelectStart

onSimulatorStarted()​

onSimulatorStarted(): void

Defined in: src/core/Script.ts:124

Returns​

void

Inherited from​

Script.onSimulatorStarted

onSqueeze()​

onSqueeze(_event): void

Defined in: src/core/Script.ts:181

Called whenever gamepad trigger successfully completes, globally.

Parameters​

_event​

SelectEvent

event.source.controller identifies the controller.

Returns​

void

Inherited from​

Script.onSqueeze

onSqueezeEnd()​

onSqueezeEnd(_event): void

Defined in: src/core/Script.ts:170

Called whenever gamepad trigger stops, globally.

Parameters​

_event​

SelectEvent

event.source.controller identifies the controller.

Returns​

void

Inherited from​

Script.onSqueezeEnd

onSqueezeStart()​

onSqueezeStart(_event): void

Defined in: src/core/Script.ts:165

Called whenever gamepad trigger starts, globally.

Parameters​

_event​

SelectEvent

event.source.controller identifies the controller.

Returns​

void

Inherited from​

Script.onSqueezeStart

onSqueezing()​

onSqueezing(_event): void

Defined in: src/core/Script.ts:175

Called whenever gamepad is being triggered, globally.

Parameters​

_event​

SelectEvent

Returns​

void

Inherited from​

Script.onSqueezing

onXRSessionEnded()​

onXRSessionEnded(): void

Defined in: src/core/Script.ts:122

Returns​

void

Inherited from​

Script.onXRSessionEnded

onXRSessionStarted()​

onXRSessionStarted(_session?): void

Defined in: src/core/Script.ts:121

Parameters​

_session?​

XRSession

Returns​

void

Inherited from​

Script.onXRSessionStarted

physicsStep()​

physicsStep(): void

Defined in: src/core/Script.ts:119

Returns​

void

Inherited from​

Script.physicsStep

query()​

query(input, tools?): Promise<string | GeminiResponse | null>

Defined in: src/ai/AI.ts:200

Parameters​

input​

GeminiQueryInput | { prompt: string; }

tools?​

never[]

Returns​

Promise<string | GeminiResponse | null>

resolveApiKey()​

resolveApiKey(modelOptions): Promise<string | null>

Defined in: src/ai/AI.ts:149

Parameters​

modelOptions​

ModelOptions

Returns​

Promise<string | null>

sendRealtimeInput()​

sendRealtimeInput(input): false | void

Defined in: src/ai/AI.ts:260

Parameters​

input​

LiveSendRealtimeInputParameters

Returns​

false | void

sendToolResponse()​

sendToolResponse(response): void

Defined in: src/ai/AI.ts:254

Parameters​

response​

LiveSendToolResponseParameters

Returns​

void

setLiveCallbacks()​

setLiveCallbacks(callbacks): Promise<void>

Defined in: src/ai/AI.ts:248

Parameters​

callbacks​

LiveCallbacks

Returns​

Promise<void>

startLiveSession()​

startLiveSession(config, model?): Promise<Session>

Defined in: src/ai/AI.ts:220

Parameters​

config​

LiveConnectConfig = {}

model?​

string

Returns​

Promise<Session>

stopLiveSession()​

stopLiveSession(): Promise<void>

Defined in: src/ai/AI.ts:239

Returns​

Promise<void>

update()​

update(_time?, _frame?): void

Defined in: src/core/Script.ts:113

Runs per frame.

Parameters​

_time?​

number

_frame?​

XRFrame

Returns​

void

Inherited from​

Script.update

createSampleKeysStructure()​

static createSampleKeysStructure(): object

Defined in: src/ai/AI.ts:301

Create a sample keys.json file structure for reference

Returns​

object

Sample keys.json structure

gemini​

gemini: object

gemini.apiKey​

apiKey: string = 'YOUR_GEMINI_API_KEY_HERE'

openai​

openai: object

openai.apiKey​

apiKey: string = 'YOUR_OPENAI_API_KEY_HERE'