Skip to main content

Class: AI

Defined in: src/ai/AI.ts:52

AI Interface to wrap different AI models (primarily Gemini) Handles both traditional query-based AI interactions and real-time live sessions

Features:

  • Text and multimodal queries
  • Real-time audio/video AI sessions (Gemini Live)
  • Advanced API key management with multiple sources
  • Session locking to prevent concurrent operations

The URL param and key.json shortcut is only for demonstration and prototyping practice and we strongly suggest not using it for production or deployment purposes. One should set up a proper server to converse with AI servers in deployment.

API Key Management Features:

  1. Multiple Key Sources (Priority Order):
    • URL Parameter: ?key=<api_key>
    • keys.json file: Local configuration file
    • User Prompt: Interactive fallback
  2. keys.json Support:
    • Structure: {"gemini": {"apiKey": "YOUR_KEY_HERE"}}
    • Automatically loads if present

Extends​

Constructors​

Constructor​

new AI(): AI

Defined in: node_modules/@types/three/src/core/Object3D.d.ts:95

This creates a new Object3D object.

Returns​

AI

Inherited from​

Script.constructor

Properties​

isXRScript​

isXRScript: boolean = true

Defined in: src/core/Script.ts:53

Inherited from​

Script.isXRScript

keysCache?​

optional keysCache: KeysJson

Defined in: src/ai/AI.ts:58

lock​

lock: boolean = false

Defined in: src/ai/AI.ts:56

model?​

optional model: ModelClass

Defined in: src/ai/AI.ts:55

options​

options: AIOptions

Defined in: src/ai/AI.ts:57

ux​

ux: UX

Defined in: src/core/Script.ts:52

Inherited from​

Script.ux

dependencies​

static dependencies: object

Defined in: src/ai/AI.ts:53

aiOptions​

aiOptions: typeof AIOptions = AIOptions

Methods​

dispose()​

dispose(): void

Defined in: src/core/Script.ts:196

Called when the script is removed from the scene. Opposite of init.

Returns​

void

Inherited from​

Script.dispose

generate()​

generate(prompt, type, systemInstruction, model): Promise<undefined | string | void>

Defined in: src/ai/AI.ts:248

Parameters​

prompt​

string | string[]

type​

"image" = 'image'

systemInstruction​

string = 'Generate an image'

model​

string = 'gemini-2.5-flash-image-preview'

Returns​

Promise<undefined | string | void>

getLiveSessionStatus()​

getLiveSessionStatus(): object

Defined in: src/ai/AI.ts:230

Returns​

object

hasSession​

hasSession: boolean

isActive​

isActive: boolean

isAvailable​

isAvailable: undefined | boolean | typeof Modality

hasApiKey()​

hasApiKey(): Promise<null | boolean | "">

Defined in: src/ai/AI.ts:270

Check if the current model has an API key available from any source

Returns​

Promise<null | boolean | "">

True if API key is available

init()​

init(__namedParameters): Promise<void>

Defined in: src/ai/AI.ts:80

Initializes an instance with XR controllers, grips, hands, raycaster, and default options. We allow all scripts to quickly access its user (e.g., user.isSelecting(), user.hands), world (e.g., physical depth mesh, lighting estimation, and recognized objects), and scene (the root of three.js's scene graph). If this returns a promise, we will wait for it.

Parameters​

__namedParameters​
aiOptions​

AIOptions

Returns​

Promise<void>

Overrides​

Script.init

initializeModel()​

initializeModel(ModelClass, modelOptions): Promise<void>

Defined in: src/ai/AI.ts:104

Parameters​

ModelClass​

typeof Gemini | typeof OpenAI

modelOptions​

ModelOptions

Returns​

Promise<void>

initPhysics()​

initPhysics(_physics): void | Promise<void>

Defined in: src/core/Script.ts:72

Enables depth-aware interactions with physics. See /demos/ballpit

Parameters​

_physics​

Physics

Returns​

void | Promise<void>

Inherited from​

Script.initPhysics

isAvailable()​

isAvailable(): undefined | boolean

Defined in: src/ai/AI.ts:171

Returns​

undefined | boolean

isLiveAvailable()​

isLiveAvailable(): undefined | false | typeof Modality

Defined in: src/ai/AI.ts:237

Returns​

undefined | false | typeof Modality

isValidApiKey()​

isValidApiKey(key): boolean | ""

Defined in: src/ai/AI.ts:167

Parameters​

key​

string

Returns​

boolean | ""

loadKeysFromFile()​

loadKeysFromFile(): Promise<null | KeysJson>

Defined in: src/ai/AI.ts:64

Load API keys from keys.json file if available Parsed keys object or null if not found

Returns​

Promise<null | KeysJson>

onHoverEnter()​

onHoverEnter(_controller): void

Defined in: src/core/Script.ts:157

Called when the controller starts hovering over this object with reticle.

Parameters​

_controller​

Object3D

An XR controller.

Returns​

void

Inherited from​

Script.onHoverEnter

onHoverExit()​

onHoverExit(_controller): void

Defined in: src/core/Script.ts:162

Called when the controller hovers over this object with reticle.

Parameters​

_controller​

Object3D

An XR controller.

Returns​

void

Inherited from​

Script.onHoverExit

onHovering()​

onHovering(_controller): void

Defined in: src/core/Script.ts:167

Called when the controller hovers over this object with reticle.

Parameters​

_controller​

Object3D

An XR controller.

Returns​

void

Inherited from​

Script.onHovering

onKeyDown()​

onKeyDown(_event): void

Defined in: src/core/Script.ts:109

Called on keyboard keypress.

Parameters​

_event​

KeyEvent

Event containing .code to read the keyboard key.

Returns​

void

Inherited from​

Script.onKeyDown

onKeyUp()​

onKeyUp(_event): void

Defined in: src/core/Script.ts:110

Parameters​

_event​

KeyEvent

Returns​

void

Inherited from​

Script.onKeyUp

onObjectGrabbing()​

onObjectGrabbing(_event): void

Defined in: src/core/Script.ts:187

Called every frame a hand is grabbing this object.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabbing

onObjectGrabEnd()​

onObjectGrabEnd(_event): void

Defined in: src/core/Script.ts:191

Called when a hand stops grabbing this object.

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabEnd

onObjectGrabStart()​

onObjectGrabStart(_event): void

Defined in: src/core/Script.ts:183

Called when a hand starts grabbing this object (touching + pinching).

Parameters​

_event​

ObjectGrabEvent

Returns​

void

Inherited from​

Script.onObjectGrabStart

onObjectSelectEnd()​

onObjectSelectEnd(_event): boolean

Defined in: src/core/Script.ts:150

Called when the controller stops selecting this object the script represents, e.g. View, ModelView.

Parameters​

_event​

SelectEvent

event.target holds its controller.

Returns​

boolean

Whether the event was handled

Inherited from​

Script.onObjectSelectEnd

onObjectSelectStart()​

onObjectSelectStart(_event): boolean

Defined in: src/core/Script.ts:141

Called when the controller starts selecting this object the script represents, e.g. View, ModelView.

Parameters​

_event​

SelectEvent

event.target holds its controller.

Returns​

boolean

Whether the event was handled

Inherited from​

Script.onObjectSelectStart

onObjectTouchEnd()​

onObjectTouchEnd(_event): void

Defined in: src/core/Script.ts:179

Called when a hand's index finger stops touching this object.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouchEnd

onObjectTouching()​

onObjectTouching(_event): void

Defined in: src/core/Script.ts:175

Called every frame that a hand's index finger is touching this object.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouching

onObjectTouchStart()​

onObjectTouchStart(_event): void

Defined in: src/core/Script.ts:171

Called when a hand's index finger starts touching this object.

Parameters​

_event​

ObjectTouchEvent

Returns​

void

Inherited from​

Script.onObjectTouchStart

onSelect()​

onSelect(_event): void

Defined in: src/core/Script.ts:98

Called whenever pinch / mouse click successfully completes, globally.

Parameters​

_event​

SelectEvent

event.target holds its controller.

Returns​

void

Inherited from​

Script.onSelect

onSelectEnd()​

onSelectEnd(_event): void

Defined in: src/core/Script.ts:92

Called whenever pinch / mouse click discontinues, globally.

Parameters​

_event​

SelectEvent

event.target holds its controller

Returns​

void

Inherited from​

Script.onSelectEnd

onSelecting()​

onSelecting(_event): void

Defined in: src/core/Script.ts:103

Called whenever pinch / mouse click is happening, globally.

Parameters​

_event​

SelectEvent

Returns​

void

Inherited from​

Script.onSelecting

onSelectStart()​

onSelectStart(_event): void

Defined in: src/core/Script.ts:86

Called whenever pinch / mouse click starts, globally.

Parameters​

_event​

SelectEvent

event.target holds its controller

Returns​

void

Inherited from​

Script.onSelectStart

onSimulatorStarted()​

onSimulatorStarted(): void

Defined in: src/core/Script.ts:78

Returns​

void

Inherited from​

Script.onSimulatorStarted

onSqueeze()​

onSqueeze(_event): void

Defined in: src/core/Script.ts:132

Called whenever gamepad trigger successfully completes, globally.

Parameters​

_event​

SelectEvent

event.target holds its controller.

Returns​

void

Inherited from​

Script.onSqueeze

onSqueezeEnd()​

onSqueezeEnd(_event): void

Defined in: src/core/Script.ts:121

Called whenever gamepad trigger stops, globally.

Parameters​

_event​

SelectEvent

event.target holds its controller.

Returns​

void

Inherited from​

Script.onSqueezeEnd

onSqueezeStart()​

onSqueezeStart(_event): void

Defined in: src/core/Script.ts:116

Called whenever gamepad trigger starts, globally.

Parameters​

_event​

SelectEvent

event.target holds its controller.

Returns​

void

Inherited from​

Script.onSqueezeStart

onSqueezing()​

onSqueezing(_event): void

Defined in: src/core/Script.ts:126

Called whenever gamepad is being triggered, globally.

Parameters​

_event​

SelectEvent

Returns​

void

Inherited from​

Script.onSqueezing

onXRSessionEnded()​

onXRSessionEnded(): void

Defined in: src/core/Script.ts:76

Returns​

void

Inherited from​

Script.onXRSessionEnded

onXRSessionStarted()​

onXRSessionStarted(_session?): void

Defined in: src/core/Script.ts:75

Parameters​

_session?​

XRSession

Returns​

void

Inherited from​

Script.onXRSessionStarted

physicsStep()​

physicsStep(): void

Defined in: src/core/Script.ts:73

Returns​

void

Inherited from​

Script.physicsStep

query()​

query(input, tools?): Promise<null | string | GeminiResponse>

Defined in: src/ai/AI.ts:175

Parameters​

input​
prompt​

string

tools?​

never[]

Returns​

Promise<null | string | GeminiResponse>

resolveApiKey()​

resolveApiKey(modelOptions): Promise<null | string>

Defined in: src/ai/AI.ts:122

Parameters​

modelOptions​

ModelOptions

Returns​

Promise<null | string>

sendRealtimeInput()​

sendRealtimeInput(input): false | void

Defined in: src/ai/AI.ts:225

Parameters​

input​

LiveSendRealtimeInputParameters

Returns​

false | void

sendToolResponse()​

sendToolResponse(response): void

Defined in: src/ai/AI.ts:219

Parameters​

response​

LiveSendToolResponseParameters

Returns​

void

setLiveCallbacks()​

setLiveCallbacks(callbacks): Promise<void>

Defined in: src/ai/AI.ts:213

Parameters​

callbacks​

LiveCallbacks

Returns​

Promise<void>

startLiveSession()​

startLiveSession(config): Promise<Session>

Defined in: src/ai/AI.ts:184

Parameters​

config​

Returns​

Promise<Session>

stopLiveSession()​

stopLiveSession(): Promise<void>

Defined in: src/ai/AI.ts:202

Returns​

Promise<void>

triggerKeyPopup()​

triggerKeyPopup(): void

Defined in: src/ai/AI.ts:246

In simulator mode, pop up a 2D UI to request Gemini key; In XR mode, show a 3D UI to instruct users to get an API key.

Returns​

void

update()​

update(_time?, _frame?): void

Defined in: src/core/Script.ts:67

Runs per frame.

Parameters​

_time?​

number

_frame?​

XRFrame

Returns​

void

Inherited from​

Script.update

createSampleKeysStructure()​

static createSampleKeysStructure(): object

Defined in: src/ai/AI.ts:259

Create a sample keys.json file structure for reference

Returns​

object

Sample keys.json structure

gemini​

gemini: object

gemini.apiKey​

apiKey: string = 'YOUR_GEMINI_API_KEY_HERE'

openai​

openai: object

openai.apiKey​

apiKey: string = 'YOUR_OPENAI_API_KEY_HERE'