Generative AI
xb.ai provides bounded Gemini and OpenAI queries plus Gemini multipart input,
image generation, and Live audio/video sessions. AI availability, credentials,
network responses, and media streams are runtime state; expose waiting,
unavailable, empty, disconnected, and failed states in the application.
Provider setup
Gemini is the default provider:
const options = new xb.Options();
options.enableAI();
options.ai.model = 'gemini';
options.ai.gemini.model = 'gemini-3.8-flash';
options.ai.promptForApiKey = true; // optional local prototype prompt
await xb.init(options);
enableAI() enables Gemini. For OpenAI, select and enable it explicitly:
const options = new xb.Options();
options.ai.enabled = true;
options.ai.model = 'openai';
options.ai.openai.enabled = true;
options.ai.openai.model = 'gpt-6-luna';
Configure the provider before initialization and check
xb.ai.isAvailable() at use time. Availability confirms local wrapper state;
each request can still fail because of credentials, quota, network, or provider
errors.
Credential safety
Never ship long-lived provider keys in production browser code.
Long-lived keys in provider options, ?key=, provider-specific URL parameters,
the optional current-page prompt, and keys.json are local-prototype
mechanisms. Production applications keep long-lived credentials server-side
and use a server-controlled proxy or a provider-supported short-lived token
flow. Gemini Live supports ephemeral tokens.
The browser prompt stores a key only for the current page session. Encoding a key or putting it in a URL does not protect it.
Gemini Live ephemeral tokens
For Gemini Live, production apps can keep the long-lived Gemini API key on their server and provide a short-lived ephemeral token to the browser. Configure an async token provider on Gemini options:
const options = new xb.Options();
options.enableAI();
options.ai.gemini.tokenProvider = async () => {
// Production: reverse-proxy this same-origin route to the token server.
const tokenEndpoint = '/api/gemini-live-token';
// Local dev instead: 'http://127.0.0.1:8787/api/gemini-live-token'
const response = await fetch(tokenEndpoint, {
method: 'POST',
credentials: 'include',
});
if (!response.ok) throw new Error('Could not get a Gemini Live token');
const {token} = await response.json();
return token; // AuthToken.name, e.g. "auth_tokens/..."
};
await xb.init(options);
The provider is called for each new Live session, including after the app stops
and starts a session again. It is not called to refresh an already-open
connection; handle session expiry by stopping and starting a new session. The
ephemeral token is used only for Live connections, requires the Gemini
Developer API's v1alpha Live endpoint, and does not replace apiKey for
Gemini queries or image generation. Keep an API key configured for those
operations, or use a server-side proxy for them.
The reference token server locks the ephemeral token to the configured Live
model with liveConnectConstraints.model and sets lockAdditionalFields: [].
That restricts the model without locking the other Live settings; the client
can still send its response modality, voice, and transcription configuration.
A local Node.js reference server (tools/gemini-token-server)
creates one-use tokens constrained to the SDK's default Live model and binds to
localhost by default. For local cross-origin requests, its ALLOWED_ORIGIN
defaults to http://127.0.0.1:8080 and supports preflight; configure it to your
exact app origin as needed. Before deploying it, authenticate and authorize
callers, apply rate limits, and keep the API key exclusively on the server.
Bounded queries
Use {prompt} for provider-neutral text:
async function askQuestion() {
if (!xb.ai.isAvailable()) {
showAIState('unavailable');
return;
}
showAIState('waiting');
try {
const response = await xb.ai.query({
prompt: 'Write a short description of this XR scene.',
});
const text =
response && typeof response === 'object' ? response.text : response;
showAnswer(text || 'No answer returned');
} catch (error) {
showAIState('failed');
}
}
Gemini also accepts current typed multipart input:
const response = await xb.ai.query({
type: 'multiPart',
parts: [
{inlineData: {data: base64Png, mimeType: 'image/png'}},
{text: 'Name the objects relevant to this task.'},
],
});
Treat null, missing text, and tool-only responses as expected branches.
Prevent repeated requests while one is active unless the application explicitly
supports concurrency.
Image generation
const dataUrl = await xb.ai.generate(
'a futuristic low-poly treehouse in 3D',
'image'
);
if (dataUrl?.startsWith('data:image/')) image.src = dataUrl;
A successful Gemini result is a data URL. It is not an object with a url
field. Validate the result before loading it as a texture or UI image.
Scene authoring
The optional Roomcraft add-on turns descriptions into manipulable 3D compositions, using trusted catalog assets or new procedural designs assembled from primitive parts. "Create a little robot" can produce one compound object without a robot catalog entry; follow-up requests can lengthen its arms or add a backpack through targeted part edits while preserving its native hand-edited transform. It validates scene plans instead of executing generated code. This is bounded procedural geometry, not a photorealistic text-to-mesh service.
Import Roomcraft from xrblocks/addons/roomcraft/index.js. Its request() method uses the configured AI facade, or an application-supplied planner callback for a server-side provider proxy. Curated layouts and part recipes work without AI through applyLayout() and applyPlan(). The public entry also exports ScenePart, ScenePartEdit, and SceneProceduralObject for typed consumers. Detected-surface placement, undo, model-loading ownership, privacy, and geometry budgets are documented with the add-on.
Gemini Live
Register callbacks before connecting and use onopen as the ready transition:
await xb.ai.setLiveCallbacks({
onopen: () => showLiveState('listening'),
onmessage: (message) => handleLiveMessage(message),
onerror: () => stopLive('failed'),
onclose: () => stopLive('disconnected'),
});
if (!xb.ai.isLiveAvailable()) {
showLiveState('unavailable');
return;
}
showLiveState('connecting');
await xb.ai.startLiveSession({
responseModalities: ['AUDIO'],
speechConfig: {
voiceConfig: {prebuiltVoiceConfig: {voiceName: 'Aoede'}},
},
inputAudioTranscription: {},
outputAudioTranscription: {},
});
Use current nested realtime input fields:
xb.ai.sendRealtimeInput({
audio: {data: base64Pcm, mimeType: 'audio/pcm;rate=48000'},
});
xb.ai.sendRealtimeInput({
video: {data: base64Jpeg, mimeType: 'image/jpeg'},
});
The flat {data, mimeType} shape is not the current Live input contract.
startLiveSession() can return a provider session before the connection emits
onopen. Keep the UI in connecting until onopen. Route remote close, local
stop, error, and dispose() through one idempotent cleanup method that stops:
xb.ai.stopLiveSession();- application microphone capture and owned media tracks;
- model-audio playback;
- screenshot or camera frame timers;
- pending UI state and the duplicate-start lock.
The GeminiManager addon at
xrblocks/addons/ai/GeminiManager.js owns the common microphone, audio,
camera/screenshot, tool, timer, and cleanup loop. Use it when that ownership
matches the application. Use the lower-level facade for a materially different
loop.
Choosing the view sent by startGeminiLive()
Choose the capture source explicitly when starting a GeminiManager session:
| Options | View sent to Gemini |
|---|---|
captureMode: 'camera' (default) | Device-camera view; in the simulator, the environment pass without virtual app objects. |
captureMode: 'screenshot', overlayOnCamera: false | Rendered virtual app content only. |
captureMode: 'screenshot', overlayOnCamera: true | Rendered virtual app content composited over the camera view. |
For an assistant that needs to see virtual objects in a simulated room, call this from the application's Start Conversation action after SDK initialization:
await manager.startGeminiLive({
captureMode: 'screenshot',
overlayOnCamera: true, // places the virtual objects over the camera capture
});
Microphone input during automated runs
startGeminiLive() enables microphone capture before opening the Live session.
It requires a usable audio input and browser permission.
Tools and grounded actions
An xb.Tool exposes a narrow model-requested action. Model arguments are
untrusted input:
const setColor = new xb.Tool({
name: 'set_object_color',
description: 'Set one allowed scene object to an approved hex color.',
parameters: {
type: 'OBJECT',
properties: {
objectId: {type: 'STRING'},
color: {type: 'STRING'},
},
required: ['objectId', 'color'],
},
onTriggered: async ({objectId, color}) => {
// Resolve objectId through an allowlist and validate color.
return {objectId, color};
},
});
Validate schema, ranges, target identity, and authorization again at execution. Return structured failures for invalid arguments, missing targets, unknown tools, denied actions, and execution errors. Keep consequential actions behind visible confirmation.
Choose the smallest grounding input that answers the question: known app state, semantic scene context, a Set-of-Mark image, a rendered screenshot, or a device camera frame. Context collection and provider transmission are separate decisions. Disclose and send only what the behavior requires.