Skip to main content

Generative AI

xb.ai provides bounded Gemini and OpenAI queries plus Gemini multipart input, image generation, and Live audio/video sessions. AI availability, credentials, network responses, and media streams are runtime state; expose waiting, unavailable, empty, disconnected, and failed states in the application.

Provider setup​

Gemini is the default provider:

const options = new xb.Options();
options.enableAI();
options.ai.model = 'gemini';
options.ai.gemini.model = 'gemini-3.8-flash';
options.ai.promptForApiKey = true; // optional local prototype prompt

await xb.init(options);

enableAI() enables Gemini. For OpenAI, select and enable it explicitly:

const options = new xb.Options();
options.ai.enabled = true;
options.ai.model = 'openai';
options.ai.openai.enabled = true;
options.ai.openai.model = 'gpt-6-luna';

Configure the provider before initialization and check xb.ai.isAvailable() at use time. Availability confirms local wrapper state; each request can still fail because of credentials, quota, network, or provider errors.

Credential safety​

danger

Never ship long-lived provider keys in production browser code.

Long-lived keys in provider options, ?key=, provider-specific URL parameters, the optional current-page prompt, and keys.json are local-prototype mechanisms. Production applications keep long-lived credentials server-side and use a server-controlled proxy or a provider-supported short-lived token flow. Gemini Live supports ephemeral tokens.

The browser prompt stores a key only for the current page session. Encoding a key or putting it in a URL does not protect it.

Gemini Live ephemeral tokens​

For Gemini Live, production apps can keep the long-lived Gemini API key on their server and provide a short-lived ephemeral token to the browser. Configure an async token provider on Gemini options:

const options = new xb.Options();
options.enableAI();
options.ai.gemini.tokenProvider = async () => {
// Production: reverse-proxy this same-origin route to the token server.
const tokenEndpoint = '/api/gemini-live-token';
// Local dev instead: 'http://127.0.0.1:8787/api/gemini-live-token'
const response = await fetch(tokenEndpoint, {
method: 'POST',
credentials: 'include',
});
if (!response.ok) throw new Error('Could not get a Gemini Live token');
const {token} = await response.json();
return token; // AuthToken.name, e.g. "auth_tokens/..."
};

await xb.init(options);

The provider is called for each new Live session, including after the app stops and starts a session again. It is not called to refresh an already-open connection; handle session expiry by stopping and starting a new session. The ephemeral token is used only for Live connections, requires the Gemini Developer API's v1alpha Live endpoint, and does not replace apiKey for Gemini queries or image generation. Keep an API key configured for those operations, or use a server-side proxy for them.

The reference token server locks the ephemeral token to the configured Live model with liveConnectConstraints.model and sets lockAdditionalFields: []. That restricts the model without locking the other Live settings; the client can still send its response modality, voice, and transcription configuration.

A local Node.js reference server (tools/gemini-token-server) creates one-use tokens constrained to the SDK's default Live model and binds to localhost by default. For local cross-origin requests, its ALLOWED_ORIGIN defaults to http://127.0.0.1:8080 and supports preflight; configure it to your exact app origin as needed. Before deploying it, authenticate and authorize callers, apply rate limits, and keep the API key exclusively on the server.

Bounded queries​

Use {prompt} for provider-neutral text:

async function askQuestion() {
if (!xb.ai.isAvailable()) {
showAIState('unavailable');
return;
}

showAIState('waiting');
try {
const response = await xb.ai.query({
prompt: 'Write a short description of this XR scene.',
});
const text =
response && typeof response === 'object' ? response.text : response;
showAnswer(text || 'No answer returned');
} catch (error) {
showAIState('failed');
}
}

Gemini also accepts current typed multipart input:

const response = await xb.ai.query({
type: 'multiPart',
parts: [
{inlineData: {data: base64Png, mimeType: 'image/png'}},
{text: 'Name the objects relevant to this task.'},
],
});

Treat null, missing text, and tool-only responses as expected branches. Prevent repeated requests while one is active unless the application explicitly supports concurrency.

Image generation​

const dataUrl = await xb.ai.generate(
'a futuristic low-poly treehouse in 3D',
'image'
);

if (dataUrl?.startsWith('data:image/')) image.src = dataUrl;

A successful Gemini result is a data URL. It is not an object with a url field. Validate the result before loading it as a texture or UI image.

Scene authoring​

The optional Roomcraft add-on turns descriptions into manipulable 3D compositions, using trusted catalog assets or new procedural designs assembled from primitive parts. "Create a little robot" can produce one compound object without a robot catalog entry; follow-up requests can lengthen its arms or add a backpack through targeted part edits while preserving its native hand-edited transform. It validates scene plans instead of executing generated code. This is bounded procedural geometry, not a photorealistic text-to-mesh service.

Import Roomcraft from xrblocks/addons/roomcraft/index.js. Its request() method uses the configured AI facade, or an application-supplied planner callback for a server-side provider proxy. Curated layouts and part recipes work without AI through applyLayout() and applyPlan(). The public entry also exports ScenePart, ScenePartEdit, and SceneProceduralObject for typed consumers. Detected-surface placement, undo, model-loading ownership, privacy, and geometry budgets are documented with the add-on.

Gemini Live​

Register callbacks before connecting and use onopen as the ready transition:

await xb.ai.setLiveCallbacks({
onopen: () => showLiveState('listening'),
onmessage: (message) => handleLiveMessage(message),
onerror: () => stopLive('failed'),
onclose: () => stopLive('disconnected'),
});

if (!xb.ai.isLiveAvailable()) {
showLiveState('unavailable');
return;
}

showLiveState('connecting');
await xb.ai.startLiveSession({
responseModalities: ['AUDIO'],
speechConfig: {
voiceConfig: {prebuiltVoiceConfig: {voiceName: 'Aoede'}},
},
inputAudioTranscription: {},
outputAudioTranscription: {},
});

Use current nested realtime input fields:

xb.ai.sendRealtimeInput({
audio: {data: base64Pcm, mimeType: 'audio/pcm;rate=48000'},
});

xb.ai.sendRealtimeInput({
video: {data: base64Jpeg, mimeType: 'image/jpeg'},
});

The flat {data, mimeType} shape is not the current Live input contract.

startLiveSession() can return a provider session before the connection emits onopen. Keep the UI in connecting until onopen. Route remote close, local stop, error, and dispose() through one idempotent cleanup method that stops:

  • xb.ai.stopLiveSession();
  • application microphone capture and owned media tracks;
  • model-audio playback;
  • screenshot or camera frame timers;
  • pending UI state and the duplicate-start lock.

The GeminiManager addon at xrblocks/addons/ai/GeminiManager.js owns the common microphone, audio, camera/screenshot, tool, timer, and cleanup loop. Use it when that ownership matches the application. Use the lower-level facade for a materially different loop.

Choosing the view sent by startGeminiLive()​

Choose the capture source explicitly when starting a GeminiManager session:

OptionsView sent to Gemini
captureMode: 'camera' (default)Device-camera view; in the simulator, the environment pass without virtual app objects.
captureMode: 'screenshot', overlayOnCamera: falseRendered virtual app content only.
captureMode: 'screenshot', overlayOnCamera: trueRendered virtual app content composited over the camera view.

For an assistant that needs to see virtual objects in a simulated room, call this from the application's Start Conversation action after SDK initialization:

await manager.startGeminiLive({
captureMode: 'screenshot',
overlayOnCamera: true, // places the virtual objects over the camera capture
});

Microphone input during automated runs​

startGeminiLive() enables microphone capture before opening the Live session. It requires a usable audio input and browser permission.

Tools and grounded actions​

An xb.Tool exposes a narrow model-requested action. Model arguments are untrusted input:

const setColor = new xb.Tool({
name: 'set_object_color',
description: 'Set one allowed scene object to an approved hex color.',
parameters: {
type: 'OBJECT',
properties: {
objectId: {type: 'STRING'},
color: {type: 'STRING'},
},
required: ['objectId', 'color'],
},
onTriggered: async ({objectId, color}) => {
// Resolve objectId through an allowlist and validate color.
return {objectId, color};
},
});

Validate schema, ranges, target identity, and authorization again at execution. Return structured failures for invalid arguments, missing targets, unknown tools, denied actions, and execution errors. Keep consequential actions behind visible confirmation.

Choose the smallest grounding input that answers the question: known app state, semantic scene context, a Set-of-Mark image, a rendered screenshot, or a device camera frame. Context collection and provider transmission are separate decisions. Disclose and send only what the behavior requires.

Executable evidence​