Skip to main content

Generative AI

xb.ai provides bounded Gemini and OpenAI queries plus Gemini multipart input, image generation, and Live audio/video sessions. AI availability, credentials, network responses, and media streams are runtime state; expose waiting, unavailable, empty, disconnected, and failed states in the application.

Provider setup​

Gemini is the default provider:

const options = new xb.Options();
options.enableAI();
options.ai.model = 'gemini';
options.ai.gemini.model = 'gemini-3.7-flash';
options.ai.promptForApiKey = true; // optional local prototype prompt

await xb.init(options);

enableAI() enables Gemini. For OpenAI, select and enable it explicitly:

const options = new xb.Options();
options.ai.enabled = true;
options.ai.model = 'openai';
options.ai.openai.enabled = true;
options.ai.openai.model = 'gpt-4.1';

Configure the provider before initialization and check xb.ai.isAvailable() at use time. Availability confirms local wrapper state; each request can still fail because of credentials, quota, network, or provider errors.

Credential safety​

danger

Never ship long-lived provider keys in production browser code.

Provider options, ?key=, provider-specific URL parameters, the optional current-page prompt, and keys.json are local-prototype mechanisms. Production applications use a server-controlled proxy. Gemini Live clients should use short-lived credentials such as ephemeral tokens.

The browser prompt stores a key only for the current page session. Encoding a key or putting it in a URL does not protect it.

Bounded queries​

Use {prompt} for provider-neutral text:

async function askQuestion() {
  if (!xb.ai.isAvailable()) {
    showAIState('unavailable');
    return;
  }

  showAIState('waiting');
  try {
    const response = await xb.ai.query({
      prompt: 'Write a short description of this XR scene.',
    });
    const text =
      response && typeof response === 'object' ? response.text : response;
    showAnswer(text || 'No answer returned');
  } catch (error) {
    showAIState('failed');
  }
}

Gemini also accepts current typed multipart input:

const response = await xb.ai.query({
  type: 'multiPart',
  parts: [
    {inlineData: {data: base64Png, mimeType: 'image/png'}},
    {text: 'Name the objects relevant to this task.'},
  ],
});

Treat null, missing text, and tool-only responses as expected branches. Prevent repeated requests while one is active unless the application explicitly supports concurrency.

Image generation​

const dataUrl = await xb.ai.generate(
  'a futuristic low-poly treehouse in 3D',
  'image'
);

if (dataUrl?.startsWith('data:image/')) image.src = dataUrl;

A successful Gemini result is a data URL. It is not an object with a url field. Validate the result before loading it as a texture or UI image.

Gemini Live​

Register callbacks before connecting and use onopen as the ready transition:

await xb.ai.setLiveCallbacks({
  onopen: () => showLiveState('listening'),
  onmessage: (message) => handleLiveMessage(message),
  onerror: () => stopLive('failed'),
  onclose: () => stopLive('disconnected'),
});

if (!xb.ai.isLiveAvailable()) {
  showLiveState('unavailable');
  return;
}

showLiveState('connecting');
await xb.ai.startLiveSession({
  responseModalities: ['AUDIO'],
  speechConfig: {
    voiceConfig: {prebuiltVoiceConfig: {voiceName: 'Aoede'}},
  },
  inputAudioTranscription: {},
  outputAudioTranscription: {},
});

Use current nested realtime input fields:

xb.ai.sendRealtimeInput({
  audio: {data: base64Pcm, mimeType: 'audio/pcm;rate=48000'},
});

xb.ai.sendRealtimeInput({
  video: {data: base64Jpeg, mimeType: 'image/jpeg'},
});

The flat {data, mimeType} shape is not the current Live input contract.

startLiveSession() can return a provider session before the connection emits onopen. Keep the UI in connecting until onopen. Route remote close, local stop, error, and dispose() through one idempotent cleanup method that stops:

  • xb.ai.stopLiveSession();
  • application microphone capture and owned media tracks;
  • model-audio playback;
  • screenshot or camera frame timers;
  • pending UI state and the duplicate-start lock.

The GeminiManager addon at xrblocks/addons/ai/GeminiManager.js owns the common microphone, audio, camera/screenshot, tool, timer, and cleanup loop. Use it when that ownership matches the application. Use the lower-level facade for a materially different loop.

Tools and grounded actions​

An xb.Tool exposes a narrow model-requested action. Model arguments are untrusted input:

const setColor = new xb.Tool({
  name: 'set_object_color',
  description: 'Set one allowed scene object to an approved hex color.',
  parameters: {
    type: 'OBJECT',
    properties: {
      objectId: {type: 'STRING'},
      color: {type: 'STRING'},
    },
    required: ['objectId', 'color'],
  },
  onTriggered: async ({objectId, color}) => {
    // Resolve objectId through an allowlist and validate color.
    return {objectId, color};
  },
});

Validate schema, ranges, target identity, and authorization again at execution. Return structured failures for invalid arguments, missing targets, unknown tools, denied actions, and execution errors. Keep consequential actions behind visible confirmation.

Choose the smallest grounding input that answers the question: known app state, semantic scene context, a Set-of-Mark image, a rendered screenshot, or a device camera frame. Context collection and provider transmission are separate decisions. Disclose and send only what the behavior requires.

Executable evidence​