Skip to main content

Spatial Audio & Speech

The xb.sound (or xb.core.sound) manager handles audio capture, playback, positional spatial audio, and Web Speech API wrappers.

Spatial (Positional) Audio​

Spatial audio pans, attenuates, and filters sounds based on the user's distance and orientation relative to an object in the virtual space.

To implement positional audio, attach a THREE.PositionalAudio node directly to the target object:

import * as THREE from 'three';
import * as xb from 'xrblocks';

class PositionalSoundScript extends xb.Script {
  init() {
    // 1. Get the Web Audio listener managed by the SDK
    const listener = xb.core.sound.getAudioListener();

    // 2. Create the PositionalAudio node
    this.sound = new THREE.PositionalAudio(listener);

    // 3. Configure audio parameters
    this.sound.setRefDistance(0.5); // Distance where volume starts reducing
    this.sound.setVolume(1.0);

    // 4. Attach the sound node to the parent object
    this.add(this.sound);
  }

  playAudio(audioBuffer) {
    // Play a preloaded decoded AudioBuffer
    this.sound.setBuffer(audioBuffer);
    this.sound.play();
  }
}

Microphone Recording & Playback​

XR Blocks supports direct microphone capture (e.g., for sending speech files to an AI model or storing voice memos).

Note: Microphone access prompts the user for permissions. Set options.permissions.microphone = true during startup.

// Start recording PCM audio from mic
await xb.core.sound.startRecording();

// Stop recording and retrieve buffer
const pcm = xb.core.sound.stopRecording(); // Returns ArrayBuffer of Int16 PCM
const sampleRate = xb.core.sound.getRecordingSampleRate();

// Play the captured PCM buffer
await xb.core.sound.playRecordedAudio(pcm, sampleRate);

Adjust the overall output levels:

xb.core.sound.setMasterVolume(0.8); // Set master volume between 0.0 and 1.0

Speech Recognition (Speech-to-Text)​

CoreSound creates the recognizer during initialization when options.sound.speechRecognizer.enabled is true. It is enabled by default. Configure it before xb.init() and listen for Three.js-style events:

class VoiceInput extends xb.Script {
  init() {
    const recognizer = xb.core.sound.speechRecognizer;
    if (!recognizer) return;

    recognizer.addEventListener('result', (event) => {
      console.log('Transcript:', event.transcript);
      console.log('Final:', event.isFinal);
    });
    recognizer.addEventListener('error', (event) => {
      console.error('Recognition error:', event.error);
    });

    recognizer.start();
  }
}

const options = new xb.Options();
options.permissions.microphone = true;
options.sound.speechRecognizer.lang = 'en-US';
options.sound.speechRecognizer.interimResults = true;
xb.add(new VoiceInput());
xb.init(options);

Speech Synthesis (Text-to-Speech)​

Enable the subsystem-managed synthesizer before initialization, then call its speak(text, lang, pitch, rate) method:

const options = new xb.Options();
options.sound.speechSynthesizer.enabled = true;
options.sound.speechSynthesizer.allowInterruptions = true;
xb.init(options);

await xb.core.sound.speechSynthesizer?.speak(
  'Hello! Welcome to the XR experience.',
  'en-US',
  1.0, // pitch
  1.0 // rate
);

// Stop speaking
xb.core.sound.speechSynthesizer?.cancel();

Speech recognition depends on the browser's Web Speech API and may be unavailable. Always guard the optional speechRecognizer and speechSynthesizer properties. For complete examples, see

demos/sound/main.js and src/sound/README.md.