Building a voice agent

A Syrinx voice agent is a VoiceAgentSession — the runtime — fed by a transport, with a pipeline (STT + TTS, or a realtime front) and a reasoner (your LLM or agent). This guide wires each piece.

Cascade: STT → reasoner → TTS

Per-slot config (API keys, model, voice) goes in the VoiceAgentSession constructor; the plugin instances are registered by slot — stt, the reasoner bridge, and tts.

Write it as a factory. A session is per-conversation, not per-process, so what you export is a function that builds a fresh one:

import { VoiceAgentSession } from '@kuralle-syrinx/core';
import { DeepgramSTTPlugin } from '@kuralle-syrinx/deepgram';
import { CartesiaTTSPlugin } from '@kuralle-syrinx/cartesia';
import { ReasoningBridge, fromStreamText } from '@kuralle-syrinx/aisdk';
import { createOpenAI } from '@ai-sdk/openai';

const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });

export function createVoiceAgent(): VoiceAgentSession {
  const session = new VoiceAgentSession({
    plugins: {
      stt: { api_key: process.env.DEEPGRAM_API_KEY!, model: 'nova-3', sample_rate: 16000, emit_eos_on_final: true },
      bridge: {},
      tts: { api_key: process.env.CARTESIA_API_KEY! },
    },
    endpointingOwner: 'provider_stt',
  });

  session.registerPlugin('stt', new DeepgramSTTPlugin());
  session.registerPlugin('bridge', new ReasoningBridge(fromStreamText({
    model: openai('gpt-4.1-mini'),
    system: 'You are a helpful voice assistant. Keep your replies short.',
  })));
  session.registerPlugin('tts', new CartesiaTTSPlugin());

  // Return it unstarted — the host calls start().
  return session;
}

What start() does, and who calls it

A freshly built session is inert. await session.start() brings it up, in this order: it wires the bus handlers and initializes the interaction policy, starts the bus drain loop, builds the init chain from every registered plugin, then runs that chain — the step that actually opens the provider sockets. Only then does the session reach Ready.

Two rules follow:

• Register every plugin before starting. The init chain is built from what is registered at that moment; anything added later is never initialized.
• Start exactly once. start() throws Cannot start session in state … unless the session is Uninitialized. The Node host and the Workers host both call it for you, which is why the factory above returns an unstarted session. Call it yourself only when you drive a session directly with no host — feeding it a WAV file, for instance.

Tip

This is a runnable example, not a sketch — hello-voice-agent.ts is this exact agent with a main() that feeds it a WAV fixture and prints the turn. Run it in one command: see Run it locally.

Each plugin only cares about the packets it consumes and produces:

• The STT plugin consumes stt.audio (the canonical audio ingress) and emits stt.interim / stt.result.
• The reasoner bridge consumes stt.result and eos.turn_complete, and emits llm.delta / llm.done (or a recoverable llm.error).
• The TTS plugin consumes tts.text and emits tts.audio.

Turn-taking — deciding when the user is done talking, and handling barge-in — is owned by the session’s interaction policy, not by the STT or TTS plugin.

Realtime: speech-to-speech

For the lowest-latency path, wrap a realtime adapter in a RealtimeBridge instead of a cascade:

import { RealtimeBridge, fromOpenAIRealtime } from '@kuralle-syrinx/realtime';
import { createNodeWsSocket } from '@kuralle-syrinx/ws/node';

const adapter = fromOpenAIRealtime({
  apiKey: process.env.OPENAI_API_KEY!,
  socketFactory: createNodeWsSocket,
});

session.registerPlugin('realtime', new RealtimeBridge(adapter));

Run the session with endpointingOwner: "timer" — the realtime model owns its own turn detection, so no STT/VAD/TTS plugins are registered. See Realtime providers for OpenAI, Gemini, and Grok.

Delegating to a reasoner from a realtime front

A realtime model is a great conversational surface but a shallow reasoner — it doesn’t run your tools or RAG well on its own. Pass a Reasoner as the bridge’s second argument and a tool name as its third, and the realtime front delegates to it as a tool call while staying the voice the user hears:

import { fromStreamText } from '@kuralle-syrinx/aisdk';

const reasoner = fromStreamText({ model, system, tools: { lookupOrder } });
const adapterWithTool = fromOpenAIRealtime({
  ...opts,
  tools: [{ name: 'ask_backend', description: '...', parameters: { /* JSON Schema */ } }],
});

session.registerPlugin('realtime', new RealtimeBridge(adapterWithTool, reasoner, 'ask_backend'));

The bridge feeds the reasoner’s answer back to the front model as a structured result so it repeats facts faithfully instead of paraphrasing, and the session emits tool_call_cue events (started / delayed / complete / failed) your client can use to show a “thinking” indicator while the reasoner runs.

Half-cascade: a realtime front with Syrinx TTS

If you want a realtime front’s reasoning but a specific TTS voice or language, run the front text-only and let a Syrinx TTS plugin speak the transcript — see the modalities: ["text"] option on realtime adapters in Realtime providers.

Adding tools to a cascade reasoner

Tools are just part of your reasoner backend’s config — the AI SDK and Mastra adapters pass them straight through:

import { tool } from 'ai';
import { z } from 'zod';

const lookupOrder = tool({
  description: 'Look up an order by id',
  parameters: z.object({ orderId: z.string() }),
  execute: async ({ orderId }) => ({ status: 'shipped' }),
});

const reasoner = fromStreamText({ model, system, tools: { lookupOrder } });

The bridge emits llm.tool_call / llm.tool_result on the bus as the reasoner invokes tools, so you can observe or log tool use without touching the reasoner itself.

Talking to it: a Node server

Everything above only builds a session. To actually talk to the agent you just wrote, serve it over the Syrinx WebSocket protocol and point a browser at it.

createVoiceWebSocketServer is the Node host. Give it a createSession factory — it calls that once per connection and starts the session for you:

import { createVoiceWebSocketServer } from '@kuralle-syrinx/server-websocket';
import { createVoiceAgent } from './my-agent.js';

const server = await createVoiceWebSocketServer({
  port: 4173,
  path: '/ws',
  createSession: () => createVoiceAgent(),
});

That is the whole server. It owns the audio wire: resampling, Opus on the downlink, frame pacing, heartbeats, and reconnect with ordered replay. Your factory only returns a session.

Serve the Studio UI from the same server

You still need something to speak into. @kuralle-syrinx/studio ships its built assets on npm, so one server can serve the UI and the agent on a single origin:

npm install @kuralle-syrinx/server-websocket @kuralle-syrinx/studio

import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
import { extname, join, normalize } from 'node:path';
import { studioDistPath } from '@kuralle-syrinx/studio/dist-path';
import { createVoiceWebSocketServer } from '@kuralle-syrinx/server-websocket';
import { createVoiceAgent } from './my-agent.js';

const TYPES: Record<string, string> = {
  '.html': 'text/html',
  '.js': 'text/javascript',
  '.css': 'text/css',
};

const http = createServer((request, response) => {
  const path = normalize((request.url ?? '/').split('?')[0] ?? '/');
  const file = join(studioDistPath, path === '/' ? 'index.html' : path);
  if (!file.startsWith(studioDistPath)) {
    response.writeHead(403).end('forbidden');
    return;
  }
  readFile(file).then(
    (body) => {
      response.writeHead(200, { 'content-type': TYPES[extname(file)] ?? 'application/octet-stream' });
      response.end(body);
    },
    () => response.writeHead(404).end('not found'),
  );
});

await createVoiceWebSocketServer({
  server: http,
  port: 4173,
  path: '/ws',
  createSession: () => createVoiceAgent(),
});

console.log('Studio + agent on http://127.0.0.1:4173');

Run it, open http://127.0.0.1:4173, allow the microphone, and talk to your agent. Studio defaults to ws://127.0.0.1:4173/ws, so on port 4173 there is nothing to configure.

The server owns turn detection, so you speak and stop naturally — no push-to-talk — and it yields when you interrupt it mid-sentence. The panel shows the live transcript, per-turn timings, and how each turn resolved.

Skipping the static-file step

If you would rather not serve the UI yourself, the hosted Studio can drive your local server directly — pass your endpoint as the ws query parameter:

https://syrinx-studio.mithushancj.workers.dev/?ws=ws://localhost:4173/ws

Use localhost, not 127.0.0.1. A page on a public HTTPS origin reaching a machine-local address is subject to the browser’s local-network access checks, and in testing the hostname form connected while the raw IP did not. Serving the UI from your own server avoids the question entirely, which is why the code above does it.

Prefer your own UI? Studio is built entirely on the browser client SDK:

import { SyrinxBrowserClient } from '@kuralle-syrinx/browser-client';

const client = new SyrinxBrowserClient({ url: 'ws://localhost:4173/ws' });

A few options matter once this leaves your laptop. resumeWindowMs (default 15 s) is how long a dropped connection can reattach to its live session. maxConcurrentSessions caps admission. sessionStore makes resume survive across processes. backgroundAudio mixes an ambient bed under assistant speech.

The endpoint is open by default

Every connection starts a session and spends provider credit, and a resumed connection attaches to a live one. Set authorize before you expose the port beyond loopback:

const server = await createVoiceWebSocketServer({
  port: 4173,
  createSession: () => createVoiceAgent(),
  authorize: (request) => isValidBearer(request.headers.authorization),
});

Return false or throw to reject the upgrade with a 4401.

On Cloudflare Workers

The same pipeline and reasoner run under withVoice(Agent, { pipeline, reasoner }) on the Workers edge — see Deploy on Cloudflare.