Streaming Transcription in Production: Audio, Tokens, and Voice-Agent Boundaries

Streaming transcription looks simple in a demo: open a microphone, send bytes, and print words. Production begins where that example stops.

A browser may capture 48 kHz float audio while the model expects 24 kHz signed PCM. A network interruption can duplicate a phrase. An interim transcript can change after an agent has already acted on it. A long-lived API key can leak into a client bundle. A perfectly accurate final transcript can still feel unusable when the first words take three seconds to appear.

Vercel added streaming transcription to AI Gateway on July 22, 2026. The beta API in AI SDK accepts live audio and emits transcript updates as a model produces them. That closes an important gap between file transcription and realtime voice, but it does not remove the engineering boundaries around capture, transport, authentication, state, consent, and action safety.

The reliable design is a pipeline, not a model call: capture and normalize audio, authenticate with short-lived credentials, stream with backpressure, separate provisional from committed text, and let agents act only on a final, policy-checked command.

This guide explains that pipeline and gives a staged route from a server-side proof of concept to an observable production service. Product details were checked on July 23, 2026; streaming transcription remains beta, so pin versions and recheck the linked documentation before release.

Choose the right voice architecture first

Three architectures are often grouped under “voice AI,” but they solve different jobs.

Architecture	Input and output	Best fit	Main trade-off
File transcription	Complete audio file → complete text	Uploaded calls, meetings, voice notes, compliance archives	Simple and reproducible, but no live feedback
Streaming transcription	Live audio chunks → interim and final text	Captions, dictation, voice search, text-agent input	Low perceived latency, but requires session and audio-state engineering
Realtime voice	Continuous audio ↔ continuous multimodal session	Natural interruption, turn-taking, spoken assistants	Lowest conversational latency, but tighter coupling and more complex safety

Streaming transcription is a good fit when the existing product already accepts text. It changes the input adapter, not the agent’s reasoning or tool layer. Add speech generation if spoken output is useful. Move to a full realtime voice session only when interruption, barge-in, acoustic turn detection, and rapid back-and-forth are core requirements.

This distinction prevents an expensive mistake: rebuilding a proven text agent around a realtime model when the product only needed a microphone button.

What the beta API actually provides

Vercel’s launch example streams raw PCM to openai/gpt-realtime-whisper through AI Gateway:

import { experimental_streamTranscribe as streamTranscribe } from "ai";

const result = streamTranscribe({
  model: "openai/gpt-realtime-whisper",
  audio: audioStream, // ReadableStream<Uint8Array | string>
  inputAudioFormat: {
    type: "audio/pcm",
    rate: 24_000,
  },
});

for await (const part of result.fullStream) {
  if (part.type === "transcript-delta") {
    process.stdout.write(part.delta);
  }
}

The stream carries partial and final transcription events, while result.text resolves to the completed text. The same AI SDK interface can address other streaming-capable transcription models, such as xai/grok-stt, by changing the model. That portability is useful, but it is not behavioral equivalence. Providers can differ in supported formats, language quality, endpointing, event semantics, session limits, retention, and price.

The experimental_ prefix is an operational warning. Keep the SDK and gateway package versions pinned, place the provider behind an internal adapter, and include a contract test for event types and finalization. A beta API should not leak through every application component.

export type TranscriptEvent =
  | { type: "partial"; text: string; sequence: number }
  | { type: "final"; text: string; sequence: number }
  | { type: "error"; code: string; retryable: boolean };

export interface LiveTranscriber {
  start(input: ReadableStream<Uint8Array>): AsyncIterable<TranscriptEvent>;
  close(): Promise<void>;
}

This small boundary lets the application survive an SDK rename, provider migration, or move from server-proxied to direct client streaming.

The reference data path

Treat audio, transcript state, and agent actions as three separate trust zones.

 microphone
    │
    ▼
 capture → resample → mono → PCM16 → chunk queue
    │                                  │
    │                         backpressure / retry
    ▼                                  ▼
 browser or mobile client ─────── streaming endpoint
    │                                  │
    │                     ephemeral credential or server proxy
    │                                  ▼
    │                         transcription provider
    │                                  │
    ◀──── partial / final events ──────┘
    │
    ├─ partial text → temporary caption only
    └─ final text   → validation → user confirmation → text agent
                                                   │
                                                   ▼
                                             governed tools

The client owns capture and immediate presentation. A token broker owns short-lived access. The transcription service owns speech recognition. The existing text-agent boundary owns intent, permissions, approvals, and tools. Do not give the transcription session direct access to purchase, delete, deploy, or message actions.

For most web products, there are two viable transports:

Server proxy: the client sends audio to an application endpoint, which calls AI Gateway. This centralizes credentials and policy and simplifies provider changes, but the application pays an extra network hop and must operate long-lived connections.
Direct provider connection with an ephemeral token: the server authenticates the user and mints a narrow, short-lived credential; the client then streams directly. This reduces relay load and latency, but browser-visible tokens, connection policy, and provider behavior need careful control.

Start with a server proxy when traffic is modest or policy is strict. Use direct streaming only after the token broker, revocation window, origin controls, and abuse limits have been designed.

Audio correctness comes before model quality

The launch example expects raw PCM with this contract:

sample rate: 24,000 samples/second
channels:    1 (mono)
sample:      signed 16-bit little-endian
container:   none (raw PCM)

Renaming a WebM, Opus, AAC, or WAV file to .pcm does not convert it. A container has headers; a codec compresses samples; raw PCM has neither. The receiver must know the exact rate, channel count, sample representation, and byte order.

Browser capture is especially deceptive. getUserMedia() gives access to a stream, but the actual audio context may run at the hardware rate, commonly 44.1 or 48 kHz. MediaRecorder typically returns a compressed container. A robust client therefore needs one of three paths:

capture PCM frames in an AudioWorklet, resample them, and encode PCM16;
send a supported compressed stream when the selected model accepts it; or
relay the browser format to a server-side decoder and resampler.

An AudioWorklet keeps processing off the main UI thread. Its normalization step is conceptually:

function floatToPcm16(samples: Float32Array): Uint8Array {
  const bytes = new Uint8Array(samples.length * 2);
  const view = new DataView(bytes.buffer);

  for (let i = 0; i < samples.length; i += 1) {
    const clamped = Math.max(-1, Math.min(1, samples[i]));
    const pcm = clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff;
    view.setInt16(i * 2, Math.round(pcm), true);
  }

  return bytes;
}

This function encodes float samples; it does not resample them. A 48 kHz input still needs a real low-pass resampler before every other sample can become a valid 24 kHz stream. Naively dropping samples creates aliasing and damages recognition, especially around consonants and noisy speech.

Validate the pipeline with known fixtures before involving a microphone:

a generated tone confirms duration, rate, and byte order;
a fixed clean-speech clip provides a reproducible transcript;
noisy, accented, code-switched, and far-field clips expose real product behavior;
silence and clipping fixtures verify voice activity and error handling.

Record the input contract in each trace. “The model was inaccurate” is often “the client sent stereo 48 kHz bytes while declaring mono 24 kHz.”

Chunking, backpressure, and reconnects

Smaller chunks can reduce time to the first transcript, but they increase framing overhead and scheduling pressure. Large chunks are efficient but make captions bursty. Do not choose a chunk size from a blog post; measure it across target devices and networks.

The client queue needs a hard bound. If encoding produces bytes faster than the connection can send them, unlimited buffering increases memory and turns a short outage into minutes of stale audio. A production policy should define:

maximum queued audio duration;
whether to pause capture, drop oldest audio, or fail the session;
a warning threshold shown to the user;
maximum session duration;
idle timeout and silence behavior.

Audio sessions also need sequence numbers and timestamps. A reconnect cannot safely “retry the request” without knowing which audio was accepted. At-least-once replay may duplicate text; at-most-once delivery may lose speech. A practical first version ends the current segment on disconnect, preserves its final committed text, starts a new segment, and tells the user that a short portion may need repeating. Seamless replay can come later if the provider exposes acknowledgements.

Use an explicit state machine rather than scattered booleans:

idle → requesting_permission → connecting → streaming
                                           │   │
                                           │   ├→ finalizing → completed
                                           │   └→ reconnecting → streaming
                                           └────→ failed

Every transition should have a timeout, a user-visible status, and one owner. Closing the tab, revoking microphone permission, switching input devices, locking a phone, or losing the network must all terminate or recover the session predictably.

Never ship a long-lived gateway key to the client

A browser bundle, mobile application, or desktop renderer is not a secret store. If the client connects directly, mint a short-lived credential from an authenticated server endpoint. Vercel’s Grok STT pattern uses experimental_transcription.getToken:

// Runs only on the trusted server.
import { gateway } from "@ai-sdk/gateway";

const modelId = "xai/grok-stt";

export async function POST(request: Request) {
  const user = await requireAuthenticatedUser(request);
  await enforceVoiceQuota(user.id);

  const { token } = await gateway.experimental_transcription.getToken({
    model: modelId,
  });

  return Response.json({ modelId, token });
}

The constrained client uses only that token:

import { createGateway } from "@ai-sdk/gateway";
import { experimental_streamTranscribe as streamTranscribe } from "ai";

export function startTranscription(
  modelId: string,
  token: string,
  audio: ReadableStream<Uint8Array>,
) {
  const clientGateway = createGateway({ apiKey: token });

  return streamTranscribe({
    model: clientGateway.transcription(modelId),
    audio,
    inputAudioFormat: { type: "audio/pcm", rate: 24_000 },
  });
}

The token endpoint needs more than authentication. Bind issuance to a user, tenant, allowed model, purpose, origin, and quota where the platform supports it. Apply bot controls and per-user concurrency. Log issuance metadata but never the token. Keep lifetime short, and do not let a token minted for transcription become a general model credential.

If a provider’s ephemeral credential cannot express the required scope, proxy the session through the server. A slightly slower secure design is better than an unbounded credential in every browser.

Interim text is UI state, not a command

Speech recognizers revise hypotheses as later sound arrives. The UI should maintain at least committed segments and a provisional tail:

type TranscriptState = {
  committed: string[];
  interim: string;
  lastSequence: number;
};

function reduceTranscript(
  state: TranscriptState,
  event: TranscriptEvent,
): TranscriptState {
  if (event.sequence <= state.lastSequence) return state;

  if (event.type === "partial") {
    return { ...state, interim: event.text, lastSequence: event.sequence };
  }

  if (event.type === "final") {
    return {
      committed: [...state.committed, event.text],
      interim: "",
      lastSequence: event.sequence,
    };
  }

  return state;
}

Render interim text differently so users know it may change. Persist only final segments. When the user stops recording, wait for finalization rather than submitting the last partial buffer.

For agent input, add another boundary:

final transcript
    → empty/noise check
    → language and policy validation
    → editable confirmation for consequential intent
    → ordinary text-agent request
    → tool policy and approval

Voice makes confirmation more important, not less. Names, amounts, negation, addresses, and product variants are easy to mishear. Read-only search can run after finalization; purchases, deletion, external messages, and account changes should show the interpreted action and require explicit confirmation. The downstream tool still enforces authorization and idempotency.

Privacy is part of the data path

Audio contains more than words. It may reveal identity, health, background conversations, location clues, and bystanders. A transcript can also contain credentials or regulated data. Design collection and retention before launch.

Decide separately whether to retain:

Data	Default posture	Reason to retain
Raw audio	Do not retain	Explicitly consented quality review or a regulated record requirement
Interim transcript	Do not retain	Short-lived debugging in a tightly controlled environment
Final transcript	Retain only for the product purpose	Conversation history, user-authored note, or required evidence
Derived metrics	Prefer aggregated or minimized	Reliability, latency, and quality monitoring
Trace payload	Redact or sample	Incident investigation and calibrated evaluation

Show a clear recording state, stop capture immediately when the user stops it, and define behavior when the page is backgrounded. Document provider processing region, retention, training use, sub-processors, and deletion. A “zero data retention” routing option can reduce supplier retention, but it does not erase copies in application logs, analytics, error trackers, or evaluation datasets.

Keep audio access separate from agent memory. A spoken preference should not silently become a durable user profile. Memory writes need their own purpose, policy, consent, TTL, and deletion path.

Measure a latency budget, not one average

End-to-end latency is a sum of stages:

capture window
+ client encoding and queueing
+ network to transcription endpoint
+ provider buffering and inference
+ event delivery and UI rendering
+ finalization
+ optional agent response

Define at least these service indicators:

microphone-to-first-partial latency;
microphone-to-stable-text latency;
stop-to-final latency;
realtime factor, or processing time divided by audio duration;
disconnects and reconnects per session;
empty and truncated transcript rate;
user correction rate;
cost per completed audio minute and per accepted command.

Use p50, p95, and p99 by device, browser, region, connection type, model, and audio duration. A global average hides a mobile browser that fails one in ten sessions.

For voice search, time to first useful result matters more than time to a beautifully worded answer. The product can acknowledge a final query, begin deterministic retrieval, and stream results while the agent prepares explanation. Do not fill latency with verbose “working on it” narration.

Evaluate recognition in the product’s language

Word error rate is useful, but it treats every word similarly. A commerce voice command cares much more about “fifteen” versus “fifty,” “with nuts” versus “without nuts,” and the exact merchant or item name.

Build an evaluation set that reflects:

supported languages, accents, dialects, and code-switching;
product names, people, addresses, acronyms, and numbers;
quiet, traffic, television, restaurant, and office noise;
built-in microphones, headsets, Bluetooth, and speakerphone;
short commands, long dictation, corrections, and hesitation;
safety-critical negation and exclusion phrases.

Track conventional WER or character error rate alongside slot accuracy for amounts, names, dates, locations, and constraints. For an agent, run end-to-end tests that verify the final interpreted intent and proposed tool arguments. A transcription can have low WER and still choose the wrong cart quantity.

Compare models with the same capture pipeline and frozen fixtures. Then canary with real traffic under consent. Changing model, prompt, endpointing, SDK, resampler, or client codec should trigger regression tests.

Reliability and failure policy

A production voice feature needs an answer for each failure before it happens.

Failure	User experience	System behavior
Microphone denied	Explain how to enable it and keep text input available	Do not repeatedly reprompt
Unsupported capture path	Fall back to file upload or text	Record capability, not raw content
Slow network	Show connection degradation	Bound the queue; pause or terminate cleanly
Provider unavailable	Preserve committed text and offer retry/text	Circuit-break and fail over only to a contract-tested model
Session limit reached	Finalize current segment and explain	Start a new session only with user intent
Empty/noisy audio	Ask the user to repeat	Do not send an empty agent request
Agent command uncertain	Show editable transcript and proposed action	Require confirmation; never guess a consequential slot

Fallback is not just switching model IDs. A second model may require a different format or produce different event behavior. Maintain a tested capability registry:

{
  "model": "openai/gpt-realtime-whisper",
  "streaming": true,
  "input": { "encoding": "pcm_s16le", "rate": 24000, "channels": 1 },
  "languages": ["evaluated-product-language-set"],
  "max_session_seconds": 1800,
  "data_policy": "approved-policy-id",
  "adapter_version": "3"
}

Populate values from current provider contracts and internal tests rather than assumptions.

A four-stage implementation plan

Stage 1: prove the server contract

Stream a fixed 24 kHz PCM fixture from a server process. Verify partial events, final text, abort behavior, errors, and session limits. Pin packages and wrap the beta API in an adapter.

Exit when the same fixture produces an accepted transcript and all terminal states close resources.

Stage 2: add real capture

Implement one supported browser and microphone path. Confirm actual sample rate, resample correctly, encode PCM16, bound the queue, and render interim versus final state. Keep text input as a fallback.

Exit when device fixtures and real microphones meet the latency and slot-accuracy threshold.

Stage 3: secure and observe

Add the authenticated token broker or server proxy, quotas, concurrency limits, consent, retention controls, trace redaction, and latency dashboards. Test disconnect, backgrounding, token expiry, and provider failure.

Exit when abuse limits, deletion, and incident reconstruction have been exercised—not merely documented.

Stage 4: connect the agent safely

Send only finalized text through the normal text-agent path. Keep read and write tools separate. Show the transcript and proposed effect for consequential actions, require confirmation, and make writes idempotent. Add end-to-end evals for numbers, negation, names, and corrections.

Exit when a mis-transcription cannot directly create an irreversible effect.

Production checklist

The design rule that survives provider changes

AI Gateway makes a live transcription model easier to reach, but the lasting system is the set of boundaries around it.

Normalize audio before it crosses the model boundary. Use short-lived authority before it crosses the trust boundary. Treat partial text as display state before it crosses the application boundary. Require confirmation before interpreted speech crosses the action boundary. Observe each stage so a latency or accuracy problem has an owner.

That architecture works whether the selected model is Whisper, Grok STT, or a future provider. It also lets a company add voice to an existing text agent without giving a probabilistic transcript more authority than it deserves.

For model routing, cost, and provider controls, continue with Enterprise LLM Strategy. For governed tool execution, see Enterprise Agent Platforms.