> ## Documentation Index > Fetch the complete documentation index at: https://docs.glot.com/llms.txt > Use this file to discover all available pages before exploring further. # Streaming audio > How a client streams live audio into a Glot translation room and renders the transcript it gets back Glot translates a live conversation as it happens. Your client streams microphone audio into a **room**, a translator joins that room, and it publishes two things back: translated audio you play, and transcript frames you render as captions. This page describes the streaming model end to end — the lifecycle of a session, how audio flows, and the exact wire format of the transcript. It is the concept guide behind the **Translation Playground**, which is a working reference client for everything below. Audio is carried over WebRTC by [LiveKit](https://docs.livekit.io), so you connect with a LiveKit client SDK using the `url` and `token` the Glot API returns. You never talk to LiveKit's API directly — Glot provisions the room and mints the tokens. ## How a session works ```mermaid theme={null} sequenceDiagram participant C as Your client participant A as Glot API participant R as Room (WebRTC) participant T as Translator C->>A: POST /v1/rooms A-->>C: id, room, url C->>A: POST /v1/rooms/{room}/token (identity, languages) A-->>C: token, url C->>R: connect(url, token) C->>R: publish microphone Note over R,T: Room holds two languages → translator joins T-->>C: translated audio track T-->>C: transcript frames on glot.transcript C->>A: DELETE /v1/rooms/{id} ``` ### The moving parts A live translation session, and also a billing record. Created with `POST /v1/rooms`. A room has **no languages of its own** — you can create it before you know who will join or what they speak. The response carries the room `id` (used to look up usage and to close the room) and the `room` name (used to mint tokens). One connection to a room. Every participant, including the first, enters with a join token from `POST /v1/rooms/{room_name}/token`, which is where languages are chosen — one participant at a time. Tokens are short-lived and bound to the `identity` they were issued for; a participant is disconnected when their token expires. ISO 639-1 codes, from `GET /v1/supported-languages`. A participant declares the languages they speak and want to hear. Send two when a single connection covers both sides of the conversation — two colleagues sharing one phone, or a test client like the playground — and one when the speaker has a device to themselves. That count is also what sets the room's [streaming mode](#streaming-modes-mono-and-dual). A room translates between **at most two** languages; a token request that would introduce a third returns `409`. Joins the room on its own once the room holds two languages, and leaves when fewer than two remain. You never dispatch or dismiss it. It publishes an audio track and transcript frames. It bills at a higher per-minute rate than a participant, which is why it is not present in an idle room. ## Streaming modes: mono and dual Every room has a **streaming mode**, which says how its audio reaches the translator. It follows from one question: are both speakers on the same device, or does each have their own? The mode decides how many **lanes** the room's audio travels on — a lane being one uplink stream of mixed audio going to the translator. Two people, one phone or laptop. Their voices are already mixed by the device's mic, so the room has **one lane** carrying both languages. Each speaker joins from their own device, wherever they are. The room has **one lane per language**, so the two sides are never mixed together. | | `mono` | `dual` | | ------------------------ | ----------------------------------------------------------- | --------------------------------------------------- | | Use it when | Two colleagues share one phone, or a room shares one laptop | Each speaker is on their own device | | Connections | One, covering both languages | One per language | | Languages per join token | Two — `["en", "zh"]` | One — `["en"]` for one side, `["zh"]` for the other | | Lanes to the translator | One, carrying both voices | One per language, routed by the token's language | Both modes translate in both directions, publish audio the same way, and emit the same transcript frames. What differs is how the audio is carried — which is why the mode has to be consistent across a room's tokens. A `mono` lane is expected to contain both voices; a `dual` lane is expected to contain one language's speakers and nobody else. A `dual` room is not limited to two participants — only to two distinct languages. Three people, two speaking English and one speaking Chinese, is a valid `dual` room: the two English devices are mixed together into the English lane. ### Set the mode, or let Glot infer it Pass `streaming_mode` when you create the room. Use this when you already know the shape of the call. ```bash title="cURL" theme={null} curl -X POST https://api.staging.glot.com/v1/rooms \ -H "Authorization: Bearer $GLOT_API_KEY" \ -H "Content-Type: application/json" \ -d '{"streaming_mode": "dual"}' ``` The room comes back with the mode it recorded, and from then on a join token asking for the wrong number of languages is an error rather than a room that quietly means something else. ```json title="JSON" theme={null} { "id": "3f1c…", "room": "room-9a2b41c7d5e0", "url": "wss://…", "streaming_mode": "dual" } ``` Omit `streaming_mode` and the room starts with none. Its **first join token** then decides, by how many languages it asks for: * two languages on one token → `mono` * one language → `dual` ```bash title="cURL" theme={null} # This token makes the room dual for the rest of its life. curl -X POST https://api.staging.glot.com/v1/rooms/room-9a2b41c7d5e0/token \ -H "Authorization: Bearer $GLOT_API_KEY" \ -H "Content-Type: application/json" \ -d '{"identity": "alice", "languages": ["en"]}' ``` Use this when the client minting the first token is the one that knows how the call will be set up. The room's response carries `"streaming_mode": null` until that token lands. Either way, the mode is fixed for the room's lifetime — there is no call that changes it — and `GET /v1/rooms/{room_id}` reports whatever the room settled on. ### Every later token must match Once a room has a mode, a join token whose language count disagrees is refused with `409`. The message names the mode and the count it expects: ```text title="409 Conflict" theme={null} This room is in dual mode: each participant joins from their own device, so a join token must request exactly 1 language. ``` So a `dual` room takes exactly one language per token, and a `mono` room takes two on every connection. An inferred mode is fixed by the first token **even if nobody ever connects with it**. Mint a one-language token by mistake against a room with no mode, and that room is `dual` for good — your only remedy is a new room. Declaring `streaming_mode` at creation turns that mistake into a `409` on the token instead. ## Start a session Languages are not part of this call. Pass `room_name` to choose your own name — it must be unique among rooms that are currently live — or omit it and one is generated. Pass `streaming_mode` to pin the room to `mono` or `dual`, or omit it and let the first join token decide. ```bash title="cURL" theme={null} curl -X POST https://api.staging.glot.com/v1/rooms \ -H "Authorization: Bearer $GLOT_API_KEY" \ -H "Content-Type: application/json" \ -d '{}' ``` ```json title="JSON" theme={null} { "id": "3f1c…", "room": "room-9a2b41c7d5e0", "url": "wss://…", "streaming_mode": null } ``` The room needs credit: a request against an organization whose balance is at or below zero returns `402`. A call already in progress is never cut off, so a long room can run the balance negative. Use the `room` name from the previous response, not the name you asked for — when you omit `room_name` the API generates one. One token per connection. The example below carries both languages, which is the `mono` shape — a shared device. For a `dual` room, mint one token per speaker with a single language each. See [streaming modes](#streaming-modes-mono-and-dual). ```bash title="cURL" theme={null} curl -X POST https://api.staging.glot.com/v1/rooms/room-9a2b41c7d5e0/token \ -H "Authorization: Bearer $GLOT_API_KEY" \ -H "Content-Type: application/json" \ -d '{"identity": "alice", "languages": ["en", "zh"]}' ``` ```json title="JSON" theme={null} { "room": "room-9a2b41c7d5e0", "token": "eyJ…", "url": "wss://…", "languages": ["en", "zh"] } ``` A join token is a live credential for a translation session. Mint it on your server and hand only the token to the browser — never ship your API key to a client. Nothing is translated until audio arrives, and the translator only joins once the room holds two languages. ```ts title="TypeScript" theme={null} import { Room } from "livekit-client" const room = new Room() await room.connect(url, token) await room.localParticipant.setMicrophoneEnabled(true) ``` Addressed by the room's `id` — the persisted UUID — not the name tokens are minted against. ```bash title="cURL" theme={null} curl -X DELETE https://api.staging.glot.com/v1/rooms/3f1c… \ -H "Authorization: Bearer $GLOT_API_KEY" ``` Metering is per participant, and we make a best effort to stop it as each one disconnects — so leaving the room does not silently run up minutes. But the room itself stays open until you close it or it ends on its own, so close it explicitly rather than relying on either. Closing an already-ended room is a no-op that still returns `204`, so a retry on the way out is safe. The usage record survives — a room is a billing record and outlives the call it measured. ## Play the translated audio The translator publishes **one audio track per language** in the room, as ordinary remote tracks. In a `mono` room the shared device wants all of them — both speakers are listening to the same speakers, in both directions. So attach every remote audio track you get subscribed to, and detach it when it goes away. ```ts title="TypeScript" theme={null} import { RoomEvent, Track, type RemoteTrack } from "livekit-client" const container = document.querySelector("#glot-audio")! room .on(RoomEvent.TrackSubscribed, (track: RemoteTrack) => { if (track.kind !== Track.Kind.Audio) return container.appendChild(track.attach()) }) .on(RoomEvent.TrackUnsubscribed, (track: RemoteTrack) => { track.detach().forEach((el) => el.remove()) }) ``` In a `dual` room, each client should play only the translation **into its own language** — the language its token was minted with. Attaching every track there means a speaker also hears their own words coming back translated, which is what the other device is meant to play. Each translated track is named after the language it carries, so a client can pick the one it wants rather than attaching all of them. The elements can live in a hidden container — they are playback sinks, not UI. Browsers block autoplay until the user has interacted with the page, so connect from a click rather than on page load. ## Live transcription The translator publishes one JSON frame per text stream on the topic `glot.transcript`, over the same room connection. There is no separate transcription endpoint or socket to open. This is deliberately **not** LiveKit's own `lk.transcription` topic. That convention assumes the sender is the participant being transcribed, and here the sender is a translator relaying somebody else's speech — so LiveKit's built-in transcription events never see these frames. Register a handler for `glot.transcript` yourself. ### Two kinds of frame The model streams a translation before it confirms it, so an utterance arrives in two forms. Both carry a `segment` id, which is what ties them together. One chunk of the translation as it is produced, for liveness only. ```json title="JSON" theme={null} { "v": 1, "kind": "partial", "segment": "seg_18", "language": "zh", "text": "我很好, ", "last": false } ``` The chunk **verbatim** — a delta, not a running total. Accumulate deltas per `segment` for the typing effect. Marks the partial that ended the utterance. It means "this stopped growing", **not** "this is complete" — it exists so you can stop a cursor even when the `segment` frame never arrives. The complete utterance, in both languages, published once the model confirms it. ```json title="JSON" theme={null} { "v": 1, "kind": "segment", "segment": "seg_18", "language": "zh", "text": "我很好，谢谢。", "source_language": "en", "source_text": "I'm good, thanks." } ``` The authoritative translation. It **supersedes** every `partial` sharing this `segment` id, so replace your accumulation rather than extending it. ISO 639-1 code of the language the speech came from. Only a `segment` frame carries direction — a partial tells you the language it is arriving *in* and nothing about where it came from. What was actually spoken, in the source language. Frame-shape version, currently `1`. Bumped only for a breaking change. The translator and your client deploy independently, so **drop a frame whose version you don't recognize** rather than half-understand it. Groups the partials of one utterance with the `segment` frame that supersedes them. ISO 639-1 code of the language this text was translated **into**. ### Reading frames off the topic Register the handler **before** you connect, so no frame published in the first moments of the session arrives without one. ```ts title="TypeScript" theme={null} import { Room } from "livekit-client" const TRANSCRIPT_TOPIC = "glot.transcript" const room = new Room() let entries: TranscriptEntry[] = [] // Reads are chained, not run concurrently. A text-stream handler returns void, so the // SDK does not await it — two overlapping readAll() promises can settle out of order and // interleave one utterance's chunks into another's. Chaining costs nothing here: each // frame is one small, self-contained stream. let reads = Promise.resolve() room.registerTextStreamHandler(TRANSCRIPT_TOPIC, (reader) => { reads = reads .then(async () => { const frame = parseTranscriptFrame(await reader.readAll()) if (frame === null) return entries = applyTranscriptFrame(entries, frame) render(entries) }) .catch((err) => { // An unreadable frame costs one caption; a rejection left on the chain would // poison every read after it. console.error("Failed to read a transcript frame", err) }) }) await room.connect(url, token) ``` Registering a second handler for the same topic on the same `Room` throws. Build a fresh `Room` per session and register once. ### Parsing a frame Parsing runs on network input inside a stream handler, where a throw takes out your read chain rather than surfacing anywhere useful. Treat malformed JSON, a non-object, an unknown `kind`, and a future `v` all as "not a frame I can read". ```ts title="TypeScript" theme={null} const TRANSCRIPT_VERSION = 1 export function parseTranscriptFrame(payload: string): TranscriptFrame | null { let raw: unknown try { raw = JSON.parse(payload) } catch { return null } if (typeof raw !== "object" || raw === null) return null const { v, kind, segment, language, text } = raw as Record if (v !== TRANSCRIPT_VERSION) return null if (typeof segment !== "string" || typeof language !== "string" || typeof text !== "string") { return null } if (kind === "partial") { return { kind, segment, language, text, last: (raw as { last?: unknown }).last === true } } if (kind === "segment") { const { source_language: sourceLanguage, source_text: sourceText } = raw as Record // Both halves of the pair or nothing: half a pair renders as a caption that silently // drops the direction it claims to show. if (typeof sourceLanguage !== "string" || typeof sourceText !== "string") return null return { kind, segment, language, text, sourceLanguage, sourceText } } return null } ``` ### Folding frames into a transcript The two kinds compose differently, and it matters: a partial **extends** its utterance, a segment **replaces** it. Appending a segment's text would duplicate the whole utterance, because the segment carries what the partials already spelled out. ```ts title="TypeScript" theme={null} type TranscriptStatus = "streaming" | "stopped" | "final" type TranscriptEntry = { segment: string /** Language translated into. */ language: string /** Accumulated while streaming, replaced wholesale when confirmed. */ text: string /** Known only once the utterance is confirmed. */ sourceLanguage: string | null sourceText: string | null status: TranscriptStatus } export function applyTranscriptFrame( entries: TranscriptEntry[], frame: TranscriptFrame, ): TranscriptEntry[] { const index = entries.findIndex((entry) => entry.segment === frame.segment) if (frame.kind === "segment") { const confirmed: TranscriptEntry = { segment: frame.segment, language: frame.language, text: frame.text, sourceLanguage: frame.sourceLanguage, sourceText: frame.sourceText, status: "final", } // Appended when unseen rather than dropped: partials can be lost across a reconnect, // and the confirmed text is the half worth showing. if (index === -1) return [...entries, confirmed] return entries.map((entry, i) => (i === index ? confirmed : entry)) } if (index === -1) { return [ ...entries, { segment: frame.segment, language: frame.language, text: frame.text, sourceLanguage: null, sourceText: null, status: frame.last ? "stopped" : "streaming", }, ] } const current = entries[index] // A partial that arrives after its utterance was confirmed is stale — the frames raced, // or the translator re-sent one. Appending it would tack a fragment onto a finished caption. if (current.status === "final") return entries return entries.map((entry, i) => i === index ? { ...current, text: current.text + frame.text, status: frame.last ? "stopped" : "streaming" } : entry, ) } ``` The three statuses are genuinely different states, not one flag: | Status | Meaning | Reached by | | ----------- | ------------------------------------- | -------------------------------- | | `streaming` | Still growing. Show a cursor. | Any `partial` with `last: false` | | `stopped` | Stopped growing, but never confirmed. | A `partial` with `last: true` | | `final` | The authoritative text landed. | A `segment` frame | An utterance interrupted by a reconnect stops without ever being confirmed, so it stays `stopped` forever. A UI that treated `stopped` as done would claim text was confirmed when it never was. ## Rendering captions The playground's transcript panel is a good default, and each of its choices comes from a property of the stream: * **One row per utterance**, keyed by `segment` id — the translation as the headline, the speech it came from underneath once it settles. * **Label the direction per row**, as `EN → ZH`. Both directions land in one list, and with two Latin-script languages the text itself won't say which way round a line goes. Until the `segment` frame lands the source is unknown, so render `… → ZH` and fill the left side in. * **Dim text that isn't `final`** and show a cursor while it is `streaming`, so a live caption reads as provisional. * **Autoscroll only while the reader is at the live edge.** Somebody who has scrolled up to re-read something must not be dragged back down by the next utterance. * **Announce new rows with `role="log"` and `aria-live="polite"`**, and set `aria-busy` on rows that are still streaming so a screen reader can tell a growing caption from a settled one. ## Pitfalls The room holds fewer than two languages. Either a second participant has to join with a different language, or one connection must mint its token with two — `{"languages": ["en", "zh"]}`. A single-language room stays monolingual and untranslated. Three different causes, and the message says which: * The room name is already in use by a live room (on `POST /v1/rooms`). * The token's language **count** contradicts the room's [streaming mode](#streaming-modes-mono-and-dual) — one language in a `mono` room, or two in a `dual` one. * The token would introduce a third language. This check is against who is connected **right now**, so once a speaker leaves, their language frees up and a different one is admitted. Its first join token inferred it. A room with no declared mode takes the mode implied by that token's language count, whether or not anyone connects with it, and nothing changes it afterwards. Create a new room, and pass `streaming_mode` at creation so the mode comes from your intent rather than from a token. You are appending `segment` text to the accumulated partials. A segment replaces, it never extends. Your reads run concurrently. Chain them, as shown above. Clear the transcript explicitly when the token changes. A fresh token is a fresh session, and state that outlives the connection won't reset itself. Disconnecting participants stops their metering — best effort, as each one leaves — but it does not end the room. Call `DELETE /v1/rooms/{room_id}` when the call is over rather than leaving the room to end on its own. **Translation Playground**, in the Glot dashboard, runs this whole flow in the browser — create a room, connect a mic, and watch partial and segment frames land in the live transcript.