Real-time speech translation, streamed into a live browser session

Built on Webfuse, Surfly's co-browsing platform. It puts a live, multilingual caption overlay inside a shared browser session: everyone speaks their own language, everyone sees a single running feed translated into the language they picked. We use it internally during prospect calls when the room doesn't share a language, and it ships to customers who need real-time translation inside their own co-browsing sessions.

The interesting parts aren't "call a translation API." They're the three that make it feel live and multiplayer: streaming transcription, a partial-vs-final split, and using the co-browsing channel itself to make one overlay appear for everyone.

How the pieces fit

The extension runs across three contexts, each with one job:

• The popup is the engine. It captures the mic, runs streaming transcription, translates finished utterances, owns the language picker, and is a detached, draggable, minimisable floating widget.
• The background is the hub. It holds the shared caption store, broadcasts your captions to the other participants, merges theirs back in, and fans everything out to the overlay.
• The content script is the overlay. It injects a closed Shadow-DOM widget into the page and renders the caption cards.

That separation is what keeps each piece small. The engine never touches the DOM; the overlay never touches the network.

Streaming, not batch

The captions appear as you speak, not after you stop. That rules out the usual "record a clip, POST it, wait" approach. Instead the mic is streamed to OpenAI's Realtime transcription endpoint over a WebSocket.

Capturing raw audio means an AudioWorklet. I inline it as a Blob URL so there's no extra file to ship in the extension, downsample to 24 kHz mono to match the API's PCM16 format, and stream every frame:

// Inline AudioWorklet - no separate file to bundle
const WORKLET_CODE = `
  class PcmProcessor extends AudioWorkletProcessor {
    process(inputs) {
      const input = inputs[0]?.[0];
      if (input) this.port.postMessage(input);
      return true;
    }
  }
  registerProcessor("pcm-processor", PcmProcessor);
`;

// 24 kHz mono to match OpenAI's expected PCM16 format
this.audioContext = new AudioContext({ sampleRate: 24000 });
const blob = new Blob([WORKLET_CODE], { type: "application/javascript" });
await this.audioContext.audioWorklet.addModule(URL.createObjectURL(blob));

this.workletNode.port.onmessage = (e) => {
  if (this.ws?.readyState !== WebSocket.OPEN) return;
  const pcm16 = float32ToInt16(e.data as Float32Array);
  this.ws.send(JSON.stringify({
    type: "input_audio_buffer.append",
    audio: arrayBufferToBase64(pcm16.buffer),
  }));
};

The other half is letting the server decide where one utterance ends and the next begins. I hand turn detection to server-side voice-activity detection, tuned so it cuts on a natural pause rather than mid-sentence:

this.ws.send(JSON.stringify({
  type: "session.update",
  session: {
    type: "transcription",
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 24000 },
        transcription: { model: "gpt-4o-transcribe" },
        turn_detection: {
          type: "server_vad",
          threshold: 0.5,
          silence_duration_ms: 500,
          prefix_padding_ms: 300,
        },
      },
    },
  },
}));

Transcripts then arrive as a stream of deltas followed by a completion. One detail that earned its place: the API has shipped deltas as incremental chunks in some revisions and cumulative strings in others. Rather than trust one shape, the accumulator handles both:

if (data.type   === "conversation.item.input_audio_transcription.delta") {
  const itemId = data.item_id;
  const chunk = data.delta || "";
  // If the delta already includes the prior text, take it as-is;
  // otherwise append. Survives both API delta formats.
  const prior = this.partials.get(itemId) || "";
  const next = chunk.startsWith(prior) ? chunk : prior + chunk;
  this.partials.set(itemId, next);
  this.onDelta({ itemId, partial: next });
}

Partials feel instant; translation waits

A live transcript is a stream of partial guesses that sharpen as you keep talking. If I translated every partial, I'd hammer the translation API dozens of times per sentence and watch the text flicker as each guess got re-translated.

So the two signals are split. Partials surface the source text immediately - that's what makes the row appear in real time, with a blinking cursor and a "Speaking" pill. Translation is deferred until the utterance is final:

new RealtimeTranscriber(
  // FINAL: translate once the utterance is complete, then broadcast it
  async ({ transcript, itemId }) => {
    const translated = await translateText(transcript, currentTarget.name);
    browser.runtime.sendMessage({
      type: "lt-caption-out", itemId, original: transcript, translated, /* ... */
    });
  },
  // PARTIAL: surface the source text instantly. No translation yet -
  // translating every delta would spam the API and flicker the UI.
  ({ itemId, partial }) => {
    browser.runtime.sendMessage({
      type: "lt-partial-out", itemId, original: partial, /* ... */
    });
  },
);

The glue is a stable per-utterance itemId - the same id rides every partial and the final. The overlay uses it to update the in-progress card in place instead of appending new ones, so a row sharpens and then settles rather than stuttering down the feed.

The multiplayer trick

This is the part I'm proudest of, because it leans on what Webfuse already is.

Webfuse streams a single proxied page to every participant. The content script injects the caption overlay into that page - so even though one participant's browser renders it, the overlay is part of the shared session and everyone sees it for free. No second channel, no per-client rendering. From the source:

The content script renders the overlay... but the DOM it injects is part of the proxied page and is streamed to all participants - so the overlay is visible to everyone.

The captions themselves are shared over the co-browsing data channel. Each participant transcribes their own mic locally and broadcasts the result; everyone merges the broadcasts into one ordered feed, keyed by that same itemId:

// Broadcast my caption to the whole session
browser.webfuseSession.broadcastMessage(JSON.stringify({
  type: "live-translator",
  kind: "partial",            // or "final"
  itemId,                     // stable per-utterance id
  participantName: selfName,
  original: message.original,
  targetName, targetShort, targetColor,
}));

// ...and merge everyone else's into the shared store
browser.webfuseSession.on.addListener((payload) => {
  // Webfuse payloads arrive as a bare string OR { message }. Handle both.
  const raw = typeof payload === "string" ? payload : payload?.message;
  const msg = JSON.parse(raw);
  if (msg?.type !== "live-translator") return;
  pushCaption({
    itemId: msg.itemId,
    participantName: msg.participantName,
    partial: msg.kind === "partial",
    /* ... */
  });
});

Because every entry carries its own language metadata, each participant translates into the language they chose - your captions land in your language, mine in mine, all in the same feed, each tagged with the speaker and a per-language colour.

No keys in the client

The extension never holds the OpenAI key. A small backend mints a short-lived ephemeral key, and the realtime socket authenticates with that instead - so nothing long-lived ever reaches the browser, and translation is proxied the same way:

const res = await fetch(`${env.BACKEND_URL}/api/openai/ephemeral-key`, {
  method: "POST",
  headers: { Authorization: `Bearer ${env.API_TOKEN}` },
  body: JSON.stringify({ session: { type: "transcription", /* ... */ } }),
});
const data = await res.json();
await transcriber.connect(data.client_secret.value);

The details that make it feel finished

• Closed Shadow DOM. The overlay lives in a closed shadow root at max z-index, so the host page's styles can't leak in and the overlay can't leak out - it looks identical on any site.
• In-place updates. Partial to final mutates the existing card's text nodes rather than swapping the element, which keeps the typing cursor and entry animation smooth.
• A detached, persistent popup that survives navigation, auto-fits its chrome to its content with a ResizeObserver, and drags via screenX/Y because it lives in an iframe where clientX/Y would be wrong.
• Defensive everywhere it talks to something it doesn't own - the two API delta formats, the two Webfuse payload shapes. The fun stops being fun in production when an upstream quietly changes shape, so I assume both.

Why it matters

It turns a language barrier on a live call into a non-event. Internally, a multilingual prospect call just works - everyone reads along in their own language while the conversation flows. Externally, it's a capability customers can drop into their own co-browsing sessions.

The thing I'd underline: the leverage came from treating the platform as part of the architecture. The hard problem - "how do I get one synchronised, multilingual overlay in front of N people" - mostly dissolved once I used Webfuse's proxied DOM and data channel as the transport, instead of building a parallel one beside it.