AI dubbing pipeline architecture: production lessons

How an AI dubbing pipeline works in production: four model stages, Persian pitfalls and the provider fallback that replaced silence after an outage.

An AI dubbing pipeline chains four model stages: transcription with speaker detection, a translation that fits the original timing, a distinct voice for each speaker, and timing alignment with the final mix. The models are the part you can rent. Whether dubs keep shipping depends on the platform around them: a gateway per AI provider with automatic fallback, durable queues, work that resumes after a crash, a price shown before the job runs and a refund when it fails. At Booq, where I am CTO, a speech provider's outage once produced dubs with silence where lines were missing. Speech now falls back through a chain of providers: a slightly different voice beats silence.

How an AI dubbing pipeline works: four stages

Research on automatic dubbing describes a similar chain. Federico and colleagues (2020) started from a speech-to-speech translation pipeline and added what dubbing needs:

  • Machine translation that produces output of a preferred length.

  • Prosodic alignment: matching the translation's phrases and pauses to the original speech segments.

  • Text-to-speech (TTS) with the duration of each utterance fine-tuned.

  • Audio rendering that adds background noise and reverberation extracted from the original audio.

A pipeline dubbing a video with several people in it also has to know who is speaking.

Diagram of the four-stage AI dubbing chain, from source video to dubbed video, with an optional subtitle review before rendering and the research problem at each stage.

Open the chain diagram at full size

The stage sections below describe research and design space: what each stage has to solve and where it breaks. They do not describe the internals of any product, Booq included. Where Booq appears in them, it is only what its case study states, and the Persian pronunciation section draws on my own text-to-speech research. The platform sections are first-hand.

Stage 1: transcription and speaker detection

Every later stage inherits the transcript. A word the recogniser invents is translated, voiced and timed as confidently as a word the speaker said. For Persian, the most useful public evidence on where automatic speech recognition (ASR) goes wrong is PSRB, a benchmark published in 2025.

Where Persian speech recognition fails

PSRB evaluates ten ASR systems, commercial and open-source. Its summary is that models generally perform well on standard Persian but struggle with regional accents, children's speech and specific linguistic challenges. The details read like a description of the audio people want dubbed:

  • Accents. Of the 12 accents evaluated, recognition accuracy was lowest for Mashhadi and Yazdi.

  • Children. Child speakers were recognised worst.

  • Several speakers. Every model made more errors on multi-speaker audio than on single-speaker audio. The authors name overlapping speech as the main challenge in such audio, even though the benchmark kept overlap to a minimum, and some models sometimes omitted an entire speaker from the transcript.

  • Genre. Audiobooks gave the lowest error rates. Talk shows, animations and films, where spontaneous dialogue, overlapping speech and background noise are common, scored substantially worse.

  • Register. The benchmark's speech is 71.3% informal.

The paper also names error types specific to Persian script and speech:

  • Word boundaries. The plural درخت‌ها ("trees") is written with a zero-width non-joiner (ZWNJ). Errors appear when the ZWNJ is omitted or replaced: درختها or درخت ها.

  • Formality flips. Informal می‌خوام برم ("I want to go") can come back as formal می‌خواهم بروم, or the reverse.

  • He-Kasreh. The short /e/ that marks possession or connection can be written out as the letter he (ه): کتاب او ("his book") can be transcribed as کتابه او. This is the ezāfe written out, the same unwritten sound Stage 3 returns to.

  • Paraphrase. خوش می‌گذره ("it's enjoyable") can become حال می‌ده ("it feels good"). The meaning survives; the words do not.

  • Hallucination. Fluent, coherent text that is not based on the audio at all.

Plain word error rate (WER) treats every substitution as a whole wrong word, so a ZWNJ slip can cost as much as a different word, or more: if the ZWNJ becomes a space, one word turns into two, which counts as a substitution plus an insertion. PSRB proposes a substitution-weighted variant, SW-WER: for each substitution, the character error rate is computed and multiplied by the number of words in that alignment, so a spacing slip costs less than a different word. Retrieval over Persian text has to normalise the same characters, as hybrid search for Persian RAG explains.

The design consequence, as I read it: the audio people want dubbed is where recognition is weakest, and a hallucinated or paraphrased line does not stay in the transcript. It becomes a confidently dubbed line. That is the argument for a review point before rendering. Booq offers both modes: run hands-off, or pause to review and edit subtitles before anything is rendered.

Speaker diarization: who spoke when

Park and colleagues define speaker diarization as the task of identifying "who spoke when". In a dub, it decides which voice each line receives. The Third DIHARD Diarization Challenge evaluated systems across 11 diverse domains and concluded that for many of them, web video included, the problem "remains far from solved".

The check I would place after diarization in any pipeline is simple: does every detected speaker have lines, and does every line have a speaker? Some systems in PSRB dropped an entire speaker from the transcript, and a dropped speaker becomes a missing voice track.

Stage 2: translation that fits the clock

A dub adds a constraint to translation: the new line has to be spoken in about the time the original took. Translate freely and the synthetic voice has to rush, or the line runs into the next one.

Isochrony: length and duration are one problem

The research name for this is isochrony. Chronopoulou and colleagues (2023) describe it as new speech that is time-aligned with the original video, "including mouth movements, pauses, hand gestures". They train a model that optimises the translation and its speech duration together, and report speech that matches the original timing better than prior work, with a simpler architecture.

Earlier work controlled length in the text. Isometric MT (Lakew and colleagues, ICASSP 2022) uses an example target of a translation within ±10% of the source length in character count, and is candid about the price: controlling output length costs translation quality, and the usual mitigation is to generate several candidate translations (N-best hypotheses) and then re-rank them. The ±10% is one paper's example, not an industry rule, and characters are only a proxy for how long a line takes to say.

When pacing goes wrong, the result is audible. YouTube's help page for automatic dubbing lists speech pacing among the reasons a video may not be eligible: original speech that is too fast "would result in an unlistenable, sped-up dub".

How strict does timing need to be?

Stricter on screen than off it, and, judging by professional practice, less absolute than research often treats it. Brannon, Virkar and Thompson (TACL 2023) studied how people dub, using 319.57 hours of video from 54 professionally produced titles. Their results challenge several assumptions common in the literature on both human and automatic dubbing. They argue for the importance of vocal naturalness and translation quality over the commonly emphasised isometric (character length) and lip-sync constraints, and for a more qualified view of how much timing constraints matter.

Whether the speaker is on screen matters too. Virkar and colleagues (2022) took the prosodic alignment model they had earlier built for on-screen dubbing and extended it to off-screen dubbing, which "requires less stringent synchronization constraints" than dubbing a speaker whose mouth is visible. The timing budget is therefore a decision per scene, not one global setting.

For Booq, the published statement is that timing is aligned to the original. That is all the case study says about timing, and all I claim here.

Stage 3: a distinct voice for every speaker

On Booq, each detected speaker gets a distinct voice. Any multi-speaker dub needs two properties from its voices: two speakers should not sound alike, and one speaker should keep the same voice from first line to last. Remember the second; the post-mortem below is about how it lost.

Open Persian speech data is thin. The authors of ParsVoice write that Persian "remains substantially underrepresented in open speech-text resources". Their corpus includes a 2,200-hour TTS-ready subset, more than 25 times larger than the largest previously available open Persian TTS dataset. Data is one constraint. The other comes before any audio exists: deciding how the text is pronounced.

Persian pronunciation: short vowels, ezāfe and homographs

Between June 2021 and September 2022 I worked on Persian text-to-speech research. It was commissioned and supervised by MCI (Hamrah-e Aval), Iran's largest mobile operator, and managed by the National Elites Foundation under its Shahid Ahmadi Roshan programme. The hard part was linguistic. In that pipeline, written text first passes through grapheme-to-phoneme conversion (G2P), which decides how it is pronounced. A text-to-speech model and a vocoder then produce the speech; the vocoder is the component that turns the model's acoustic output into an audio waveform. Persian script makes the first step ambiguous in three ways:

  • Short vowels are not written, so the text alone does not determine the pronunciation.

  • The ezāfe construction is not written either, and has to be inferred.

  • Homographs share one spelling across different words and cannot be told apart without context.

The published research says the same. The authors of GE2PE (Findings of EMNLP 2024) describe the ezāfe as an /e/ sound connecting nouns to adjectives and descriptive nouns, and give a homograph example: /kerm/, /kerem/ and /karam/ are identical in written form, کرم. They conclude that Persian G2P needs a phrase-level or sentence-level input to achieve acceptable results. Doostmohammadi and colleagues (2020) add that the ezāfe is almost always left unmarked in Persian script, causing mistakes in reading complex sentences and errors in language processing. The project is written up in the Persian text-to-speech case study.

The design consequence is my inference, not a finding from those papers: do not feed Persian TTS the fragments a subtitle track is cut into. A subtitle break can split an ezāfe chain or separate a homograph from the words that disambiguate it. If ketāb-e dust-e man ("my friend's book") is split across two subtitles and the first fragment ends at ketāb, nothing left in that fragment says ketāb takes the ezāfe /e/. Synthesise whole sentences, then fit the audio to the timeline.

Stage 4: timing alignment and the final mix

The last stage turns clips into a soundtrack that belongs to the video. In the Federico pipeline, rendering adds background noise and reverberation extracted from the original audio, so the new voice sits in the same acoustic scene as the picture.

On Booq, the finish can include styled burned-in subtitles for vertical and widescreen video, with the original audio or background music kept softly under the dub. Burned-in subtitles are part of the picture, so their layout has to be decided per aspect ratio before rendering; no player can fix it later.

Two design questions belong to this stage. The first is what to do with a line whose synthesised audio still runs long after translation. Research and practice offer three levers, and choosing between them is a design decision, not a finding:

  • Shorten the line at translation time. Isometric MT and models that optimise translation and duration together work at this point.

  • Tune the duration of each utterance in TTS. The Federico pipeline fine-tunes duration per utterance.

  • Speed the voice up, within limits. This lever comes from practice rather than research, and it only works up to a point: YouTube's help page warns that a sped-up dub can become unlistenable.

The second question is what the listener hears where a line is missing. That one turned out to be an architecture question, not an audio one.

The platform around the models

Everything above is research you can read and models you can rent. On Booq, every capability depends on outside AI providers, and each of them can fail or change independently. As CTO I defined Booq's architecture and took it from first commit to production; the Booq case study describes the platform. Every lesson in its post-mortem sits around the models, not inside them.

Diagram of the platform around the models on Booq, in four rows: safeguards around the work; a speech fallback chain through per-provider gateways; background moderation and refunds; release and backup operations.

Open the platform diagram at full size

One gateway per AI provider

Booq is a set of independent services behind one web app, with a separate gateway for each AI provider. That one decision is what lets a model be switched, a key rotated or a failing provider bypassed automatically, without touching the rest of the product. Models and provider keys change from the admin panel, without a redeploy. The case study pairs each gateway with automatic fallback, and the post-mortem below shows why speech needed it.

Durable priority queues and checkpoints

Dubbing a long video does not belong inside a web request. On Booq, heavy processing waits in durable priority queues with per-service limits instead. Limits per service are what stop one busy service from starving the others.

Translations save checkpoints and pick up where they stopped after a crash. Without checkpoints, a crash near the end of a long transcript repeats every provider call made before it, and each of those calls is paid for again.

Resumable uploads on uneven connections

Large video files reach Booq over uneven connections, and uploads, multi-gigabyte ones included, resume on a poor network instead of starting over.

The general mechanism is being standardised. Resumable Uploads for HTTP is an active Internet-Draft in the IETF HTTP working group, not yet an RFC: if the server can indicate how much data it processed before the interruption, the client resumes from that point instead of sending everything again. It is work in progress, not a description of Booq's implementation.

Price before the job, refund on failure

Billing is part of the pipeline. Each job on Booq shows its price, or its starting estimate, before it runs. The wallet is prepaid in Toman with VAT shown up front, every transaction lands in a permanent ledger, and a failed job refunds automatically. Refunds are protected against paying out twice.

That last property deserves the hardest testing: a retry after a timeout is exactly when a naive refund pays twice.

Switches, not releases

Product areas on Booq are turned on or off without a release. In Pete Hodgson's taxonomy of feature toggles, that is close to an ops toggle: one that lets operators disable or degrade a feature quickly in production.

Access is separated in the same spirit. Customers, administrators and support staff each use a separately deployed app, a sign-in from one does not work in another, and the consoles require a second factor with role-based permissions.

Post-mortem: when voice consistency lost to silence

Speech on Booq was first pinned to a single provider on purpose, because a different provider means a different voice. In a dub, consistency looked like the property to protect. Then that provider had an outage, and dubs were assembled with silence where lines were missing.

Speech now falls back through a chain of providers, on the principle that a slightly different voice beats silence.

Designing a fallback chain on purpose

The original choice was not careless; it protected a real property. The case study sums up the changes of direction that running the platform in production led to: each one "traded a comfortable assumption for a safer default". The lesson is to decide which failure you accept before an outage decides for you. When the preferred speech provider is down, a pipeline can leave a gap, fail the job and refund it, or use another voice. Only the gap hands the listener a dub with holes in it.

A second decision follows, and what comes next is my analysis, not a description of Booq: how much a fallback replaces. If only the missing line switches provider, that speaker's voice changes mid-scene. If the speaker's whole track switches, the dub stays consistent, but more work is redone.

Google's SRE book describes a related idea for overloaded services, graceful degradation: doing less work by lowering the quality of responses. The part that carries over is its warning that "the code path you never use is the code path that (often) doesn't work", and its advice to exercise the degraded mode regularly. What I would add to any speech fallback chain, as a recommendation rather than a description of Booq, is scheduled test jobs on internal clips through every provider in the chain, so the fallback is known to work before an outage needs it.

The right failure depends on the product. The assistant on this site does the opposite: when nothing on the site supports a question, it refuses with fixed copy rather than guess, and when it does not take the question to be about me, the language model is never called. In question answering, saying nothing is the safe failure, because a confident wrong answer is worse than no answer. The design is in RAG hallucination guardrails, and the lab's RAG vs raw page is built on the same rule. In a dub, silence is the failure. Which failure is acceptable is a product decision, not a model decision.

Moderation without making the user wait

Content screening on Booq used to run inside project creation, so users waited between a finished upload and their project page. It now runs in the background. Images, video, audio and text are screened, with an appeal route, and a finished result is held at "content review in progress" until the review completes. Output is not presented as ready while its review is still running.

The pattern carries over to any slow check in a media pipeline: take it off the request path, and make the in-between state explicit.

Release gates and backups someone has read

Production deploys on Booq moved from automatic to a manual approval that only unlocks after staging checks pass. Every release passes build, tests, a staging deployment and a smoke test before a person approves it for production, and an emergency rollback to the previous version can be triggered manually. The platform runs on a container cluster defined as code.

Backups moved to hourly, and the newest one was read end to end, because a backup nobody has ever read is not a backup. Google's SRE book puts it more bluntly: "No one really wants to make backups; what people really want are restores." It also sets a higher bar: you only know you can recover your recent state if you actually do so, so recovery tests should be automated wherever possible and run continuously. Reading a backup end to end is the floor, not the finish line.

Build or buy: a checklist for any AI dubbing pipeline

Failure and fallback

  • When a speech provider fails mid-job, what does the listener get: silence, a different voice, or a failed job and a refund? Decide before an outage decides.

  • When the fallback does kick in, does it replace one line or the speaker's whole track?

  • Is the fallback path exercised on purpose, with scheduled test jobs, or only during outages?

Platform and billing

  • Is each AI provider behind its own gateway, so that a model change, a key rotation or a bypass is not a release?

  • Does heavy work run from durable queues with per-service limits, outside the web request?

  • Can a long translation resume from a checkpoint, and can a large upload resume after a dropped connection?

  • Is the price or estimate shown before the job runs, and can a refund never pay out twice?

  • Is moderation off the request path, with an explicit "in review" state?

Recognition, timing and voices

  • Has recognition been tested on your own audio: accents, children, informal speech, several speakers, film and talk-show sound, not only audiobooks?

  • Does your WER scoring treat a ZWNJ spacing slip as a whole wrong word? Normalise first, or use a weighted metric such as SW-WER.

  • After diarization, does every detected speaker have lines and a voice?

  • Is there a point to review and edit subtitles before rendering?

  • How strict does timing need to be for this content: faces on screen, or off-screen narration?

  • Does Persian TTS receive whole sentences, not subtitle fragments that cut an ezāfe chain?

Release and recovery

  • Does a person approve production after staging and smoke checks, and can you roll back by hand?

  • Has anyone read a backup end to end?

FAQ

How does AI dubbing work?

An AI dubbing pipeline transcribes the original speech and detects who is speaking, translates each line so it can be spoken in about the original time, gives each speaker a distinct synthetic voice, then aligns the timing and mixes the new speech with the original background. In production, those stages run inside a platform of queues, provider gateways with fallback and, ideally, a review point before rendering.

Can AI dub English video into Persian?

Technically, yes: the chain is the same for any language pair. Dubbing into Persian adds a language-specific problem in the voice stage, because the script leaves short vowels and the ezāfe unwritten, so pronunciation has to be inferred from context. Mainstream coverage is uneven: as of September 2026, YouTube's automatic dubbing help page lists Farsi as a source language dubbed into English, and English videos are not listed as dubbable into Farsi.

Does AI dubbing match lip movements?

Not necessarily, and it does not always need to. Isochrony research sets that goal: new speech time-aligned with the original video, including mouth movements, pauses and gestures. How tightly to match is a design choice. Off-screen dubbing needs less stringent synchronisation than dubbing a speaker whose mouth is visible, and a large study of professional human dubbing argues that vocal naturalness and translation quality matter more than the commonly emphasised lip-sync constraints.

Why does an AI dub sometimes contain silence or a different voice?

One cause is a speech provider failing mid-job. A pipeline pinned to one provider keeps voices consistent but can leave gaps during an outage; one with a fallback chain fills the line with a slightly different voice. Booq moved from the first design to the second after an outage produced dubs with silence where lines were missing. Silence can also start upstream, when recognition drops a speaker and leaves nothing to translate.

How should Persian speech recognition be evaluated?

On your own audio, not on clean read speech alone. Include regional accents, children's voices, several speakers and film or talk-show sound, all conditions where PSRB found more errors, plus informal speech. Normalise ZWNJ and the Arabic and Persian forms of the letters yeh and kaf before scoring, or use a substitution-weighted metric such as SW-WER, and read the errors as well as the rate: one number does not separate a spacing slip from a fluent sentence the speaker never said.

If you are weighing a speech or dubbing build, the Speech AI service covers TTS, ASR and dubbing pipelines, including the hard Persian cases.

Sources