Pairing WhisperX with LLM's

I've found some time to continue my project. In my last article, I mentioned that I was using faster-whisper for both transcription and translation, as well as dealing with the annoying issue of overlapping or misaligned subtitles that needed manual correction. This is a common problem for users of the library (you can read more about it in this guide on fixing Whisper timestamp alignment).

To fix this, I decided to switch to the WhisperX library. While it is a much heavier tool, it uses faster-whisper under the hood and layers forced phoneme alignment on top. This resulted in fantastic transcriptions, but the built-in translations were still poor for less popular languages.

To solve this, I bypassed Whisper's native translation entirely and routed the transcriptions through Google's Gemini LLMs to translate them into my preferred language.

Technical Details

Next, I had to figure out how to structure the data for translation. In my code, a subtitle "Cue" is defined simply as

@dataclass(frozen=True)
class Cue:
    start: float
    end: float
    text: str

When building the prompt for Gemini, I decided it needed three distinct layers of context to produce an accurate translation:

Global context: The overarching theme or subject of the entire video.
Local context: A sliding window of cues immediately before and after the target cue.
Target cue: The specific text block needing translation.

Conveniently, Google's genai client is asynchronous, which allowed me to batch all the API calls and process the cue translations concurrently.

I am incredibly happy with the results. While Whisper's built-in translation model struggles with low-resource languages, its transcription engine remains solid—and pairing it with an LLM via smart context batching completely solved my subtitle pipeline.

A translation exploration