Pairing WhisperX with LLM's
Pairing WhisperX with LLM's I've found some time to continue my project. In my last article, I mentioned that I was using faster-whisper for both transcription and translation, as well as dealing with the annoying issue of overlapping or misaligned subtitles that needed manual correction. This is a common problem for users of the library (you can read more about it in this guide on fixing Whisper timestamp alignment ). To fix this, I decided to switch to the WhisperX library. While it is a much heavier tool, it uses faster-whisper under the hood and layers forced phoneme alignment on top. This resulted in fantastic transcriptions, but the built-in translations were still poor for less popular languages. To solve this, I bypassed Whisper's native translation entirely and routed the transcriptions through Google's Gemini LLMs to translate them into my preferred language. Technical Details Next, I had to figure out how to structure the data for translation. In my...