A translation exploration

Being a speaker of more than one language—English, Serbo-Croatian, and beginner-level Italian—some months ago I gained an interest in video translation and subtitling. I discovered many online apps that aid with this; with a plethora of online AI tools and a slightly lighter wallet, you can quickly find yourself with a subtitled video in the language of your choice. You may just have to make a few corrections to capture local idioms that get lost in translation.

Given that I simply wanted to spit out some subtitles without paying to call a third-party LLM or as above paying to use a web application, I turned to Hugging Face’s open-source models. Soon enough, I had a script running locally that output my subtitles in .srt format. I downloaded Kdenlive, an open-source video editor available for my Ubuntu machine, and overlaid the subtitles. Voilà! I had my first subtitled video, allowing me to share Serbo-Croatian videos with my wife (who doesn’t speak the language) for the first time.

However, even though it was free, I wasn’t completely satisfied. The quality of the subtitles was quite low compared to those generated by paid online applications. There were weird inconsistencies, like overlapping text from different time sequences or subtitles that didn't quite line up with what was spoken at that exact moment.

My conclusion? The faults in the quality were a result of the model I was using to generate the translations and subtitles. I also didn’t enjoy having to run a command locally to generate the srt files, then maintain my folders locally. Great – lets make a web app then. If I try to scale the model locally with optimised libraries like faster-whisper, it will still take very long to generate more accurate subtitles. I need more compute power.

Technical details

I wanted a web app and I wanted to run the faster-whisper library to generate subtitles. It quickly became clear that it’s far too intensive to have them running in the same application. So, they need to be separated.

We have a FastAPI Subtitle Service exposing a REST endpoint for MP3 audio uploads. When a request comes in, the application uploads the audio file to Google Cloud Storage (GCS) and transactionally creates a message record in an Outbox table. I employ the outbox pattern for this see diagram below.

Background Outbox Relay Workers poll this table and publish pending tasks to our message queue. On the consuming side, Inbox Workers pick up those queue events and record them in an Inbox table to ensure idempotent processing.

Finally, our Hugging Face Transcription Service checks the inbox table for pending jobs, downloads the audio from GCS, runs transcription/translation to generate .srt files, and uploads the final subtitles back to GCS. (Note: While DB polling by the Hugging Face service isn't ideal for scale due to idle GPU costs, it served as our pragmatically simple solution given time constraints. I won’t go into my proposed solutions for this.)

Source: https://microservices.io/patterns/data/transactional-outbox.html.

Conclusion

The system was put into production and successfully generating subtitles. However, early testing with the Large-v3 model hasn't met my expectations - surprisingly, the transcripts from the fancy Google-hosted GPU are not a whole lot more accurate than what I was getting locally running the Medium model on my CPU. I will continue to tinker around with the models, but the speed improvements are great. No more waiting around on my desktop to finish the run.

In future articles I can explore how I created Effectively-Once Processing using Idempotency. I need to create a nice solution to not have the GPU server running the model running permanently, as it polls the db.