2w ago

VLC player demos real-time AI subtitling for videos

www.theverge.com VLC player demos real-time AI subtitling for videos

The video player is testing a new AI tool to create subtitles in over 100 languages.

cross-posted from: https://lemmy.ca/post/37011397

The popular open-source VLC video player was demonstrated on the floor of CES 2025 with automatic AI subtitling and translation, generated locally and offline in real time. Parent organization VideoLAN shared a video on Tuesday in which president Jean-Baptiste Kempf shows off the new feature, which uses open-source AI models to generate subtitles for videos in several languages.

Linux and Tech News @lemmy.linuxuserspace.show

Leo @lemmy.linuxuserspace.show

2w ago

VLC player demos real-time AI subtitling for videos

www.theverge.com /2025/1/9/24339817/vlc-player-automatic-ai-subtitling-translation

38 8

Opensource @programming.dev

Otter @lemmy.ca

2w ago

VLC player demos real-time AI subtitling for videos

www.theverge.com /2025/1/9/24339817/vlc-player-automatic-ai-subtitling-translation

63 4

Technology @lemmy.zip

BrikoX @lemmy.zip

3w ago

VLC player demos real-time AI subtitling for videos

www.theverge.com /2025/1/9/24339817/vlc-player-automatic-ai-subtitling-translation

29 10

90 comments

This might be one of the few times I’ve seen AI being useful and not just slapped on something for marketing purposes.
- And not to do evil shit
  
  But the toppings contains potassium benzoate.

This sounds like a great thing for deaf people and just in general, but I don't think AI will ever replace anime fansub makers who have no problem throwing a wall of text on screen for a split second just to explain an obscure untranslatable pun.
- Bless those subbers. I love those walls of text.
- Translator's note: keikaku means plan
- They are like the * in any Terry Pratchett (GNU) novel, sometimes a funny joke can have a little more spice added to make it even funnier
- It's unlikely to even replace good subtitles, fan or not. It's just a nice thing to have for a lot of content though.
  
  I have family members who can't really understand spoken English because it's a bit fast, and can't read English subtitles again, because again, too fast for them.
  Sometimes you download a movie and all the Estonian subtitles are for an older release and they desynchronize. Sometimes you can barely even find synchronized English subtitles, so even that doesn't work.
  This seems like a godsend, honestly.
  Funnily enough, of all the streaming services, I'm again going to have to commend Apple TV+ here. Their shit has Estonian subtitles. Netflix, Prime, etc, do not. Meaning if I'm watching with a family member who doesn't understand English well, I'll watch Apple TV+ with a subscription, and everything else is going to be pirated for subtitles. So I don't bother subscribing anymore. We're a tiny country, but for some reason Apple of all companies has chosen to acknowledge us. Meanwhile, I was setting up an Xbox for someone a few years ago, and Estonia just... straight up doesn't exist. I'm not talking about language support - you literally couldn't pick it as your LOCATION.
- That still happens? Maybe wanna share your groups? ;)

Finally, some good fucking AI
- I was just thinking, this is exactly what AI should be used for. Pattern recognition, full stop.
  
  Yup, and if it isn't perfect that is ok as long as it is close enough.
  Like getting name spellings wrong or mixing homophones is fine because it isn't trying to be factually accurate.
- Finally some good AI fucking 🤭

What’s important is that this is running on your machine locally, offline, without any cloud services. It runs directly inside the executable
YES, thank you JB
- Justin Bieber?
  
  JB stands for Jean-Baptiste, who is the main maintainer of vlc
  
  Jack Black?
  
  Ah JBK of course!

As long as the models are OpenSource I have no complains
- And the data stays local.

The nice thing is, now at least this can be used with live tv from other countries and languages.
Think you want to watch Japanese tv or Korean channels with out bothering about downloading, searching and syncing subtitles
- I prefer watching Mexican football announcers, and it would be nice to know what they're saying. Though that might actually detract from the experience.
  
  GOOOOOOAAAAAAAAALLLLLLLLLL

Amazing. I can finally find out exactly what that nurse is yelling about while she gets railed by the local basketball team.
- Something about a full-court press?

Will it be possible to export these AI subs?
- Imagine the possibilities!

Now I want some AR glasses that display subtitles above someone's head when they talk à la Cyberpunk that also auto-translates. Of course, it has to be done entirely locally.
- I guess we have most of the ingredients to make this happen. Software-wise we're there, hardware wise I'm still waiting for AR glasses I can replace my normal glasses with (that I wear 24/7 except for sleep). I'd accept having to carry a spare in a charging case so I swap them out once a day or something but other than that I want them to be close enough in terms of weight and comfort to my regular glasses and just give me AR like overlaid GPS, notifications, etc, and indeed instant translation with subtitles would be a function that I could see having a massive impact on civilization tbh.
  
  I believe you can put prescription lenses in most AR glasses out there, but I suppose the battery is a concern..
  I'm in the same boat, I gotta wear my glasses 24/7.
  
  It'd be incredible for deaf people being able to read captions for spoken conversations and to have the other person's glasses translate from ASL to English.
  Honestly I'd be a bit shocked if the AI ASL -> English doesn't exist already, there's so much training data available, the Deaf community loves video for obvious reasons.
  
  soon
  Breaking news: "WW3 starts over an insult due to a mistranslated phrase at the G7 summit. We will be nuked in 37 seconds. Fuck like rabbits, it's all we can do. Now over to Robert with traffic."

As vlc is open source, can we expect this technology to also be available for, say, jellyfin, so that I can for once and for all have subtitles.done right?
Edit: I think it's great that vlc has this, but this sounds like something many other apps could benefit from
- It's already available for anyone to use. https://github.com/openai/whisper
  They're using OpenAI's Whisper model for this: https://code.videolan.org/videolan/vlc/-/merge_requests/5155
  
  Note that openai's original whisper models are pretty slow; in my experience the distil-whisper project (via a tool like whisperx) is more than 10x faster.
  
  Has there been any estimated minimal system requirements for this yet, since it runs locally?
- In the *arr suite, bazarr has a plugin called Subgen which you can add and you can set it to generate subtitles on your entire library if you want, or only missing subtitles. The sync is spot on compared to 90% of what Opensubtitles delivers. I sometimes re-gen them with this plugin just because opensubtitles is so constantly out of sync (e.g. highly rated subtitles 4 lines will be at breakneck pace and the next 10 will be super slow and then everything is 3 seconds off)
  It isn't in-player but it works. The downside is it is a larger model and takes ~20 minutes to generate a movie length of subtitles.
- I hope it's available for Stash App. I wanna know what this JAV girls are saying.
- crunchyroll is currently using AI subtitles. it's obvious because when someone says "mothra. Funky..." it captions "mother fucker"
  
  That explains why their subtitles have seemed worse to me lately. Every now and then I see something obviously wrong and wonder how it got by anyone who looked at it. Now I know why. No one looked at it.

The technology is nowhere near being good though. On synthetic tests, on the data it was trained and tweeked on, maybe, I don't know.
I corun an event when we invite speakers from all over the world, and we tried every way to generate subtitles, all of them run on the level of YouTube autogenerated ones. It's better than nothing, but you can't rely on it really.
- No, but I think it would be super helpful to synchronize subtitles that are not aligned to the video.
  
  This is already trivial. Bazarr has been doing it for all my subtitles for almost a decade.
- is your goal to rely on it, or to have it as a backup?
  For my purpose of having backup nearly anything will be better than nothing.
  
  When you do live streaming there is no time for backup, it either works or not. Better than nothing, that's for sure, but also maybe marginally better than whatever we had 10 years ago
- Really? This is the opposite of my experience with (distil-)whisper - I use it to generate subtitles for stuff like podcasts and was stunned at first by how high-quality the results are. I typically use distil-whisper/distil-large-v3, locally. Was it among the models you tried?
  
  I unfortunately don't know the specific names of the models, I will comment additionally if I will not forget to ask people who spun up the models themselves.
  The difference might be that live vs recorded stuff, I don't know.
- You were not able to test it yet calling it nowhere near good 🤦🏻
  Like how should you know?!
  
  Relax, they didn't write a new way of doing magic, they integrated a solution from the market.
  I don't know what the new BMW car they introduce this year is capable of, but I know for a fact it can't fly.

And yet they turned down having thumbnails for seeking because it would be too resource intensive. 😐
- I mean, it would. For example Jellyfin implements it, but it does so by extracting the pictures ahead of time and saving them. It takes days to do this for my library.
  
  Yeah, I do this for plex as well, and stash. I think if the file already exists in the directory vlc should use it. It’s up to you to generate them. That is exactly how cover art for albums on songs worked in VLC for a decade before they added the feature to pull cover art on the fly.
- Video decoding is resource intensive. We're used to it, we have hardware acceleration for some of it, but spewing something around 52 million pixels every second from a highly compressed data source is not cheap. I'm not sure how both compare, but small LLM models are not that costly to run if you don't factor their creation in.
  
  All they’d need to do is generate thumbnails for every period on video load. Make that period adjustable. Might take a few extra seconds to load a video. Make it off by default if they’re worried about the performance hit.
  There are other desktop video players that make this work.

When we getting amd's fsr upscaling and frame-gen? Also would subtitles make more sense to use the jellyfin approach.
- I have an AMD card, add VLC as a game in the drivers, and you can turn on AMFM (frame gen).
  If it doesn't work you could just turn it on system wide in display settings of the Adrenaline Software (gear upper right corner, display/gaming).
  I think it requires at least a 6000 series GPU however.
  If you have a Samsung TV or other modern smart TV connected to a laptop, you can also turn on frame-gen using Auto Motion Plus, set to Custom.
  Judder Reduction 10 is double frames, so 24 FPS -> 48.
  
  I'm using Linux. There probably is a way to have a similar outcome but an integrated solution as vlc and amd once mentioned would be better.

Hes maaaa, hes what? Hes maaaaaa!

Wonderful! Now we need descriptive audio for the visually impaired!

I hope Mozilla can benefit of a good local translation engine that could come out of it as well.
- They technically already do with Project Bergamot.
  
  I know they do, but it's lacking so many languages.

I will be impressed only when it can get through a single episode of Still Game without making a dozen mistakes

No such comment yet? I'll be the first then.
Oh no, AI bad, next thing they add is cryptocurrency mining!
- ai for accessibility is nowhere near the same thing as crypto mining
  
  Lol, that is his opinion as well, this was sarcasm, i’m 99% sure Yours does not seem so sarcastic…
  
  But it's burning amazon forests for capitalist greed!
  /s

90 comments