Skip Navigation

Fediverse @lemmy.ml

2y ago

PSA: Lemmy.world has been compromised! (Edit: Multiple Instances are down)

FYI!!! In case you start getting re-directed to porn sites.

Maybe the admin got hacked?

edit: lemmy.blahaj.zone has also been hacked. beehaw.org is also down, possibly intentionally by their admins until the issue is fixed.

Post discussing the point of vulnerability: https://lemmy.ml/post/1896249

Github Issue created here: https://github.com/LemmyNet/lemmy-ui/issues/1895

242 comments

Yea, I switched to this alt. It appears to be one of the assistant admins accts. Seems like an old fashioned anon prank, to me, they're mainly just trying to make stuff offensive and redirect people to lemonparty.
So, y'know, old school.
I don't know if any data is actually in danger, but I doubt it. I don't see why assistant admins would need access to it.
- All the bean memes are in danger! On a serious note, old-skool or not, it's a huge loss of trust in something the community-at-large is excited to see replace reddit.
  
  Par for the course. This system will never be immune to things like that. That's part of what happens when you decentralize your power. Instead of a single target that can be made highly secure, you have a distributed array of targets.
  People should certainly be engaging on here with full awareness of the reality of the Fediverse, not expecting reddit 2.0. We never will be able to offer exactly what they did. We'll be naturally worse in some areas and naturally better in others.
  
  idk, im surprised it took this long. there's a huge variety of admin teams with varying degrees of security awareness and it's been over a month since the first big influx of users started. it'll happen again too and probably not before too long
  
  On the other hand, look at where we are. This is proof that one hack can't take down Lemmy.
  
  i did switch from reddit to lemmy.world because i expected it to be a safe alternative that would atleast pay a lot of attention to security. so yes, the trust in security is broken a lot with this. especially since it happend so soon after so many people joined. i already think about maybe making my own instance to keep my account safe in the future.
- My concern is that configuring the site to automatically redirect users sounds like they have pretty large control over the site - the kind of control that I would assume is usually limited to users with root access on the server.
  Obviously hope nothing of value is lost and that there is a proper off-site backup of the content.
  Edit: See Max-P's comment, it looks like the site redirection was accomplished in a way that IMO suggests they do NOT have full control over the site. We'll obviously have to wait for the full debrief from the admins.
  
  If it was just DNS that doesn't mean too much. If it was just DNS it seems to be back up. It's like changing the number in a phone book.
  
  Yeah the "redirect somewhere else" attack definitely doesn't necessarily require any particular control of the site. Usually it's noticing that you can trick some text into being run as Javascript, instead of interpreted as text... And then you just stick in a cheeky little <notarealscript>window.location = "https://www.badsite.horse"</notarealscript> into that spot.
  Then every time that comment, username, (in this case apparently) custom emoji, etc. gets loaded, whoops, the code runs and off you go!
  So no control of the site is required at all.
- I don't see why assistant admins would need access to it.
  because it's easier than figuring out what permissions they actually need
  
  Lemmy permission system is very limited, it's a boolean for admin
- probably even the top admin don't, it's gonna be encrypted, so even they don't know your password(except if they changed the code to store it in .txt) but always use differnt password in the internet
  
  Nothing is encrypted except a user's password. If you have access to the database you can replace that with a known password hash.

Main instance hacked? Time to use an alt!
The first hack is a rite of passage for every site that gets big. It means we've been recognized!
Luckily, this seems to be a standard troll (with some tech knowledge) - they've defaced the site and put redirects to shock sites, rather than injecting actual malware or quietly collecting everyone's passwords. This could be much worse.

I tried to reproduce the exploit on my own instance and it appears that the official Docker for 0.18.1 is not vulnerable to it.
It appears that the malicious code was injected as an onload property in the markdown for taglines. I tried to reproduce in taglines, instance info, in a post with no luck: it always gets escaped properly in the <img alt="exploit here"> property as HTML entity.
lemmy.world appears to be running a git commit that is not public.
- I actually consider it good news that the redirection is happening this way (something that can be done just by having the lemmy credentials of an admin) vs something indicating they have access to the server itself.
  
  Yep, same. It was also the most likely scenario.
  It looks like it was an individual admin getting hacked. Not good but not the worst. Most fallout will probably be whether their security practices were sufficient for an admin and whether lemmy has good enough contingencies for this sort of thing. Lemmy’s 2FA is probably a hot issue now though.
- It does look like most instances will be vulnerable judging by the fix. It's not custom code; it's in lemmy-ui proper.
  https://github.com/LemmyNet/lemmy-ui/pull/1897/files
- It seems the database and the server itself is not compromised? Just an admin account that used to post a markdown XSS exploit?
  
  Pretty much, and it's not even XSS (it's not cross-site), it's just plain basic HTML injection breaking out of Markdown. At least as far as I was able to find.
- Last I saw, they were on 0.18.1, unless a very recent update was installed. Do you happen to have a full list of domains they were redirecting to? Just want to be sure they were only going to "harmless" offensive sites, and not something worse.
  
  Only lemonparty (which then redirects to chaturbate) and the pedo image hosted in the pictrs of lemmy.world itself. I saw no evidence of anything else, as people said, it's a pretty oldschool type of hack to disturb not spread malware.
  But I didn't dig that much further than that, and it's only a snapshot of what I gathered before it got fixed. I Ctrl+F "lemonparty" in view source and pasted the JSON in VScode and that's about it. Didn't dig much deeper if that was just a red herring.
  
  As for the version, my instance reports it as
  
  0.18.1-2-ga6cc12afe
  So it seems to be using some extra patches, but I can't find that commit on GitHub which indicates it might not be public, or cherry-picked locally.
  So with this in mind, either it's just innocent performance patches, or someone potentially also introduced the markdown vulnerability.
  Although it's also entirely possible I suck and wasn't able to reproduce it correctly/had wrong quoting or something. Hopefully the devs can shine some light in the details.
- Max-P doing the Lord's work

How did it happen and what does this mean for me as a user of lemmy.ml who also follows people on lemmy.world?
- One of the admin accounts appears to have been compromised. The owner/other admins appear to be aware now because that account had its admin access revoked and offending posts are being removed.
  Definitely opens up a big question about the security of Lemmy instances that I am sure will be discussed over the next few days.
  
  I wouldn't assume reasons why or that it's fixed until that consensus has been more widely reached.
  
  Definitely opens up a big question about the security of Lemmy instances that I am sure will be discussed over the next few days.
  They added 2FA login to lemmy in one of the newer updates. Probably pretty pertinent for any admins to use it....
  
  Thanks for the context
  
  They really need to improve their 2fa implementation
- Not a whole lot - you might see some spam being federated from lemmy.world but I'd expect the lemmy.ml and lemmy.world admins will fix it, and them clean it up.
  That's probably good stress test to figure out how to handle that.
  
  Thanks for the response very helpful.

God damn, spez-funded hacker groups already is trying to disrupt the resistance.
- Fuck spez
  
  This is going to turn into some obligatory response.
  "Thank you everyone for coming together to discuss the planned future for the news community." Everyone: "Fuck spez."

Hmmm. Don’t know what the fall out of this will be. But a lot of lemmy is on that server. Unfortunately. Maybe we’ll learn a lesson in the value of decentralisation.
Ruud also runs mastodon.world, FYI.
- This is why it makes sense for communities to not all pile into one instance, it gives one instance admin too much power and responsibility over everything.
- was just some of the admin in the lemmy, i don't think they share the same admins
- mastodon.world seems okay, but whos to say where the silos are between that and lemmy.world.
- I don't think there will be too much fallout. Sites get hacked.

lemmy.world was briefly back to normal and there had been a post saying that everything was fine now - it's not.
The site has just started doing the same thing again.
Please do not try using lemmy.world for the time being.
- the post saying everything was fine now was coming from the same account that was originally compromised
  
  We've changed our name to Israel. - The Admins.
  
  Lol so how do you expect to be notified then? You don't think they can get their account back? They'll get it back eventually.
- i just got logged out of my account from Jerboa and can't login anymore. my is completely wiped from my app now.
  edit: okay seems the admins have taken down lemmy.world and thats probably why it happend in the app. but its weird that it just wipes the login and data of the instance in the app.. weird.
  
  My self hosted instance has hiccups sometimes and Jerboa just doesn't handle it super well. You can swipe away the app and reopen once the server is back and it should come right back up.
  
  Jerboa tries to log in with session info passed to the server, if the server doesn't respond properly then it just calls you Anonymous, because it can't acquire your username and info. That's probably what's happening.

Being a part of Lemmy in these early days has been kind of interesting, seeing all of the bugs and bits that will be ironed out over time. One day when Lemmy is as old as Reddit it will all be folklore. Maybe.
- This'll definitely be remembered. It's good for us, we needed the wakeup call.

GitHub PR fixing the bug: https://github.com/LemmyNet/lemmy-ui/pull/1897/files
If your instance has custom emojis defined, this is exploitable everywhere Markdown is available. It is NOT restricted to admins, but can be used to steal an admin's JWT, which then lets the attacker get into that admin's account which can then spread the exploit further by putting it somewhere where it's rendered on every single page and then deface the site.
If your instance doesn't have any custom emojis, you are safe, the exploit requires custom emojis to trigger the bad code branch.
- But won't custom emojis from remote instances still trigger the exploit?
  
  Apparently not per the post-hack report: https://lemmy.ml/post/1901079
  
  Apparently the custom emojis are rendered as static images when federated to outside instances so it's clean.
- I see a new lemmy-ui docker image has been pushed an hour ago, tagged 0.18.2-rc.1. Anyone know if it fixed the issue?
  Edit: yep, it's fixed: https://github.com/LemmyNet/lemmy-ui/commit/e80bcf53acb8ce25ed5ef6b7eb16b90f0b07e8f1
- I'm not particularly familiar with XSS but I'm curious how a frontend exploit can compromise an instance?
  Presumably the injected XSS stores the admin's JWT somewhere for the exploiter?
  Then using that JWT they can effectively login as the admin which gives them access to whatever admin dashboard there is, but does that actually compromise the backend at all?
  edit: for anyone curious there's a bit of a breakdown of how it works here: https://feddit.win/comment/244427
  
  Inject exploit into a comment using custom emoji.
  Front-end parses the emoji incorrectly allowing JavaScript to be injected.
  JavaScript loads for everyone to views a page with the comment and sends their token and account type to the hackers domain.
  Hacker parses received tokens for admins and uses that to inject redirects into the front page of the Lemmy instance.
  To answer your other questions:
  IMO there probably should be better parsing to remove this stuff from the back-end, so I'm not sure the front-end solution is the complete solution, but it should get things largely under control.
  Back-end is theoretically not compromised besides needing to purge all the rogue comments. Attacker presumably never had access to the server itself.
  Probably needs to be a mass reset of ALL passwords since lots of people's tokens were sent during the attack, so their accounts could be compromised.

lemmy.blahaj.zone got hacked too, looks like the same people
https://lemmywinks.xyz/post/320087
- They also changed the allowed/blocked instances to allow threads.net and defederate lemmy.ml, just like they did on lemmy.world: https://lemmy.blahaj.zone/instances
- Huh... so this probably is more sophisticated than a single acct breach then. Lovely.
  
  Yeah, I'd recommend any server admin that doesn't have 2FA turn it on ASAP until we know what their exploiting
- blahaj admins are aware and have the site down with a splash screen now
- Links to this video: https://www.youtube.com/watch?v=Z1K4BUtHsO4
  
  Yup they must of just put that up after I posted and @ the admins

- we did it Reddit! /s
- I saw this and laughed. Yes, that's definitely how copyright works.
- lmao
- Twitter taking Threads down and posting this lol

4AM in the Netherlands where the instance owner Ruud lives... hopefully his assistant admins can clean it up, but it might be a bit before he even knows anything is wrong.

They're stealing jwt tokens and noting when they're admin tokens.
https://lemmy.sdf.org/post/696053 https://lemmy.sdf.org/comment/850269

- It looks like they're in the process. The compromised account was demoted from admin and I see posts are being removed. There will definitely need to be some sort of investigation into how this happened, though.

Don't know if this will be relevant at all, but I'm almost hoping this will force Lemmy devs to abandon the obscure markdown crate they use for pulldown-cmark.
Using an obscure markdown implementation just because it supports spoiler tags always sounded like a silly decision to me!

Just went there and didn't immediately see anything out of the ordinary, but then was redirected to Chatroulette, lol yikes
- Really hoping it's "only" redirecting to offensive sites, and not to malware. I got redirected a few times, before I closed my browser.
  
  TBF modern browsers are remarkably secure from being a vector to pwn your computer these days.
  EDIT: I don't endorse hanging out on a compromised lemmy.world. Focus on the implication for the bigger lemmyverse though. A hack coming through to you is unlikely.
  
  You can't get malware or viruses just by visiting a site

The admins now appears to have taken down the backend in an effort to stop the defacing.

Damn first vlemmy.net (my original instance) dies, and now one of the largest is hacked…
- Yea, bad timing it seems, especially as lemmy just got on top of its scaling issues.
  They seem to be unrelated. The vlemmy story is mysterious, unless something new came out, but either their home server died or they got scared of whatever bad/illegal stuff landed on their home server and just wiped it all and walked away. A bad story that shouldn’t happen, but, if true, a bad admin that we are probably better off without unless they do things somewhat better.
  The lemmy.world story seems to be that an admin had their credentials hacked. Not good but also somewhat ordinary. Hopefully they just need some better security practices. There are questions around how much lemmy the software contributed to this hack and how much it can prevent a rogue admin from causing damage. I’d bet that there are improvements to be made but that in the end any admin of anything is a vulnerable point of attack. This may just be an individual’s bad luck or bad practices.
  For me, it highlights the issues with having relatively centralised instances like lemmy.world. One admin gets hacked and a quarter of lemmy is under their control!
  
  The real issue IMO is that recent events have pretty much proven that both big instances and small instances are problematic for different reasons.

For those not aware, the beehaw server did intentionally shut their instance down to avoid any issues.
See announcement here: https://hachyderm.io/@beehaw/110687918465426082

The "Hot" sort topic:

Looks like this thread is getting mass downvoted by bots btw

I'm seeing zero comments come out of Lemmy.world in the past 15 minutes, app users shouldn't have been redirected... and users commenting from other servers should be going to communities homed there. I wonder if they shut off federation. I normally see over 10 comments a minute: https://lemmyadmin.bulletintree.com/query/comments_ap_id_host_prev?output=table&timeperiod=15
- Last post received in my instance from them was over an hour ago. I usually see one or two a minute. Comments stopped at the same time and those are usually about every 5 seconds.
- Hmm. They seem to have cleaned up a lot of things by now. If federation is an issue that might something the hacker did? Though pausing federation as a precaution makes sense.

Wtf? What even is happening?

I literally just made a community over there 20 mins ago fml
- No biggie. Choose another server and create it there, too. Largest communities will win in the long run.

Is @Ruud's mastodon.world instance still okay?
- Seems to be.

Compromised in what way? Can you post proof?
- Just go to https://lemmy.world and see for yourself, although be careful it's nasty.
  As of now it looks like this:
  And then it randomly redirects to gore sites like lemonparty or chaturbate or some pedo shit. It's pretty bad.
  
  Alright thanks
- One of their admins (MichelleG) began posting messages about federation with only Threads. The site is redirecting users to Lemonparty (now there's a throwback). Site information has been vandalized with racist slurs.
  
  Well, that escalated quickly.
- Just go to lemmy.world and see for yourself. (Or don’t actually, might give you a virus or something idk)
  
  Yeah I would like someone to post a screenshot i dont want to leak my ip
- image here ![] (https://lemmy.ml/pictrs/image/0332b83a-ab01-4c99-9155-2a08b02fb652.png)
  among several others
  
  Could you spoiler that weirdo image

It was cleaned up on the home page, but now back to being defaced as of this comment time.
Another user on the site confirmed this:
- Oh wow again? 10 min ago it was clean! The .world admins are having a productive day lol
  
  Now I'm unable to open lemmy.world, even on liftoff. Mods must have taken it down.

Time to make an alt! Been thinking about switching instances anyway, so this is a nice test. Hope the situation gets resolved soon.

Well shit, just set up shop here. Had made a back up account on vlemmy the other day and that's down too. What's going on?
- Vlemmy.net went down? Anyone know why or how?
  
  No one knows, site went down suddenly with no explanation

It appears that the deface attack is back in full swing (racial slurs and all the redirects)

Yikes, hope they can find and disclose whatever they used to compromise an admin account.

Technical details, is it the sidebar: https://lemmy.ml/post/1896249
- It's actually custom emoji code.
  https://github.com/LemmyNet/lemmy-ui/pull/1897/files

F

Just clicked into Lenny.world and saw “site has been seized by Reddit for copyright infringement “

Looked like the admin u/MichelleG over there got hacked. That account posted a couple stickied posts and then all the css and links started changing on the site.
- You would think an admin account would have 2fa enabled (unless the hack was due to a security issue in lemmy itself, but it doesn't seem to be the case).

Ah, crap...

I really hope they have backups in place.
- I was once doing work at a company that provided tech support and security for local businesses. There were a couple big instances of the companies being hacked with ransomware etc. On every occasion, we of course ask, "when was your last backup done?" And without fail, every one of them always responded, "backup?"
  
  Good ol' FAFO

Lemmy.world front page is back up, but I am now logged-out
- They still have fake content on the front page. In a bogus community.
  
  Do not know
  
  Login currently does not work. After you log in, you are not logged in.

Seems fine now

Oof. Oh well, at least they’re getting recognized I guess.

It’s fine now.

I decided to check it and it tells me that 'The site has been seized by Reddit for copyright infringement'.
- Same, I think I’m staying up to watch the drama tonight lmao
  
  The ADHD has found an excuse not to let the brain turn off

242 comments