Suboptimal ways to respond to a public security incident
Suboptimal ways to respond to a public security incident
This issue is already quite widely publicized and quite frankly "we're handling it and removing this" is a much more harmful response than I would hope to see. Especially as the admins of that instance have not yet upgraded the frontend version to apply the urgent fix.
It's not like this was a confidential bug fix, this is a zero day being actively exploited. Please be more cooperative and open regarding these issues in your own administration if you're hosting an instance. ๐
You're viewing a single thread.
IMO itโs not a good idea to be discussing attack vectors publicly when a number of other instances are unpatched and the exploit has been in the wild for less than a day.
I agree that admins need to work together, but discussing it in public on Lemmy so soon after the attack isnโt the way. There exists a Matrix channel for admins, thatโs where this type of thing should go.
When a vulnerability at this level happens and a patch is created, visibility is exactly what you need.
It is the reason CVE sites exist and why so many organizations have their own (e.g. Atlassian, SalesForce/Tableau )
It is also why those CVE will be on the front page of sites like https://news.ycombinator.com to ensure folks are aware and taking precautions.
Organizations that do not report or highlight such critical vulnerabilities are only hurting their users.
It is common practice to notify affected parties privately and then give full details to the public after the threat is largely neutralized. Expecting public disclosure with technical details on how to perform the attack in less than 24 hours goes against established industry norms.
That only stands true when the issue is not being actively exploited.
Exactly
Yes, the vulnerability is out there. Maybe the root cause actually introduced a LOT of vulnerabilities. The fix is being pushed at a frantic pace. To expect the devs to take time out of the mad rush to notify those impacted to do a proper writeup is just insanity.
The way I see it? This (hopefully) got fixed pretty much instantly and there is active work to get the fix applied by the people who need to apply it. That is what should be done. Give it a week or two to see how they handle the public disclosure side of things.
I strongly disagree with some of your points.
Yes, the vulnerability is out there. Maybe the root cause actually introduced a LOT of vulnerabilities. The fix is being pushed at a frantic pace. To expect the devs to take time out of the mad rush to notify those impacted to do a proper writeup is just insanity.
It's not insanity. It's called incident management and it's something the development team needs to build a proper procedure around, given the expanded scope of this project. I agree that the devs working on identifying, mitigating, and fixing the vulnerability should not be expected to also handle the communication. They need to designate someone for that role.
A 0-day was actively being exploited in the wild. There was confusion, misinformation, and a general lack of information.
You need to:
- Indicate that you are aware of an ongoing problem and are working to identify it. This let's people know there is an issue and that you're aware of it. You can do this without giving specific details on how to replicate the exploit. This includes server admins publicly acknowledging that they are aware of the issue and will provide updates when they have them, to alleviate the concerns of their user base.
- Once a mitigation are known, you publish that, in as many channels as you need to get that information out to the people who need it. So that server admins are aware of what they need to do to reduce their risk.
- Once a fix is in place, you publish that, same as above.
The way I see it? This (hopefully) got fixed pretty much instantly and there is active work to get the fix applied by the people who need to apply it. That is what should be done.
And how do you know this since it's not been communicated? Most of the information I (as a person running a lemmy server) have been able to glean is from random threads spread across random communities.
Give it a week or two to see how they handle the public disclosure side of things.
A couple of weeks for a postmortem. Sure. A couple of weeks for an active, in the wild, 0-day, to officially communicate that the problem exists and how to mitigate/patch it. Absolutely not. I still don't see a security alert on the GitHub telling me I should be updating to <insert version> to patch an active exploit and it's been how many hours now?
And if this were a large company? Yes.
This is an open source project with less than 200 devs with the VAST majority coming from two.
Part of this is very much the learning curve and why you should very much think twice about using open source passion projects in "production". This is the kind of stress testing that comes from lemmy/mastodon/The Fediverse actually having users.
But also?
- It has been indicated that there is an ongoing problem and the fixes are (hopefully) done with frantic work based on https://github.com/LemmyNet/lemmy-ui/pulls?q=is%3Apr+is%3Aclosed
- Publish mitigation in appropriate bulletin systems: It is far from textbook, but this is a social media platform. Users are spreading word of mouth while devs do actual work
- Publish fix in appropriate bulletin systems: It is far from textbook, but this is a social media platform. Users are spreading word of mouth while devs do actual work
Most of the information I (as a person running a lemmy server) have been able to glean is from random threads spread across random communities.
So you are saying that you were told there is an issue. And you can do exactly what I did while writing this message: Check the github page.
Do I think the lemmy devs are doing everything by the book? Hell no.
Do I think, given the resources available and the timeframe of the attack, that they are doing it correctly? Yes. They identified the vulnerability, (hopefully) implemented a mitigation, and pushed that all within 24 hours. Popular docker containers have already been updated, users are spreading The Good Word, and so forth. And I would much rather they use their limited resources to focus on actual fixes than doing proper writeups, just so long as the fixes are getting propagated.
Optimally? I want those proper reports filed within the next day or two. Given that this is likely NOT a full time job and all the chaos of the past 24 hours or so? I'll give them a week.
And if your complaint is that they aren't behaving the same way large corporations and massive projects (that often became corporations) do? Maybe Lemmy is not for you. And I don't mean that in an insulting manner. If I were tasked with finding a message board solution or whatever for my company, there is absolutely zero chance I would recommend Lemmy. It is not production quality.
But for shitposting and actively not providing PII or anything useful? Let's see how things get hardened from here on out.
Is the project small? Yes.
Did it explode in popularity leaving the devs overwhelmed? Certainly.
Do I expect them to strictly follow established ITIL incident management? No.
Do I expect them to communicate in a consistent way when an incident happens? Yes.
I agree the primary developers should be left to fixing the problems but there are enough active members of that project that someone could have handled communication in a more concise and official way. I don't consider random posts in asklemmy or selfhosting by random users just guessing to be a substitute for that.
If the project is going to persist and grow it needs to get better at that. Pointing it out isn't shitposting.
Again, how many "active members" are likely to understand the issue well enough to make that report. Or are they going to need to use up the time of those core developers to understand well enough to write it up?
I've been through similar a decent number of times on the corporate side. Something has gone very wrong. People want answers. A good manager assesses the situation and responds back "Look, we know what is going on and all hands are on deck to fix it. Making a powerpoint is not fixing it. We'll do a proper write up for next week but we can either have So and So fix it or report on it."
Obviously that stops being an option as you begin impacting investors. But that is when it becomes a trade off of "Okay, Jen barely understands what Roy and Moss are doing. But she can say something that hopefully won't be too wrong and then apologize and give a correction tomorrow"
But people very much don't seem to understand how small this project is. Spend time with passion projects and "open source" projects that AREN'T on the scale of a small-medium sized company and you understand that standards are going to be lower because people have day jobs and so forth.
I mean, there is a reason reddit hired so many people over the years. And if you are going to jump down the throats of people who prioritize fixing an issue and counting on "active members" to notify users over writing up the reports that many of those users won't even look at? You want a production quality piece of software. That means Reddit or Threads or Bluesky.
I mean, there is a reason reddit hired so many people over the years. And if you are going to jump down the throats of people who prioritize fixing an issue and counting on โactive membersโ to notify users over writing up the reports that many of those users wonโt even look at? You want a production quality piece of software. That means Reddit or Threads or Bluesky.
Why are you getting so defensive? The only throat getting jumped down is mine, by you. I'm expressing my opinion of gaps in the communication of the project and how I think it can be improved. In a conversation thread on selfhosted no less. I'm not out in !lemmy@lemmy.ml bitching them out, submitting issues, or otherwise harassing the devs. Pointing out a gap and suggesting solutions is neither shitposting nor jumping down someone's throat.
Iโve been through similar a decent number of times on the corporate side. Something has gone very wrong. People want answers. A good manager assesses the situation and responds back โLook, we know what is going on and all hands are on deck to fix it. Making a powerpoint is not fixing it. Weโll do a proper write up for next week but we can either have So and So fix it or report on it.โ
I think you're the one confusing this with a large corporate project. Not me. There's no managers here, there's no powerpoints, and at no point have I asked for a detailed write-up. I asked for someone on the project, who isn't actively working on identifying and coding the fix, to be the "point man". Post a simple sticky at the top of !lemmy@lemmy.ml xposted to !lemmy_support@lemmy.ml that indicates there's a problem, they're aware of it, and a fix it being worked on. Once mitigations are identified or fixes are published, update the post with that. Ideally, a github security incident would be also be published with the same info so people not watching lemmy at the moment can notified via that channel.
But people very much donโt seem to understand how small this project is. Spend time with passion projects and โopen sourceโ projects that ARENโT on the scale of a small-medium sized company and you understand that standards are going to be lower because people have day jobs and so forth.
I get it. I have pretty low standards. I'm just saying that a consistent communication strategy going forward for this project would be beneficial.
I'm with you. I figured out through various comments that I should update my UI to
0.18.2-rc.1, and also run an update statement on my database to fix the modlog. Only after that did I find the matrix channel. Eventually I also found !lemmy_admin@lemmy.ml which is great, but the only thread there on this issue doesn't even mention updating the UI. I think if we can get to the point where critical information that admins need to know is consistently posted in one place, it'll make everybody's life easier. I don't think that's too much to ask.
whilst I differ somewhat on sharing information on the exploit - knowing something about what was going on allowed some instance admins to take evasive steps - I agree with you completely that there could be a better channel for coordinating communication - I imagine a lot of the discussion went on via Matrix - under the circumstances the response wasn't so bad given the complete lack of formal organization but yes, it definitely could be improved - you sound quite well-versed in how to handle security/critical incidents. Maybe consider contacting the devs and offering them some help in this area?
I don't think I'm asking for a lot. A post on !lemmy@lemmy.ml xposted to !lemmy_support@lemmy.ml that gets pinned to the top. Edit the post when relevant information comes out. Release a security advisory on github as soon as you have enough info to warrant one and keep it up-to-date as well.
I'm not asking for the troubleshooting to happen out in the open.
you sound quite well-versed in how to handle security/critical incidents. Maybe consider contacting the devs and offering them some help in this area?
I know enough. I'm certainly not an infosec guy I'm just a sysadmin who's been doing this long enough to know what should be done. At least partly due to this there's currently 400 open issues just in lemmy-ui on github. Right now I think the best most of us can do is wait for the dust to settle.
Right, but Lemmy.ml is really just one of a thousand plus instances. We need something instance independent or a way to propagate info that doesn't rely on any single failure points, or Lemmy as the communication channel. What happens when lemmy.ml is down, or if no instances are able to post due to concerted DoS?
It's impossible to stop anyone randomly posting stuff on Lemmy. Attackers can post misinformation as well, especially if they compromise admin accounts. Who are we gonna trust in the midst of the next incident? The account posting most prolifically about the UI exploit in progress was using a burner account that had just been created to post about it. I'm sure there were good reasons for wanting to be anonymous when discussing the work of unknown malicious actors, but it made me think twice about what was being posted at the time.
I think the authoritative source should be the GitHub repo. A security advisory should be posted there with references to outside resources as necessary.
checks all the boxes - authoritative (authenticated user accounts), central location, not on fediverse, already relatively well-known by lemmy users and provides visibility to remediation. It's a good idea.
Your typical dev is not a technical writer, and shouldn't be doing the proper write-up.
If you feel (and it seems you do) that this skill is missing from the Lemmy team, perhaps you should volunteer some time.
If this was not a zero day being actively exploited then you would be 100% correct. As it is currently being exploited and a fix is available, visibility is significantly more important than anything else or else the long tail of upgrades is going to be a lot longer.
Keep in mind a list of federated instances and their version is available at the bottom of every lemmy instance (at /instances), so this is a really easy chain to follow and try to exploit.
The discovery was largely discussed in the lemmy-dev Matrix channel, fixes published on github, and also discussed on a dozen alternate lemmy servers. This is not an issue you can really keep quiet any longer, so ideally now you move along to the shout it from the mountaintop stage.
FYI for anyone looking to deface more instances, That list is only updated every 24 hours. Depending on when it last run on your home instance, the info could be out of date.
I think it also only shows backend version, not frontend, so it won't reflect this fix.
OK, as long as all the well-meaning people stop discussing it, nobody will ever find out about it.
Son, this is not it.
Where is this Matrix Channel? Is it private? How can I get access as an instance admin?
Presumably that channel (I hadn't seen it there) and https://matrix.to/#/#lemmydev:matrix.org, which is actually where I was watching for the fix.
It isn't private, I think there's a link on the girhub
If the only criteria to be in a private channel for admins is being an admin, thereโs no use making it private. ;) Unless your just looking to filter out bad actors who donโt want to take 5 min and 5$ to make an instance.
I have an account that's in there without being an administrator.
Yep, it's public on github.
I was actually speaking of the matrix channel
This is my take on it. I am running Lemmy in a docker using the dessalines image. I hope that there will be an update come this afternoon.
There's already an update available, but it's for
lemmy-uinotlemmy. Just update the tag to0.18.2-rc.1and you'll have this fix.This is probably a dumb question but I used the Ansible install for Lemmy and just did a git pull and --become again but UI wasn't updated so I assume 0.18.2 isn't in release yet (which is fine) but is there documentation on updating UI? I see where it's showing in the docker-compose.yml file but I am uncertain what to do after changing it there (or if that's the right place to change it).
Yep, that's the plan! Thanks for letting me know. Lemmy is awesome and I am having so much fun with it. I expect it only to get better as the days and weeks progress.
According to https://github.com/LemmyNet/lemmy/commits/main, the bug was fixed with https://github.com/LemmyNet/lemmy/commit/00f9f79a44887869dcdc3fe5bd1dabbbdc080cec and is part of release 0.18.1, right? I usually wouldnยดt recommend to install the release candidate, except for testing, but since this is still 0.X anyway...
This is probably a dumb question but I used the Ansible install for Lemmy and just did a git pull and --become again but UI wasnโt updated so I assume 0.18.2 isnโt in release yet (which is fine) but is there documentation on updating UI? I see where itโs showing in the docker-compose.yml file but I am uncertain what to do after changing it there (or if thatโs the right place to change it).
There is already an update. 0.18.2-rc1
You can apply it now.
I will have to wait until I can get home from work. Work does deep packet inspection and blocks SSH. I've tried doing SSH on port 993, one I know for a fact is open because I get my email that way on my phone and I still get a connection refused. Bunch of fascists!
Woah seriously
Yep, seriously!
You could try using a VPN or some other kind of proxy which wraps your SSH traffic to prevent packet inspection. Then it should look like normal UDP traffic ;)
They block UDP traffic too. I tried some things like that.
Given how strict it is, I assume your company implements some sort of certification such as ISO27001 and really stick their gun on it? So, can you like, not using your company's wifi on your phone if it's heavily monitored? Or is the cell reception poor at your office?
I use my employer's guest wifi. Right now I can't afford adding the hotspot to my plan.
No need to enable hotspot on your phone. Just install Termux from f-droid if you're on android, or Prompt if you're on iOS, and use SSH directly from your phone.
This is probably a dumb question but I used the Ansible install for Lemmy and just did a git pull and --become again but UI wasnโt updated so I assume 0.18.2 isnโt in release yet (which is fine) but is there documentation on updating UI? I see where itโs showing in the docker-compose.yml file but I am uncertain what to do after changing it there (or if thatโs the right place to change it).