Major IT outage affecting banks, airlines, media outlets across the world

Reading into the updates some more... I'm starting to think this might just destroy CloudStrike as a company altogether. Between the mountain of lawsuits almost certainly incoming and the total destruction of any public trust in the company, I don't see how they survive this. Just absolutely catastrophic on all fronts.

The amount of servers running Windows out there is depressing to me

>Make a kernel-level antivirus
>Make it proprietary
>Don't test updates... for some reason??

never do updates on a Friday.

Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao

Here's the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching "C-00000291*.sys" 4)Boot the system normally

This is going to be a Big Deal for a whole lot of people. I don't know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.

CrowdStrike: It's Friday, let's throw it over the wall to production. See you all on Monday!

Wow, I didn't realize CrowdStrike was widespread enough to be a single point of failure for so much infrastructure. Lot of airports and hospitals offline.

The Federal Aviation Administration (FAA) imposed the global ground stop for airlines including United, Delta, American, and Frontier.

Flights grounded in the US.

The System is Down

Ironic. They did what they are there to protect against. Fucking up everyone's shit

An offline server is a secure server!

The thought of a local computer being unable to boot because some remote server somewhere is unavailable makes me laugh and sad at the same time.

Clownstrike

https://www.theregister.com/ has a series of articles on what's going on technically.

Latest advice...

There is a faulty channel file, so not quite an update. There is a workaround...

1. Boot Windows into Safe Mode or WRE.

2. Go to C:\Windows\System32\drivers\CrowdStrike

3. Locate and delete file matching "C-00000291*.sys"

4. Boot normally.

Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!

Yep, this is the stupid timeline. Y2K happening to to the nuances of calendar systems might have sounded dumb at the time, but it doesn't now. Y2K happening because of some unknown contractor's YOLO Friday update definitely is.

I'm so exhausted... This is madness. As a Linux user I've busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.

My work PC is affected. Nice!

A few years ago when my org got the ask to deploy the CS agent in linux production servers and I also saw it getting deployed in thousands of windows and mac desktops all across, the first thought that came to mind was "massive single point of failure and security threat", as we were putting all the trust in a single relatively small company that will (has?) become the favorite target of all the bad actors across the planet. How long before it gets into trouble, either because if it's own doing or due to others?

I guess that we now know

My dad needed a CT scan this evening and the local ER's system for reading the images was down. So they sent him via ambulance to a different hospital 40 miles away. Now I'm reading tonight that CrowdStrike may be to blame.

crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It's not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming...

We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **

Honestly kind of excited for the company blogs to start spitting out their disaster recovery crisis management stories.

I mean - this is just a giant test of disaster recovery crisis management plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

If a company uses IPMI (Called Branded AMT and sometimes vPro by Intel), and their network is intact/the devices are on their network, they ought to be able to remotely address this.
But that’s obviously predicated on them having already deployed/configured the tools.

Been at work since 5AM... finally finished deleting the C-00000291*.sys file in CrowdStrike directory.

182 machines total. Thankfully the process in of itself takes about 2-3 minutes. For virtual machines, it's a bit of a pain, at least in this org.

lmao I feel kinda bad for those companies that have 10k+ endpoints to do this to. Eff... that. Lot's of immediate short term contract hires for that, I imagine.

lol

too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404

My favourite thing has been watching sky news (UK) operate without graphics, trailers, adverts or autocue. Back to basics.

Linux and Mac just got free advertisment.

Annoyingly, my laptop seems to be working perfectly.

One possible fix is to delete a particular file while booting in safe mode. But then they'll need to fix each system manually. My company encrypts the disks as well so it's going to be a even bigger pain (for them). I'm just happy my weekend started early.

AWS No!!!

Oh wait it's not them for once.

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas.

Never trust a texan

Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half...

oh joy. can’t wait to have to fix this for all of our clients today…

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I'll be working from home for a few days before anything is fixed.

Stop running production services on M$. There is a better backend OS.

This is a better article. It's a CrowdStrike issue with an update (security software)

I see a lot of hate ITT on kernel-level EDRs, which I wouldn't say they deserve. Sure, for your own use, an AV is sufficient and you don't need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.

The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.

Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.

I'm not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn't warranted here.

Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.

No one bother to test before deploying to all machines? Nice move.

I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.

Irrelevant but I keep reading "crowd strike" as "counter strike" and it's really messing with me

Huh, so that's why the office couldn't order pizza last night lmfao

This is proof you shouldn't invest everything in one technology. I won't say everyone should change to Linux because it isn't immune to this, but we need to push companies to support several OS

I picked the right week to be on PTO hahaha

If these affected systems are boot looping, how will they be fixed? Reinstall?

Apparently at work "some servers are experiencing problems". Sadly, none of the ones I need to use :(

Everyone is assuming it’s some intern pushing a release out accidentally or a lack of QA but Microsoft also pushed out July security updates that have been causing bsods on the 9th(?). These aren’t optional either.

What’s the likelihood that the CS file was tested on devices that hadn’t got the latest windows security update and it was an unholy union of both those things that’s caused this meltdown. The timelines do potentially line up when you consider your average agile delivery cadence.

Servers on Windows? Even domain controllers can be Linux-based.

A lot of people I work with were affected, I wasn't one of them. I had assumed it was because I put my machine to sleep yesterday (and every other day this week) and just woke it up after booting it. I assumed it was an on startup thing and that's why I didn't have it.

Our IT provider already broke EVERYTHING earlier this month when they remote installed" Nexthink Collector" which forced a 30+ minute CHKDSK on every boot for EVERYONE, until they rolled out a fix (which they were at least able to do remotely), and I didn't want to have to deal with that the week before I go in leave.

But it sounds like it even happened to running systems so now I don't know why I wasn't affected, unless it's a windows 10 only thing?

Our IT have had some grief lately, but at least they specified Intel 12th gen on our latest CAD machines, rather than 13th or 14th, so they've got at least one win.

don't rely on one desktop OS too much. diversity is the best.

So that's why my work laptop is down for the count today. I'm even getting that same error as the thumbnail picture

Meanwhile Kaspersky: *thinks if so incompetent people can even make antivirus at all*

Xfinity H&I network it down so I can't watch Star Trek. I get an error msg connection failure. Other channels work though.

Interesting day

I think we're getting a lot of pictures for !pbsod@lemmy.ohaa.xyz

I legit have never been more happy to be unemployed.

This is the best summary I could come up with:

There are reports of IT outages affecting major institutions in Australia and internationally.

The ABC is experiencing a major network outage, along with several other media outlets.

Crowd-sourced website Downdetector is listing outages for Foxtel, National Australia Bank and Bendigo Bank.

Follow our live blog as we bring you the latest updates.

The original article contains 52 words, the summary contains 52 words. Saved 0%. I'm a bot and I'm open source!

A bunch of shitty sysadmins/cybersec people just learned why you don't blindly deploy new updates to production without testing them first.

I've used Crowdstrike before. It has support for deploying version N to a pilot group, N-1 to the test environment and N-2 to production.

Permanently Deleted

It's a fair point but I would rather diversify and also use something that is open / less opaque

Some intern is getting their ass beat right now, never release into prod without extensive test.

Buy the dip!

Webroot had something similar ish earlier this year. Such a pain.

Permanently Deleted

Windows moment 🤣

Why is no one blaming Microsoft? It's their non resilient OS that crashed.

I'm used to IT doing a lot of their work on the weekends as to not impact operations.

This is why you create restore points if using windows.

Permanently Deleted

Bahaha 😂😂 continue using proprietary software, that's all you are going to get in addition to privacy issues... Switch to Linux.

Good ol microsloth

play stupid games win stupid prizes

Why do people run windows servers when Linux exists, it’s literally a no brainer.

This is a a ruse to make Work From Home end.

All IT people should go on general strike now.

It's Russia, or Iran or China or even our "ally" Saudi Arabia. So really, it's time to reset the clock to pre 1989. Cut Russia and China off completely, no investment, no internet, no students no tourist nothing. These people mean and are continually doing us harm and we still plod along and some unscrupulous types become agents for personal profit. Enough.

best day ever. the digitards get a wakeup call. how often have been lectured by imbeciles how great whatever dumbo closed source is. "i need photoshop", "windows powershell and i get work done", "azure and onedrive and teams...best shit ever", " go use NT, nobody will use a GNU".

yeah well, i hope every windows user would be kept of the interwebs for a year and mac users just burn in hell right away. lazy scum that justifies shitting on society for their own comfort. while everyone needs a drivers license, dumb fucking parents give tiktok to their kids...idiocracy will have a great election this winter.