The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center
The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center
Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…
Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…
I’m shocked that the data center required retinal scans but that the employee with access could then just hold the door and let him and others in.
I used to work at a data center with lots of security. To get into the area with the servers you had to go through a man trap. It was a room a little larger than a telephone booth with automatic doors on both sides. To open the first door you needed a physical card key. Once inside the door closed, then to open the inner door you needed to both enter a PIN and have your hand scanned in a biometric scanner. Only after all that could you get inside. The booth also weighed you, and if your weight was off by a certain amount after your last pass through then it wouldn’t let you in. That was to prevent somebody from piggybacking with you.
They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved.
Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. “I’ve never loaded a semi before,” James admitted.
The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost.
The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. ... So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there. “I can’t believe it worked,” James says.
LMAO who's this James guy and why does he understand Musk so thoroughly like his own spouse?
It's a total lack of the concept of scale. I've moved servers like this, but when decommissioning them. Things are different when running a corporate data center than when moving a home lab. He doesn't grasp the difference, because he doesn't understand the scale.
Running and moving one computer is different from 150k of them. Hooking them back up the the network without a plan or documentation must have been a challenge.
Elon Musk is a privileged manchild who never grew out of his teenager phase, throwing around his inherited wealth like the kid from Blank Check and throwing temper tantrums anytime someone calls him out on his bullshit. Any claims to success he may have had been entirely in spite of him, not because of him. He doesn’t have any fucking idea what he’s doing and if any one of you or I failed even a fraction as much as Musk had, we’d have all been fired ten times over.
So glad I ended up not working at Tesla
His most valuable lieutenants at Tesla and SpaceX had learned ways to deflect his bad ideas and drip-feed him unwelcome information, but the legacy employees at X didn’t know how to handle him.
This is fucking insane and would drive me nuts.
This guy really is a Space Karen
The CEO then told him that some of the floors could not handle more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk replied that the servers had four wheels, so the pressure at any one point was only 500 pounds. “The dude is not very good at math,” Musk told the musketeers.
This guy is considered to be a genius? This guy is a fucking billionaire?
I’m dead.
I do not think it is a coincidence that Tesla has recently released the updated Model 3 to some decently positive reviews. I think that is in no small part to Musk being so distracted by Twitter that he hasn't been able to fuck up things over at Tesla in a while.
OMG OK that's it. Tesla cars are now out of the question for me and if I ever get the chance to ride on a SpaceX ship (not very likely) I think I'd decline. Totally different companies ofc but the same master "mind" behind.
This guy represents everything that you do not want to see in a CEO.
NO THANK YOU
He got lucky nothing disappeared.
At a previous work place they rounded up a few employees to move stuff from one office to the new office. That ended up with a few monitors less than they started with. They couldn't ask who took it because they never wrote down who they rounded up for the move.
And that's how companies end up with a bunch of silly regulations how you're not allowed to move any hardware to the next room
Note the pattern: a willingness to ignore the details of what could go wrong, YOLO it and just test it out, and the assumption that if nothing goes wrong when you do that, it means that everything is fine and nothing else could possibly go wrong.
Did anyone else reading this bit immediately think of that other rich idiot that died in his ridiculous submarine?
One of the richest people in the world, everyone.
This is why I wouldn't trust a thing that comes out of his mouth. He lies, he says really stupid shit and then he gives people an ultimatum to turn his stupid shit into reality or get fired. Safety, security and reality be damned. If you've ever wondered why people end up dying in fiery crashes because of "autopilot", or "full self drive", this is why.
That reads like something out of The Hitchhiker's Guide to the Galaxy.
Zaphod Beeblebrox, because of "an accident with a contraceptive and a time machine", father, grandfather, and great-grandfather are actually his direct descendants.
Musk must have been the offspring of an unspeakable accident between Zaphod and one of those Sacramento racks.
What a giant twat muskrat is.
I've heard of chaos testing going as far as walking into a DC and pulling plugs until alarms start going off, but this is just awful all around.
A fractal Rube Goldberg machine
Welcome to software engineering Elon.
Hahahaha, absolutely in character for that fucking moron
Well that was a crazy read! I really feel for the workers who had to witness all that nonsense.
This is some Antwan Soonami Studios shit. Elon is a parody of a leader.
At the risk of sounding like an apologist for this prat, the frustration at being told it will take months to move systems is understandable. Also, the idiot developers who hard-coded the data center location deserve to be fired. Data center floor tiles can be removed easily with a flat blade as a lever instead of using suction cups.
Obviously a coke-fuelled man-child doing it in the middle of the night is ridiculous but if you have unlimited resources you can move any number of servers in a few days, easily. In some ways it's impressive that he was able to pull this off on a whim without a catastrophe (the DeSantis fiasco notwithstanding). It definitely should not have been suggested by a competent data center engineer that it would take months to move anything if the CEO wants it in weeks. Even though he's an ass, I don't blame him for being annoyed about that.
Every great genius inventor and businessman can be a little eccentric. Remember that before you decide to call them mean names like "abuser", "megalomaniac" etc. It's actually quirky and endearing once you factor in that he's a genius inventor and businessman (I'm also one of the misunderstood people in the same category).
I love this. I'm a former IT engineer/CTO turned renegade entrepreneur, so this story tickles both of my feet.
Yeah, any reasonable person would know this idea to move the servers without a plan was ridiculous.
Yet as a roll-up-the-sleeves entrepreneur your entire job is to fucking destroy the red tape that is put up in front of you constantly. Or else you and everyone who works for you is out of a job. Of course there will be problems, but that's why you have smart people who can sort it out afterwards faster than they can preplan for it.
And a lot of really smart people make "doing it the right way" a religion, so when the cash is going to run out shortly, well, sometimes the big guy needs to just roll the dice.
I guess I'm also batshit crazy.
It's Twitter. Who cares if people can't tweet for an hour.
I'm with Elon on this, don't overcomplicate the closing of a data center.
That manager, when asked to do it in 90 days, if s/he was competent should have said: I'll do it, but you'll have to accept a downtime risk.
(But aside from this entertaining story, I do think Elon lost his shine. Wasting $40B on twitter and sabotaging Ukraine while simping for Putin and Trump and not paying taxes... yeah, get rekt Elon).
I really dislike Musk, but I find it hard to criticize this when it generally worked.
The platform formerly known as Twitter is still running, and there’s no more $100 million/year data center.
6-9 months would have meant $50-75 million dollars. I don’t know what the outages and re-engineering ended up costing them, but that’s a ton of money.