All of Japan's Toyota Assembly Plants Shut Down for a Day Because Their Server Ran Out of Disk Space

I haven't read the article because documentation is overhead but I'm guessing the real reason is because the guy who kept saying they needed to add more storage was repeatedly told to calm down and stop overreacting.

Sysadmin pro tip: Keep a 1-10GB file of random data named DELETEME on your data drives. Then if this happens you can get some quick breathing room to fix things.

Also, set up alerts for disk space.

This happens. Recently we had a problem in production where our database grew by a factor of 10 in just a few minutes due to a replication glitch. Of course it took down the whole application as we ran out of space.

Some things just happen and all head room and monitoring cannot save you if things go seriously wrong. You cannot prepare for everything in life and IT I guess. It is part of the job.

I blame lean philosophy. Keeping spare parts and redundancy is expensive so definitely don't do it...which is just rolling the dice until it comes up snake eyes and your plant shuts down.

It's the "save 5% yearly and stop trying to avoid a daily 5% chance of disaster"

Over prepared is silly, but so is under prepared.

They were under prepared.

There's some irony to every tech company modeling their pipeline off Toyota's Kanban system...

Only for Toyota to completely fuck up their tech by running out of disk space for their system to exist on. Looks like someone should have put "Buy more hard drives" to the board.

This is the best summary I could come up with:

TOKYO, Sept 6 (Reuters) - A malfunction that shut down all of Toyota Motor's (7203.T) assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.

The system halt followed an error due to insufficient disk space on some of the servers and was not caused by a cyberattack, the world's largest automaker by sales said in a statement on Wednesday.

"The system was restored after the data was transferred to a server with a larger capacity," Toyota said.

The issue occurred following regular maintenance work on the servers, the company said, adding that it would review its maintenance procedures.

Two people with knowledge of the matter had told Reuters the malfunction occurred during an update of the automaker's parts ordering system.

Toyota restarted operations at its assembly plants in its home market on Wednesday last week, a day after the malfunction occurred.

The original article contains 159 words, the summary contains 159 words. Saved 0%. I'm a bot and I'm open source!

Idiots, they ought to have switched to tabs for indenting. Everybody knows that.

This is a fun read in the wake of learning about all the personal data car manufacturers have been collecting

Free disk space is just inventory and therefor wasteful.

Kanban

Was this that full shutdown everyone thought was going to be malware?

The worst malware of all, unsupervised junior sysadmins.

Just delete some p0rn

And that's why I have a weekly cronjob in my server to call BleachBit, remove cycled logs, and compress the images on my storage. Having to make do with a rather limited VPS for years taught me to be resourceful with what I had

Storage has never been cheaper.

There's going to be a seppuku session in somebody's IT department.

Someone messed up log rotation and the whole /var went ro.

well well, someone will be fired

Search for non system files older than 2020 and lmza zip archive that shit stat!

Free space is a wasted disk space.

Server less DBs ftw