A Data Hoarder, I Am Not
A Data Hoarder, I Am Not
So I’ve been consolidating all of my storage and removing all the duplicates and junk files.
In actual physical storage, this was spread across 12TB worth of hard drives, all partially full.
After everything was said and done, I’m using 1.3TB of space if you don’t include games. ¯\_(ツ)_/¯
This is stuff dating back to 2015. Sometimes it’s actually worth it to just clean up your junk files.
WHAT IF I NEED IT YO
There were some folders that were 50+GB that had like 3 txt files and a song worth saving in them. In some ways, I was impressed.
Now Jesse, we've talked about this...
On what types of junk files did you save most of the space?
I had a bad habit whenever I would go to reinstall an OS to just copy the entire user folder into one of many places on my largest hard drive. I had at least 4 or 5 of these. So tons of it was cache files.
Storage is dirt cheap. Just add more. IT at work bugs out their eyes when I talk about adding more storage space. I have more at home than they do in the office. Lol.
I’ve been buying used 8TB HGST drives from eBay. Dirt cheap and haven’t had one fail yet.
I’ve always been one for moderation. Plus, the big issue was just being able to find all of it. Everything was so scattered and in some cases I had half my Steam library downloaded onto two or three of the hard drives, all outdated.
Now I know where it all is and can easily back it up, where before I only had one copy of a lot of my most important files.
This is actually what lead me to set up a software RAID - my family is primarily Windows and I didn't want to remember if the files were on D:, F:, G:, K:, etc. I'd rather have a root folder that's extendable.
I’m a big advocate of unraid servers. Mix And match any size of drives you have available into a single large NAS with protection against drive failures. You can use old pc hardware you might have lying around. It’s commercial software but you can demo it for free. It’s good enough that I own two full pro licenses.
Any good tools to deduplicate files? Got a bunch of images with different names, on different folders, sometimes with different resolutions, spread out across 10 external hard drives that I need to go through...
This was done by hand on my end. If I had needed to do it procedurally, I’m not sure what my approach would have been, but the direction I’d probably head in would be to look into finding duplicate MD5 hashes.
md5sum to get the low hanging fruit
Dupe Guru has been a useful GUI on windows. It is also cross platform.