300TB of data. Dropbox and Google are dead to me. Next options. Cloud? Tape? NAS?
300TB of data. Dropbox and Google are dead to me. Next options. Cloud? Tape? NAS?
So I run a video production company. We have 300TB of archived projects (and growing daily).
Many years ago, our old solution for archiving was simply to dump old projects off onto an external drive, duplicate that, and have one drive at the office, one offsite elsewhere. This was ok, but not ideal. Relatively expensive per TB, and just a shit ton of physical drives.
A few years ago, we had an unlimited Google Drive and 1000/1000 fibre internet. So we moved to a system where we would drop a project onto an external drive, keep that offsite, and have a duplicate of it uploaded to Google Drive. This worked ok until we reached a hidden file number limit on Google Drive. Then they removed the unlimited sizing of Google Drive accounts completely. So that was a dead end.
So then we moved that system to Dropbox a couple of years ago, as they were offering an unlimited account. This was the perfect situation. Dropbox was feature rich, fast, integrated beautifully into finder/explorer and just a great solution all round. It meant it was easy to give clients access to old data directly if they needed, etc. Anyway, as you all know, that gravy train has come to an end recently, and we now have 12 months grace with out storage on there before we have to have this sorted back to another sytem.
Our options seem to be:
- Go back to our old system of duplicated external drives, with one living offsite. We'd need ~$7500AUD worth of new drives to duplicate what we currently have.
- Buy a couple of LTO-9 tape drives (2 offices in different cities) and keep one copy on an external drive and one copy on a tape archive. This would be ~$20000AUD of hardware upfront + media costs of ~$2000AUD (assuming we'd get maybe 30TB per tape on the 18TB raw LTO 9 tapes). So more expensive upfront but would maybe pay off eventually?
- Build a linustechtips style beast of a NAS. Raw drive cost would be similar to the external drives, but would have the advantage of being accessible remotely. Would then need to spend $5000-10000AUD on the actual hardware on top of the drives. Also have the problem of ever growing storage needs. This solution we could potentially not duplicate the data to external drives though and live with RAID as only form of redundancy...
- Another clour storage service? Anything fast and decent enough that comes at a reasonable cost?
Any advice here would be appreciated!
You're viewing a single thread.
A lot of people have suggested charging clients for long-term storage. I agree with that sentiment. If you go this route, you may be able to use cloud storage a la Dropbox/gDrive - which seems most convenient for you. Costs for consumer-facing cloud storage run roughly $10USD for 2TB. Expensive for hundreds of terabytes indefinitely, but if a single client needs access to (idk) 0.5 tb you could easily charge $30-50 a year to provide them a shared folder in google drive. Maybe more if you want redundancy against the cloud provider losing data.
For anything you need to actively use for work, a giant NAS is probably your best bet. Those YouTubers you’ve seen also use it as part of their team workflow, and maybe that’d also apply to you anyways. You should probably run a regular backup job of these to the other office or to AWS/backblaze. Should be manageable cost if you only need 10-20TB of data for active work.
For everything else… maybe tape if you really want to keep everything. A lot of big organizations seem to be moving away from tape towards networked spinning disk as the price drops. Seems mostly driven by tape being seen as a massive pain to use (not that I have personal experience with it) and expensive equipment. It’s really an organizational decision to directly quantify long term archival needs and value. Once you have a $/TB value to the business, see what fits your budget (could be nothing!) You could try Backblaze or AWS glacier but those get expensive and the cost is ongoing forever.
There are a whole bunch of niche and small-scale companies doing cloud data storage, but I don’t know how they’d get lower cost per byte stored over some big companies (lower margins? Slower speeds? Lower guarantees?). I’d be suspicious of them for mission-critical storage. It’s one thing for a home-user to use them to store their torrented movies, but it’s very different for a business. It could be worth it to just search around. Look at what’s supported as a target by whatever NAS software you use if that’s your route.