You just finished setting up all your services and it works fine - how do you now prepare for eventual drive failure?

There's lots of very good approaches in the comments.

But I'd like to play the devil's advocate: how many of you have actually recovered from a disaster that way? Ideally as a test, of course.

A backup system that has never done a restore operations must be assumed to be broken. similar logic should be applied to disaster recovery.

And no: I use Ansible/Docker combined approach that I'm reasonably sure could quite easily recover most stuff, but I've not yet fully rebuilt from just that yet.

I've had a complete drive failure twice within the last year (really old hardware) and my ansible + docker + backup made it really easy to recover from. I got new hardware and was back up and running within a few hours.

All of your services setup should be automated (through docker-compose or ansible or whatever) and all your configuration data should be backed up. This should make it easy to migrate services from one machine to another, and also to recover from a disaster.

Pre.....pare....? What's that? Some sorta fruit?

My configuration and deployment is managed entirely via an Ansible playbook repository. In case of absolute disaster, I just have to redeploy the playbook. I do run all my stuff on top of mirrored drives so a single failure isn't disastrous if I replace the drive quickly enough.

For when that's not enough, the data itself is backed up hourly (via ZFS snapshots) to a spare pair of drives and nightly to S3 buckets in the cloud (via restic). Everything automated with systemd timers and some scripts. The configuration for these backups is part of the playbooks of course. I test the backups every 6 months by trying to reproduce all the services in a test VM. This has identified issues with my restoration procedure (mostly due to potential UID mismatches).

And yes, I have once been forced to reinstall from scratch and I managed to do that rather quickly through a combination of playbooks and well tested backups.

I rsync my root and everything under it to a NAS, will hopefully save my data. I wrote some scripts manually to do that.

I think the next best thing to do is to doco your setup as mich as possible. Either by typed up notes, or ansible/packer/whatever, any documentation is better than nothing if you have to rebuild.

Radical suggestion:

• Once a year you buy a hard drive that can handle all of your data.
• rsync everything to it
• unplug it, put it back in cold storage

Infrastructure as code/config as code.

The configurations of all the actual machines is managed by Puppet, with all its configs in a git repo. All the actual applications are deployed on top of Kubernetes, with all the configurations managed by helmfile and also tracked in git. I don't set anything up - I describe how I want things configured, and the tools do the actual work.

There is a "cold start" issue in my scheme - puppet requires a server component that runs on Kubernetes but I can't deploy onto kubernetes until the host machines have had their puppet manifests applied, but at that point I can just read the code and do enough of the config by hand to bootstrap everything up from scratch if I have to

• Back everything up
• rm -rf /
• Now rebuild.

Congratulations, you now know what’s required. :-P

1. Most systems are provisioned in proxmox with terraform.
2. Configuration and setup is handled via ansible playbooks after the server is available. 2.a) Do NOT make changes on the server without updating your ansible scripts - except during troubleshooting. 2.b) Once troubleshooting is done delete and re-create the VM from scratch using only scripts to ensure it works.
3. VM storage is considered to be ephemeral. All long-term data/config that can't be re-created with ansible is either stored on an NFS server with a RAID5 dive configuration or backed up to that same file-server using rsnapshot.
4. NFS server is backed-up nightly to backblaze using duplicacy.
5. Any other non-VM systems like personal laptops and the like are backed up nightly to the file-server using rsnapshot. Those snapshots are then backed up to backblaze using duplicacy.

I have all my configuration as Ansible and Terraform code, so everything can be destroyed and recreated with no effort.

When it comes to the data, I made some bash script to copy, compress, encrypt and upload them encrypted. Not sure if this is the best but it is how I'm dealing with it right now.

• Install Debian stable with the ssh server included.
• Keep a list of the packages that were installed after (there aren't many but still).
• All docker containers have their compose files and persistent user data on a RAID1 array.
• Have a backup running that rsyncs once a day /etc, /home/user and /mnt/array1/docker to another RAID1 to daily/, from daily/ once a week rsync to weekly/, from weekly/ once a monthb timestamped tarball to monthly/. Once a month I also bring out a HDD from the drawer and do a backup of monthly/ with Borg.

For recovery:

• Reinstall Debian + extra packages.
• Restore the docker compose and persistent files.
• Run docker compose on containers.

Note that some data may need additional handling, for example databases should be dumped not rsunced.

My docker containers are all configured via docker compose so I just tar the .yml files and the outside data volumes and backup that to an external drive.

For configs living in /etc you can also backup all of them but I guess its harder to remember what you modified and where so this is why you document your setup step by step.

Something nice and easy I use for personal documentations is mdbooks.

By using NixOS and tracking the config files with git

I actually run everything in VMs and have two hypervisors that sync everything to each other constantly, so I have hot failover capability. They also back up their live VMs to each other every day or week depending on the criticality of the VM. That way I also have some protection against OS issues or a wonky update.

Probably overkill for a self hosted setup but I’d rather spend money than time fixing shit because I’m lazy.

My server has a raid1 mdadm boot drive. And an 8 dive raid6 with zfs. It's been running for 14 years now. The only thing that I haven't replaced over it's lifetime is the chassis. In fact the proc let out the magic smoke a few weeks ago, after some new parts it's still going strong.

Permanently Deleted

My whole environment is in docker-compose which is "backed" to github.
My config/system drive is backed with veeam to one drive.
The backup is backed with rsync to another drive every week.

But: I only have a 1-drive NAS because I don't have the place for a proper PC with drive caddies and a commercial nas (synology, qnap) are not my jam because I'd need a transcoding capable gpu and those models are overpriced for what I need.
And with plain debian I get unlimited system updates (per distro release) and learn linux along the way.

Most of the docker services use mounted folders/files, which I usually store in the users home folder /home/username/Docker/servicename.

Now, my personal habit of choice is to have user folders on a separate drive and mount them into /home/username. Additionally, one can also mount /var/lib/docker this way. I also spin up all of these services with portainer. The benefit is, if the system breaks, I don't care that much, since everything is on a separate drive. In case of needing to re-setup everything again, I just spin up portainer again which does the rest.

However, this is not a backup, which should be done separately in one way or the other. But it's for sure safer than putting all the trust into one drive/sdcard etc.

I used to (over a span of about 4 years now) just rely on a RaidZ2 (ZFS) pool (faulted drive replacements never gave any issues) but I recently did an expansion of the array plus OS reinstall and only now am I starting to incorporate Docker containers into my workflows. The live data is in ~ and nightly rsynced onto the new larger RaidZ2 pool but there is also data on that pool which I've thus far never stored anywhere else.

So my answer to the question would be an off-site unraid install which is still in the works. This really will only be that. A catastophe insurance. I probably won't even rely on parity drives there in order to maximize space since I already have double parity on ZFS.

As far as reinstallation goes, I don't feel like restoring ~ and running docker compose for all the services again would be too much of a hassle.

I put all docker data in one directory (or rather, a btrfs subvolume) and both snapshot and back it up daily to multiple machines. docker-compose files are also kept in the same subvolume.

My latest server is NixOS, so I don't even bother backing up the root subvolume, since the actual config is tracked on git and replicated on multiple machines. If I want to reinstall, I can just install NixOS and deploy the config, then just copy over the docker subvolume, and rebuild the containers. Some of this could be automated further (nixos-anywhere and disko look promising for the actual OS install) but my systems don't typically break often enough for that to be a significant issue.

You can go even further and either just use nix for the services, or use nix to build containers themselves, but I have a working setup already and it's good enough, and I can easily switch to another distribution if issues start occurring in NixOS.

I run everything on a 2 node proxmox cluster with ZFS mirror volumes and replication of the VMs and CTs between them, run PBS with hourly snapshots, and sync that to multuple USB drives I swap off site.

The docker VM can be ZFS snapshotted before major updates so I can rollback.

Routine backups of the VM's and raid disk for the hypervisor running them. If the box hosting the backups went screwy there's a problem but with something like 20TB of space used copies off-box are a bit cumbersome. To that end I just manually copy the irreplaceable stuff to a separate external storage and wish the movies and stuff good luck.

It ends up with a situation though where I'd have to lose both the disks on the hypervisor and if that happened several disks on the NAS (12 disks in a ZFS pool with each vdev being a mirror pair) or for the whole pool to get screwed up to lose the VMs fully. Depending on the day I might lose up to a week of VM state though since they only do a full copy once a week.

carefully configured services on your rpi

I have a back up on an SD Card waiting for the day the SD Card fails. Slot it in and reboot

On my main server: I have my SSD RAID1 ZFS snapshots of my container appdata, VM VHDs and docker image, that is also backed up as a full backup once per night to the RAID10 array, then rsynced to the backup server which then is uploaded to the cloud.

The data on the RAID is backups, repos or media that I’ve deposited there for an extra copy it for serving via Plex/Jellyfin. I have extra copies of the data, and if I were to lose the array totally, I wouldn’t be pleased, but my personal pictures/videos wouldn’t be in danger.

I run two back up servers, which both upload to the cloud. One of which takes bare metal images of all my computers (sans servers bulk drives), the other which takes live folders.

This is more due to convenience so that I can pull a bare metal image to restore a device, or easily go find a file with versioning online if necessary on both accounts.

As a wise man said, you can never have too many backups.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

8 acronyms in this thread; the most compressed thread commented on today has 6 acronyms.

[Thread #287 for this sub, first seen 18th Nov 2023, 10:35] [FAQ] [Full list] [Contact] [Source code]

I use duplicati for docker containers. You just host it in docker and attach all the persistent volumes from the other containers to it, then you can set up backup jobs for each.

I use docker so don't really have to worry about reproducibility of the Services or configurations. Docker will fetch the right services and versions. I've documented the core configurations so I can set them back up relatively easily. Anything custom I haven't documented I'll just have to remember or find I need to reset up.

virtualize the machine with proxmox, use proxmox backup server, load vm on new system if you get catastrophic failure on the machine running the vm currently.

configs are backed up I can spin up a new container in minutes, I just accept the manual labor. It's probably a good thing to clean out the spiders and skeletons every now and then.

I eat a cyanide tablet. Drive won't fail on me if I'm dead. Taps temple