How to stagger automated upgrade?

Cron with the -y option on apt commands.

My suggestion is to use system management tools like Foreman. It has a "content views" mechanism that can do more or less what you want. There's a bunch of other tools like that along the lines of Uyuni. Of course, those tools have a lot of features, so it might be overkill for your case, but a lot of those features will probably end up useful anyway if you have that many hosts.

With the way Debian/Ubuntu APT repos are set up, if you take a copy of /dists/$DISTRO_VERSION as downloaded from a mirror at any given moment and serve it to a particular server, that's going to end up with apt update && apt upgrade installing those identical versions, provided that the actual package files in /pool are still available. You can set up caching proxies for that.

I remember my DIY hodgepodge a decade ago ultimately just being a daily cronjob that pulls in the current distro (let's say bookworm) and their associated -updates and -security repos from an upstream rsync-capable mirror, then after checking a killswitch and making sure things aren't currently on fire, it does rsync -rva tier2 tier3; rsync -rva tier1 tier2; rsync -rva upstream/bookworm tier1. Machines are configured to pull and update from tier1 (first 20%)/tier2 (second 20%)/tier3 (rest) appropriately on a regular basis. The files in /pool were served by apt-cacher-ng, but I don't know if that's still the cool option nowadays (you will need some kind of local caching for those as old files may disappear without notice).

A cron job that runs when you want it to instead of the unattended updates metapackage.

To effectively manage and stagger automated upgrades across multiple groups of Ubuntu servers, scheduling upgrades on specific days for different server groups offers a structured and reliable method. This approach ensures that upgrades are rolled out in a controlled manner, reducing the risk of potential disruptions.

Here's an example Ansible playbook that illustrates how to set this up. It installs unattended-upgrades and configures systemd timers to manage upgrades on specific weekdays for three distinct groups of servers.

  ---
  - hosts: all
    become: yes
    vars:
      unattended_upgrade_groups:
        - name: staging_batch1
          schedule: "Mon *-*-* 02:00:00"  # Updates on Monday
        - name: staging_batch2
          schedule: "Wed *-*-* 02:00:00"  # Updates on Wednesday
        - name: staging_batch3
          schedule: "Fri *-*-* 02:00:00"  # Updates on Friday

    tasks:
      - name: Install unattended-upgrades
        apt:
          name: unattended-upgrades
          state: present

      - name: Disable automatic updates to control manually
        copy:
          dest: /etc/apt/apt.conf.d/20auto-upgrades
          content: |
            APT::Periodic::Update-Package-Lists "1";
            APT::Periodic::Download-Upgradeable-Packages "0";
            APT::Periodic::AutocleanInterval "7";
            APT::Periodic::Unattended-Upgrade "0";
          mode: '0644'

      - name: Setup systemd service and timer for each group
        loop: "{{ unattended_upgrade_groups }}"
        block:
          - name: Create systemd service for unattended-upgrades for {{ item.name }}
            copy:
              dest: "/etc/systemd/system/unattended-upgrades-{{ item.name }}.service"
              content: |
                [Unit]
                Description=Run unattended upgrades for {{ item.name }}

                [Service]
                Type=oneshot
                ExecStart=/usr/bin/unattended-upgrade
              mode: '0644'

          - name: Create systemd timer for {{ item.name }}
            copy:
              dest: "/etc/systemd/system/unattended-upgrades-{{ item.name }}.timer"
              content: |
                [Unit]
                Description=Timer for unattended upgrades on {{ item.schedule }} for {{ item.name }}

                [Timer]
                OnCalendar={{ item.schedule }}
                Persistent=true

                [Install]
                WantedBy=timers.target
              mode: '0644'

          - name: Enable the timer for {{ item.name }}
            systemd:
              name: "unattended-upgrades-{{ item.name }}.timer"
              enabled: yes

          - name: Start the timer for {{ item.name }}
            systemd:
              name: "unattended-upgrades-{{ item.name }}.timer"
              state: started

Ubuntu only does security updates, no? So that seems like a bad idea.

If you still want to do that, I guess you'd probably need to run your own package mirror, update that on Monday, and then point all the machines to use that in the sources.list and run unattended-upgrades on different days of the week.

Maybe you could switch to an image based distro which is easy to roll back and won't boot into a broken image.

What you're asking for is a CI/CD pipeline that deploys a set of OS updates as a set revision. I don't the details on how to do it but that's the concept you're asking for.

In an ideal world, there should be 3 separated environments of the same app/service:
devel → staging → production.

Devel = playground, stagging = near identical to the production.

So you can test the updates before fixing production.

It's called a staging environment. You have servers you apply changes to first before going to production.

I assume you mean this for home though, so take a small number of your machines and have them run unattended upgrades daily, and set whatever you're worried about to only run them every few weeks or something.