We're back! Instance Updates - 06/23/2023

So Lemmy 0.18.0 dropped today and I immediately jumped on the bandwagon and updated. That was a mistake. I did the update during my lunch hour, quickly checked to make sure everything was up (it was, at the time) and came back a few hours later to everything imploding.

As far as I can tell, things broke after the DB migrations occurred. Pict-rs was suddenly dumping stack traces on any attempt to load an image, and then at some point the DB itself fell over and started spewing duplicate key errors in an endless loop.

I wound up fiddling with container versions in docker-compose.yml until finding a fix that restored the instance. We are downgraded back to the previous pict-rs release (0.3.1), while Lemmy and Lemmy-UI are both at 0.18.0. I'm still trying to figure out what exactly went wrong so I can submit a bug report on Github.

Going forward, I will plan updates more carefully. We will have planned maintenance windows posted at least a few days in advance, and I may look into migrating the instance to my Kubernetes cluster so we can do a rolling deployment, and leave the existing pods up until everything is passing checks. In the meantime, I'm spinning up a sandbox Lemmy instance and will use that to validate upgrades before hitting this instance.