Scheduled downtime: Saturday 27th January @ 19:00 UTC
Scheduled downtime: Saturday 27th January @ 19:00 UTC
What's happening?
feddit.uk will be down for maintainence between 19:00 and 21:00 UTC (GMT in old money) on Saturday the 27th of January.
We are aiming to move the site between hosts, and restore missing images.
How long do you expect it to take?
The target time is 1 hour. The window is 2 hours. That's enough time for about 1 season of Peep Show, if you're bored.
What is the rollback plan?
If we're not happy with things by 20:30, we will bring the site up in it's previous state, and schedule a longer window in the future. We'll let you know if/when the decision is made.
Where can I see status updates?
We have a public channel on Matrix for people who wish to keep up with the status. It's part of the feddit.uk Matrix space which is there as a fallback in case of any downtime and to provide end-to-end encrypted communication where required.
edit: there is now a WhatsApp channel.
Why are you doing this?
As previously discussed in here, it's been quite the journey but, in summary: Tom, who set up the instance and was the only Admin, went AWOL. GreatAlbatross and I contacted him offering to take over and Tom made us Admins but we didn't have any access to the server, so we were limited in what we could do.
The situation got increasingly problematic and plans were drawn up to take over the whole running of the site or to start a new one from scratch. It took a while but just as we were about to pull the trigger on the new instance Tom gave us access. GA is now in charge of the domain and I have overall responsibility for the new server.
The site has been suffering outages because the size of the image folders ate up all the available space and Tom removed the first image folder (why various graphics are still missing). That made moving the site to a new hosting plan more urgent and we've been working away in the background to get to the point that we were in a position to start the process. Hence the downtime.
We'd like to thank everyone for their patience and enthusiasm - it has helped carry us through what hasn't been a straightforward process.
I have more questions
There will be more information after the move about how we are going to move forward with feddit.uk to ensure it will thrive for as long as people want it. However, if there's anything that is unclear about current events then fire away in the comments below.
If this message is posted...Then we are migrated. I just need to sort a few missing pictures.
And...... we're back!
Top job everyone. Thanks a million.
Looking good to me. Well done folks!
Good work guys
I actually thought the site was down for the last few weeks. It was just showing an error.
It was only when I cleared the cookies, and relogged in that it would load again. Think it was something to do with the upgrade to 0.19, but I do wonder how many other users are just seeing the error and making new accounts elsewhere.
We were able to update the error message and point people to Matrix if they had any issues.
We don't know how many people we've lost in this whole process but we will try and spread the word around and hopefully hook people back in (their accounts and all our communities will still be there, so folks will wander back, even if they may be using a new account). Fingers crossed, everything should be a lot more stable going forward and we can all get back to the more interesting stuff.
I responded to somebody a while back with the same issue. The server error message is very unhelpful!
The image folder has now been restored on the new location, for anybody wondering.
It was deemed to be a safer way to do things, than try to fold thousands of images into a folder on a live site.Do either you or Emperor have any experience with web servers or are you muddling along? Or maybe getting help from Tom?
What Linux distro is the new server running?
Do either you or Emperor have any experience with web servers or are you muddling along?
My previous answer stands.
Or maybe getting help from Tom?
Other than providing us with access to the relevant bits and bobs Tom is not involved with the move.
What Linux distro is the new server running?
We will only be revealing some general technical information about the new server to reduce the attack surface.
Issues with image uploads should now be fixed.
Unfortunately, this has interrupted my long-running task restoring the images uploaded between when I made the image backup, and when the old server was turned off (about 5 days).
We're aiming to restore them soon, but it's going to be a background task, and it's going to take a while (there are millions of images, and the sync task needs to scan each one, see if it's already present, and if not put it in place).
I'm sacrificing a goat to appease the gods of live server maintenance as I type
For anyone worried about this, no animals will be harmed during the migration, other than some potential gnashing of teeth and/or tearing hair out (all human, all self-inflicted). However, any "spare" goat meat can be sent to my PO Box.
Hey, users from other instances, listen up...
spoiler
Let's say nice things about British people when their server is down. Trolled.
"feddit.uk is asleep, post tea without milk in it"
Or worse, milk in the water before the teabag.
The Geneva Suggestions don't even cover that sort of behaviour.
When I was browsing instances, I vaguely remember that feddit.uk was hosted in Finland. Where will the new hosting be based?
Good luck team. I've been waiting for this day so I can create new banners for !Homevideo@feddit.uk.
Does that mean all missing avatars and icons have been there all along but just been disabled? Will that get repopulated from other servers? Just curious.
P.S. I like the gateway message. Nice touch.
Does that mean all missing avatars and icons have been there all along but just been disabled?
The first folder of images (which held a lot of early, core graphics) was zipped up and removed from the server.
Will that get repopulated from other servers?
I don't want to guarantee anything until we can see that everything has worked, but that folder is currently reintegrated into the new server. It may be caches need to ge cleared or something needs the delicate technical equivalent of a swift kick but all graphics should work within a day or so (or immediately for some people, with luck).
We'll be able to update everyone on this tomorrow if the move has been successful, so if caches need clearing for some folks they'll know pretty quickly.
P.S. I like the gateway message. Nice touch.
Thanks! I was having a ponder on what the site "was". And thinking of it as a gateway, with a some social spaces too, felt right.
Kinda early web style, perhaps?
We now have a WhatsApp channel. Seems like the right time to roll it out.
There will be other ways of engaging with feddit.uk coming up
Aw man, why WhatsApp? Facebook apps should be avoided by everyone.
Meta apps cause bone cancer. I think Emperor帝 is using it to trace people for GCHQ too!
While I'd love it if everyone I knew abandoned WhatsApp, it'll never happen, so I and many others use it.
We are going to be increasing our... outreach to other services going forward and this will be one option amongst many. If people don't want it then they don't have to use it - we'd still encourage people to join our space on Matrix.
@Emperor@feddit.uk and @GreatAlbatross@feddit.uk - why did you not clear this with me before you made the plan?
This is the first I've heard of it so I find this quite a surprise which is unacceptable considering my importance to the smooth running of the site!
I provide this site with much needed wit and wisdom.
Could you explain this oversight please?
If I don't receive an adequate explanation I'll not give my blessing for the move.
If I don’t receive an adequate explanation I’ll not give my blessing for the move.
So if something goes wrong, it's your fault? Duly noted if we need to apportion blame tomorrow evening.
i will be truthful, i had lost complete confidence in the instance after seeing the crashes back in december. i had moved to using lemmy anonymously since then, to "wait and see" if i had to setup an account elsewhere.
seeing this though? full confidence restored. 👍
i am glad there is now more than one single point of failure. i hope that this adventure becomes a valuable lesson to other instances that have just one admin running everything. if you're reading this, and your instance has one admin - you need to fix that, yesterday!
once you have an OpenCollective or something like it setup, i will be able to contribute. make sure to let us know when a fresh donation link is up (because i don't trust the old ko-fi link in the sidebar)
thank you so much for your hard work. ❤️
And thanks for your kind words.
i am glad there is now more than one single point of failure.
And that was a definite concern for us and other Lemmy instance Admins.
Going forward feddit.uk is going to be future-proofed so it can survive and thrive. It's now everyone's site, the Admins are just running it for you. So it's not just a change in getting to the point where the site can survive the loss of an Admin, it's a change in... philosophy.
once you have an OpenCollective or something like it setup, i will be able to contribute. make sure to let us know when a fresh donation link is up (because i don’t trust the old ko-fi link in the sidebar)
Yes the Ko-fi link will go once we are 100% confident we don't need access to the old server and so we are only using our own resources. I'll get in the case with researching Open Collective so we can use the momentum of the changes to ensure the finances are solid.