Lemmy.world down, probably following the upgrade. A reminder to move to smaller instances for a better experience
Lemmy.world down, probably following the upgrade. A reminder to move to smaller instances for a better experience
A reminder to move to smaller instances for a better experience
A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.
Github Issue 2910 is the kind of PostgreSQL problems that the developers ignored for months and people still defend the developer choices to have the code doing real-time counting of every single comment and post for numbers nobody needs to needs done in real-time.
PostgreSQL is voodoo to this project, they do everything they can to avoid going to !postgresql@lemmy.ml community and asking for help, learning 101 about how to fix their SQL TRIGGER logic like Github Issue 2910 spelled out June 4.
I know you are salty about how you are getting treated over at GitHub, but you should look at it objectively, Blaze is clearly advocating that people join top instances that's not lemmy.world or lemmy.ml, not nobody instances that only have 1-2 users. They certainly aren't going offline as quickly as they come online.
I know you are salty about how you are getting treated over at GitHub
No, it isn't about my personal treatment. It's about the cultist attitude you have towards Lemmy and the leaders without any ability to see what they are doing behind the scenes with the code. I know cults and religious faith is how many people enjoy the world.
A 2-line SQL TRIGGER removal takes about minutes to fix. It was crashing the entire site constantly. They sat by and asked for donations of money.
A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.
I am always advocating for any of the top 25 instances that are not Lemmy.world or Lemmy.ml
For the rest of your post, I don't know what that has to do with people aggreating on LW.
A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.
And this will always.. always be the biggest problem in the FOSS community.
"I dont like X, so I'm going to leave and make my own version of X"
So userbases get spread thin, manpower gets spread thin, developers get spread thin, and the user experiences degrades for everyone until it pushes them back to the bullshit websites and products.
Your answer didn't justify lemmy.world being treated the same as Lemmy as a whole. It's just a bunch of people who don't understand federation.
For the rest of your post, I don’t know what that has to do with people aggreating on LW.
And, factually, the project leaders telling everyone to create 1000 new instances and shutting down sign-up on Lemmy.ml caused more performance problems.
They had a bug in their PostgreSQL TRIGGER logic where 1500 instances were updating + 1 comment and +1 post counting instead of WHERE site_id = 1, a single database row. So each new Lemmy server that went online made the table larger and crashes more frequent on lemmy.ml
The amount of disk writing by lemmy was ignored
The developers of Lemmy seem to make every effort they can to avoid using Lemmy itself to discuss their !postgresql@lemmy.ml learning 101. They have made massive mistakes in SQL TRIGGER logic that they avoided to such a degree that their social motives are in question now. Github Issue 2910 was opened June 4, almost a month before the Reddit API deadline, and they ignored it. Just like they hang out on Matrix Chat and don't use Lemmy their own self to discuss code.
They have cultivated a kind of voodoo attitude towards PostgreSQL where people using Lemmy won't actually scrutinize the Rust code or PostgreSQL tuning parameters.
I mean having a bunch of new servers is not a problem. Just choose one that's been up for a while and more stable.
But each additional row in site_aggregates table was causing the instability itself. The SQL code had major flaws. Adding more servers actually made Lemmy crash more.
You can also just have multiple accounts. I have one on lemm.ee just for this reason.
Definitely!
Yeah, I have .world where I first signed up. It occasionally had some issues so I added another that I just mostly read on if the other isn't working
It'd be nice to export/import subscriptions but I'm sure eventually the servers will get past it
You can use that script https://github.com/CMahaff/lasim
It's funny, at this point I don't feel like I even have a main account. I just randomly pick an instance every time I go Lemmying
I'm trying to pick ones with meaningfully different aims so that I can try to keep some sort of separation in my head. Otherwise I'd quickly lose track of what conversation having where, with whom and why.
Same, I have accounts on lemmy.ml, lemmy.world, and Kbin.social. I'm just waiting for a good app to be compatible with Kbin. Right now all the apps coming out are geared toward Lemmy.
I forgot that I had a lemmy.ee account and set up my own instance a few days ago while lemmy.world was down.
Right now it’s just a free Google cloud VM instance and a $3/year domain. It was super easy to set up with Lemmy-Easy-Deploy and I’ll never have an issue with going down. I’ll either be busy looking through Lemmy, or busy trying to figure out why it went down!
Is there a readily available tool that can export/import community subs? I have two accounts on different instances but would like them to have to same subscriptions
I've not used it myself but heard good things about https://github.com/CMahaff/lasim
Lemmy.one, my instance of choice, has been down since Thursday - just a reminder that smaller instance isn't always the solution. Having a few solid account choices on multiple instances is the way to go.
It’s a bit frustrating though. If I didn’t know better about the fediverse, I would’ve thought Lemmy in general was down, with no heads up or somewhere to see what’s going on.
Lemmy One still down by the way
I have a few for this reason. Is there, or perhaps will there be a way for subscriptions to the shared across accounts?
Maybe this is already I thing and I’m oblivious.
I haven't had a chance to try it yet but here is this:
Lemmy.dbzer0.com is pretty stable, both in terms of release timing/server uptime, and also in terms of not flooding your feed with "politically instable" instances on both sides. Avoid the echo chambers.
One might argue that such a selection is, itself, an echo chamber.
That's my favorite instance as well but I'm not sure what you mean by politically instable on both sides
Quoting myself from a previous post:
First of all, it's really fine to stay on LW for now, no need to rush anything. But if at some point you have some time for this, then read the following.
So, to pick your instance, you can have a look at https://lemmy.fediverse.observer/list, filter by "1m" to see what are the most popular ones. As you can see, with a 27433 monthly users, Lemmy.world is by far the most popular, which is why you might experience some issues from time to time.
You should have a look at the next instances on the list. Short story: lemm.ee, sh.itjust.works, lemmy.one, sopuli.xyz and reddthat.com are solid choices.
You are looking at instances with quite a lot of people (the more people help with filling your "All" feed), just not the most populous one (lemmy.world), the original one (lemmy.ml), and instances that are too specific, either due to country or specific focus.
Long story:
spoiler
- lemmy.ml is the original insance, also quite crowded, not really the best choice
- lemm.ee can be nice, you can have a look at it and see how fast it is for you. The admin communicates a lot and is very helpful.
- sh.itjust.works had some rough time in the last few days. You might also not like the name, that's okay.
- beehaw.org does not federate with the big instances, so if you go there, you will be in their own space. It can a valid choice, but please have a look at their guidelines first, they tend to moderate a lot. Can work for you, or not.
- feddit.de, lemmy.ca, discuss.tchncs.de, feddit.uk, aussie.zone are country specific instances, so probably not interesting to you if you are not from there
- lemmynsfw is a NSFW instance, probably not the one you want to move to
- programming.dev is an instance focused on programming
- lemmy.blahaj.zone is a pro queer instance
.
To migrate your settings (including subscriptions and blocked instances), you can use that script: https://github.com/CMahaff/lasim
I think there is a huge misconception many people have that a larger instance is more likely to stay around, but due to the nonlinear costs involved in hosting fediverse instances this is not true.
Basically there is a sweet-spot around a few thousand (~2500) members where costs are low enough for a single admin paying things out of their own pocket long term is possible, but also enough members willing to occasionally donate or contribute otherwise to cover costs.
If anyone's looking, my instance is open for everyone and I plan on supporting it long-term. https://lemmings.world
LASIM author here, ironically on my own alt: Just an FYI that support for Lemmy 0.18.3 is not yet out, but keep an eye out for it soon (I have it working on a branch but I need to test it more before release).
This is the first breaking API change since it's creation, so here are the limitations:
- Old version (0.1.2) will only support API 0.18.1 and 0.18.2
- New version (0.2.0) will only support 0.18.3 (and above until there are more breaking API changes)
- Profiles downloaded with 0.1.2 (and below) will automatically be converted to work with 0.2.0.
So that all means:
- You can use the old LASIM to migrate between 0.18.2 Lemmy instances
- You can use the new LASIM to migrate between 0.18.3 Lemmy instances
- You can use the old LASIM to download from an 0.18.2 instance then use the new LASIM to upload to a 0.18.3 instance
- You cannot use the new LASIN to download from a 0.18.3 instance and then the old LASIM to upload to a 0.18.2 instance (unless you are comfortable doing some manual work editing the JSON file so "old LASIM" understands it).
This will be true of every release with breaking API changes.
EDIT: PR is out. Once it builds, I'll publish a new release! https://github.com/CMahaff/lasim/pull/21
EDIT 2: Release is published! https://github.com/CMahaff/lasim/releases/tag/v0.2.0
geddit.social is a another good server, I have alt there. It's run by @stux@geddit.social who has his own hosting company and also hosts mstdn.social.
Fwiw, sh.itjust.works is a horrible name, but a great instance lol.
What's funny though, I'm getting beehaw posts in my all feed since yesterday. No idea if they've refederated with us, or if it's an artifact of connect (my app of choice), or what,
Interesting. You are still on the list of their blocked instances, so I'm not sure: https://beehaw.org/instances
The DB migration at the end of this upgrade is significant, I was surprised how long it took when I upgraded my instance. Lots of room for things to go wrong considering the size of their DB.
That makes sense, especially how large the instance is. Just curious, how long did the migration take for you? Half an hour?
Took me the whole day. The new migrations require PostgreSQL version 15, and my Akkoma and Lemmy are using a shared database server of version 13. First shut down both services, then update Debian from 11 to 12, PostgreSQL from 13 to 15 and after all this, redeploy Lemmy to start the migrations. The new Lemmy queries use more RAM compared to the previous version, so the database was getting OOM and I needed to upgrade to a bigger instance.
Not fun, but everything works now and is stable.
Just make a second account, the one I run, lemmy.myserv.one is so underutilized its a joke. Smaller instances like mine basically have to beg for users and the server goes unused while bigger instances struggle under the constant traffic.
Because using random tiny servers is worse in other ways. With all due respect, nobody knows you and they don't know how committed you are or how much time you have. When your server gets DDoSed or hits a bug causing data loss, what will you do? Do you have the technological know-how to recover and quickly? If your server suddenly grew and it became more expensive to run, how does anyone know if you will keep paying the bills? If Lemmy has a bad zero day, will you upgrade quickly?
There's no need to answer these questions. I'm not actually asking you personally. But these are the kinds of questions that users have to worry about from random, small, unproven instances.
(Also, Lemmy does not favour small instances because the "all" feed, searching, and going to new communities are all better the more diverse users you have.)
Yes obviously the barrier to entry is high. But nobody knows for the big servers either since they are basically just small instances that happened to get big. Thats why lemmy.fmhy.ml just died one day due to domain seizure. End of the day all you can do is look at how long a server has been around and if it has be online a reasonable amount of time. That kind of reputation just increases slowly and nobody can make it happen faster.
As always, I have to ask: is there a second admin, what would happen to the instance if something happened to you tomorrow (which I really wish will not!)
The Vlemmy.net disaster is still fresh in people's mind
Ultimately I am the one paying the bill currently so if I die nobody elses credit card is being charged.
In terms of other admins, this is actually happening. Some smaller instances like mine are in the process of setting up a sharing admin work between instances so that if someone is on holiday, the instance still has an admin who can login. This was only just started and is in the process. We have to create a lot of documentation and basic stuff to get it fully functional where another admin can login and fix something. Its not at that stage yet and will be a couple more weeks before it is. We did a test last night where another instance admin (boulder.ly) could connect to my instance via ssh but without documentation on what to do and check anything more than the basics of rebooting or restarting something isnt going to happen. Eventually we will get it to what to do if site has a critical vulnerability or is being attacked but not ready yet. Its a work in progress unfortunately.
P
I think without some kind of "incorporation" (or whatever the tech/FOSS equivalent of that is), most of these kinds of thing will be vulnerable to issues with the owner's payment methods failing. Even with donation options available it's almost always still being used to pay to the server owner in parallel to them paying server / domain costs out of pocket (and then reimbursing themselves with the donations)
That said, I have to assume there's some way to set up some kind of automated payment option where community donations actually fill a fund that is used to pay costs directly in case the maintainer drops off the face of the earth.
when I start writing this comment, the post is 47 minutes old. if I understand the linked page properly, lemmy.world has been functional (all green checkmarks) for the past 10 minutes which is the furthest back the data goes. All the other instances are all green except for lemmy.one which is all red. I am assuming that 47 minutes ago, lemmy.world had red boxes?
Maybe a different link would have explained the point better but I don't really see how a 30 minute (??) server outage during an upgrade is compelling to avoid a large instance. Are you suggesting it's better to use a server whos admins don't upgrade? If not, is there really any size of server that would meaningfully avoid this kind of occasional disruption? Seems to me that the dynamism of the environment will inevitably lead to various problems. That's part of the experience. TBH threadiverse uptime on the whole is pretty impressive for such a ragtag groups of admins and devs.
I have accounts on some smaller servers but they have their drawbacks too. Using a bigger server is more convenient because the people and content is already there. It's easier. I didn't plan to use lemmy.world but I ended up making account there to use sometimes.
I think in a year or so the situation might be different. I see the ideological point and I would like it to be true. Maybe the technology will catch up. I think it would be nice to be able to programmatically seed content, but maybe that would be obnoxious to admins.
I didn't track the timing too closely, but in the last few weeks there have been quite long disruption of service due to DDoS attacks on the largest instances.
I am personally quite tolerant towards Lemmy as a platform in its very infancy, but some other users might want to quit it due to this kind of annoyances, hence my comment about moving to a smaller instance.
the people and content is already there. It’s easier
What do you mean? You can access all of the content in the Threadiverse from whatever instance, modulo defederation, but Lemmy.world defederates quite a few instances too, so that's valid for both big and small instances. If you are talking about the "All" feed, which will indeed be empty if you are in a 10 people instances (communities need to be subscribed by an instance member for the instance to get the community content), then it's a valid issue, and that's why I suggest people to move to one of the 24 biggest instances that are not LW or Lemmy.ml
Lemm.ee, sh.itjust.works, sopuli.xyz, reddthat.com, lemmy.one, your country instance (if it's big enough). I'm on sopuli since a while now, and I'm very happy with the experience. 680 monthly active users, so the All feed is pretty much identical to the one on Lemmy.world, except the vey niche community I either don't care about or would already know by myself.
The biggest issue I see with having everyone on LW is that at some point the costs will be too high for the admins. It's quite a big risk, and that why I'm advocating to use smaller instances.
@Blaze as I understand it, if you are user on a small server, you only see content from communities that others on your sever have previously subbed to previously, or if you do so yourself. And then you only seen content from the moment of subscription on. There is no way to see back prior.
So if you want to use a community like !fediverse it's OK because its popular and there will be prior subs. but if you are interested in !rockingchairrepair you will miss all prior discussion. Am I incorrect?
Also in practice, from my experiments, there seem to be inconsistencies in how even this works.
Of course I have one or two other accounts, but I personally like Lemmy.world. They serve as a necessary stress test that shows the devs and admins how to optimize further, and I just like learning admin practices at this scale of a userbase from a work perspective. Plus I don't want to be on an instance so small they can't or don't know how to handle compliance stuff and evaporate if something like that comes up. Not saying I know how to handle all of those situations, that's the job of someone else at work.
I think that's one of the issues that the rest of the instances are facing to appear as trustful as LW. LW admins have a long established reputation and experience managing Fediverse services, and provide very good transparency and a large team.
Other instances are usually nowhere close to that (some will be in the future I hope). The question I usually raise when someone start promoting their instance is "how many admins do you have?´What happens if you run under a bus tomorrow (hopefully you'll stay safe of course)? Is there a back up plan in place?"
Yeah, longevity and name recognition are why I went with sdf.org. They've been running many-user services for decades, even if the Lemmy service is pretty new.
ETA: they've been around since BBSes. I'm on a wicked nerdy old-school geek instance, and I love my local communities.
you should see how many of these big servers were (and still are) running their instances as 'root'...
I specifically stopped trying to grow my instance to focus on security and sysadmin back-end administrative infrastructure.
Lemm.ee is amazing with a top admin
💯 checking out the admin of a Fediverse instance you want to join is a good idea. Do they communicate with their users? Do they seem responsible ? Do they have any previous experience as a network admin? Lemm.ee was an obvious yes on all counts.
The admin for vlemmy.net was the complete opposite and look how that turned out.
As I said elsewhere
There are certain things that are memory intensive and CPU intensive. If you have 10k on one server doing that it really adds up. However having them across a wide range of smaller servers, its not such a big deal.
As a user, you literally lose out on nothing not being on lemmy.world. You can partake in all the same conversations, communities and everything. In fact when lemmy.world is down, you can still see everything and when it comes back up, your posts will synchronize. There's genuinely no upside to being on lemmy.world. That's the way the system was designed.
Not sure people will listen though. I will always talk up the amazing admin I have on lemmy.tf, but it's also worth mentioning that I have a bunch of communities hosted on other instances and each and every one of them is amazing.
Definitely.
I moved to self hosting mine instance :)
Well done!
There are 63 instances according to Jerboa.
Do note that this doesn't actually account for an accurate total of Lemmy instances across the Fediverse. It is a hardcoded list of instances.
Edit: Correction, its a generated list of instances (that gets copied into Jerboa) that have a MAU amount of more than 50 as pointed out below.
working fine for me , actually I have to say lemmy.world has been very stable
Perhaps I open Jerboa too often but in the last week lemmy.world has been down around 4 times for me, sometimes for a short time sometimes for much longer. When I was on lemmy.fmhy.ml it was only down once for me until their domain was seized and that was for the server upgrade.
Has anyone any idea what's going on with lemmy.one? It's been down for quite a while now
Probably some hiccup in upgrading to 18.3
Just waiting patiently to get approved on midwest.social
Hopefully they'll get back to you soon!
I was on lemmy.world and had a really bad experience. The I moved to aussie.zone and its better than reddit. Never down, commenting, posts everything feels so fast and snappy.
This is why I love small instances as well. I'm on Lemmy.today and it flies.
People seem to be super worried about an instance disappearing for some reason. It has happened to like 3 instances, we are soon 1400 instances.
Also if it would happen, how hard is it to move to another and click subscribe on communities? Takes like 20 minutes if you have many of them.
I really think all this fear is silly. Enjoy Lemmy the way it was meant to be used. Use small instances.
The instance lemmy.kya.moe that I created my alt on performs well too if anyone else is looking for an alternative :D
Smaller instances are usually worse, in my experience
Where is this screenshot from? Link me plz :)
Danke!
The major issue I see with Lemmy communities and Kbin magazines, is that they both rely on a single server to be up and running to even work. Sure, you can cross-post to several communities at once, but that generates one different thread per community. Add to that the fact that to even start to implement a single distributed thread properly cross-posted to several communities, they must take into account that a given user or server may be defederated from one community but not another - should the user not receive the message if at least one server bans the user, or should the user be allowed to receive it if at least one server allows the user to receive it?
Horizontal scaling could be better, probably a long-term improvement to be considered
I said that in a support post on lemmy.world, and got accused of pushing an agenda 🙃
Sorry to hear that. Based on this thread, it seems to become more and more of a general agreement
We're back!
While this thread has some interesting points in it, the majority of it is chaotic and confrontational. I'm closing this, as I believe we need to have a bit of a cooldown.
Getting server errors?
Are you saying you’re getting server errors accessing https://1up2down.social/lemmy ?
This is the problem with Lemmy. It's hard for it to go mainstream. :(
every site has scaling problems unfortunately.
Please come to lemmy.today if you like a small fast instance with no downvotes.
Also federates with all other instances.
I don't understand the appeal of no downvotes. Do you really think it's a good thing that trolls, bigots, dangerously wrong answers, general assholes, spam, etc can't be downvoted? I won't pretend downvotes aren't misused sometimes, but their existence is critical for quality control.
Edit: wait, I just saw you post in another thread as an "enlightened centrist", so I guess that explains it.
I saw it being misused on reddit a lot so I like it. It was common for people to use it as a "i disagree" button, which created a big echo chamber. I read that many people felt that there was no point even posting to a discussion since they knew they would be downvoted.
I think reddit really turned to shit, and I was hoping Lemmy could take another direction here. You can still upvote to support your opinions riding to the top without needing to downvote someone.
I understand this is a hot topic for many though. I guess it depends on what your previous experiences at reddit has been.
And also you mention the word quality control. I'm not sure the majority is some kind of a quality control. I rather hear people's opinions and make up my own mind. It would feel weird to have other people push down comments I may want to read.
Lemmy's machine-generated ORM SQL and hand-made flawed PostgreSQL TRIGGER logic is so bad, bloated. The developers on GitHub brag about "high performance". It's unbeliable.
In reality, small instances work because it has so many SQL performance problems that it mostly only is stable with little posts and comments in the database. They dd everything they could to avoid using Lemmy itself to discuss !lemmyperformance@lemmy.ml topics and hang out on Matrix Chat to avoid using the constantly-crashing servers they created.
If you go to a server with no users creating comments and posts and only has a tiny amount of data, it does crash a lot less.
Oh geez, why would the Lemmy developers want to do any kind of discussions with you over at the !lemmyperformance@lemmy.ml community, which you moderate?
I see you think that their behavior has only been since June 2, 2023 when I created my account.
The problems with them avoiding Lemmy, "eating their own dog food", to discuss !postgresql@lemmy.ml have noting to do with me. They hang out on Matrix Chat and do not ask for Rust or PostgreSQL help to their constantly crashing code.
Your style of arguing is to say I wear glasses and have "4 eyes", childish. You obviously can't go see they created a new Rust front-end on Github all on your own .... and that it was not me personally who created GitHub issue 2910 on June 4, 2023 - almost a month before the Reddit API change.,
Is it the pro-China stance that you like about the developers?
You're spamming this all over this thread. Why don't you go create a PR instead? If you think you have a better solution then go discuss it with the people who have the full context and try to get it fixed instead of complaining here.
Why don’t you go create a PR instead?
It's amazing how you have fallen hook line and sinker into believing that the problem is difficult to solve. It's the agenda that is the problem.
They have people like you who will not read actual code to see that they only care about the fact that "Rust is cool programming language" and crashing code doesn't get any priority.
They even started a new front-end Rust application this month, because they don't care to bother with the core of the site, PostgreSQL doing INSERT and SELECT statements to load comments.
You're rocketderp based on your instability I'm guessing? Looks like your the only one that is nonsensical in that thread. Your PR isn't a PR. You were rushing to someone that was being helpful but you still acted like a child with your advise of GitHub bugs and PRs and not using them properly. The other person in agreement with you at least is calm and rational. I'm guessing you never worked on a group project before or have had anyone disagree with you. Your commits broke the pipeline. Someone even tried calling you down, but you wouldn't listen.
You aren't the victim here. You're the bully.
You’re rocketderp based on your instability I’m guessing?
Yes. My name is Stephen Alfred Gutknecht. It is clearly detailed here on my profile that I am "RocketDerp" on GitHub.
Looks like your the only one that is nonsensical in that thread.
Issue 2910? The word "nonsense" is being discussed about a Lemmy TRIGGER on June 4, 2023.
I’m guessing you never worked on a group project before or have had anyone disagree with you.
You would be guessing wrong. Since late 2019 I am in constant mental anguish and barely surviving. My communications is a constant struggle and I openly disclose this so that it is understood, I am not attempting to cover it up.
I’m guessing you never worked on a group project before
The "i'm guessing" game. And I note you can't actually cite the Github issues by number (2910) or keep details here in fact. A lot of "guessing" going on.
based on your instability I’m guessing?
"guessing" again. Should I guess you do not know how difficult or easy it is to remove a TRIGGER in PostgreSQL that is causing server crashes?
You aren’t the victim here.
Why do you think this is about "me"? I haven't donated money to the project or had blind faith in the developers.
You’re the bully.
I'm beyond frustrated that they seem to be going so far out of their way to have servers crash that their motives for doing so need explanation. "Social hazing" is the best answer I can come up with. I've now outright asked why they are dong it, because the pattern of behavior with trying to let server crashes has been going on now for months.
I have to admit that it's amazing the number of people they have attracted with the server crashing. It's been a social experience in some ways like Elon Musk is doing with Twitter. Perhaps it bothers me when other people actually seem to enjoy it. That seems to be the hiveMind reaction here. In that sense, if it is "social hazing", it has worked very well.
That's an interesting community.