How to speed up federation?
How to speed up federation?
Hi everyone
I am pretty new to all of this, so I'm currently in what I consider my "sandbox phase" of hosting a Lemmy instance.
Before promoting my instance to others I want to ensure that it's working as it should and federation is quick and "in sync" with the major instances.
Looking at the logs I see federation is at it and working at approx 4-8 req/s - but is there a way to speed this up?
Would more workers (and thus more cores on the server) in postgres speed things up? Or perhaps more parallel workers?
As stated I'm pretty new to this, and when searching for similar questions on lemmy/github I can't seem to find the answer to this specific question - it's usually implied that one should know all of this before hosting.
I've looked into how postgres is working, but am unsure if this is the only parameter that can be changed to speed up federation.
You're viewing a single thread.
There is no way to “speed up” federation. Lemme doesn’t backfill comments, nor posts, and will only receive new content if at least one user on your instance subscribed to a specific community. This is because the federation process (beyond the initial search) is not a pull model in which your server requests information from other instances and stores it; but rather it operates on a push model in which you tell larger instances that you’d want new information (by means of having at least one user on your instance subscribe to the community) and they push that to you. Once that subscription is made, only new contents created afterwards are pushed to you, as fast as the other instance could. This as you can imagine puts a lot of stress on the larger servers as they’d need to update every interaction to thousands upon thousands of subscribed instances. “Seeding” content en mass (making dummy accounts to subscribe to everything) makes it worse because now the other instances are pushing entire community’s new updates to the new instance where no one ever touches it.
Instead. It’s probably best to just subscribe to a small handful of communities that you actually care about and intend to read, so only content you will consume gets pushed to you. Then invite like minded people to join and gradually, they too will add more that they’d actually read, so your instance isn’t just sucking the resources out of the fediverse at large but not actually using it.
I understand the idea of the "fediverse", but personally think that the "no backfill" is the biggest missing piece of the great puzzle.
If I, 2 years from now, wanted to create an instance on the ever-so-popular fediverse, why should I? I can't attract any new users, as they'll lack all of the historical posts that makes a community a community (the inside jokes, the posts we still remember, that sort of thing).
Where we are today, better load your instance up with content from the popular communities at the popular instances, to ensure you are able to attract new users in the future.
I get why that defeats some of the purpose of the fediverse, but as for "new instances" it is crucial to have content - or else you have no new users.
I've played around with the Lemmony script, and altered it to take only popular instances in account. Right now it only parameterize by active users pr community, but I've changed it to also look at users pr instance, so you can subscribe to e.g. top 10 communities of the top 50 instances, so you have the content on your instance for new users joining... But this is just me playing around (and causing the load on the popular instances) with the dream of some day being able to provide my users with the content they would also be able to view on the major instances.
I really hope you’ve considered how much storage you’re going to need to sustain this.
Hey, when life gives you lemons, buy more storage!
In the most polite way possible: The community is not yours. The content you so badly want to extract quickly, for better or for worse, belongs to the communities on the other instances, and while your users can easily join and contribute there, that's not what should be the sole value/reason people join your instance.
Lemmy (and the Fediverse at larger) is intended such that like minded people can congregate and discuss on subjects that are pertinent to their interest. The right way should be to subscribe to content you intend to read, bring like minded people who are interested in similar contents you read to share your instance, and then grow organically from there.
If a new user on my instance was able to subscribe to a comunity on another instance, and all the posts/comments/likes of that community was being fetched so it was browsable on my instance - I would see no issue.
But as of now, if a user wants to browse content from several communities on different instances - and discover new interesting comunities on different instances - while being a user on a instance filled with "like minded users, they would need a user on all instances which hosts communities of interest to 1) discover these communities 2) view the posts from these communities.
What you are describing are single entities of communities, which defeats the purpose of federation completely, in my opinion.