Lemmy PERFORMANCE CRISIS: popular instances with many remote instances following big communities could stop federating outbound for Votes on comments/posts - code path identified

I spent several hours tracing in production (updating the code a dozen times with extra logging) to identify the actual path the lemmy_server code uses for outbound federation of votes to subscribed servers.

Major popular servers, Beehaw, Leemy.world, Lemmy.ml - have a large number of instance servers subscribing to their communities to get copies of every post/comment. Comment votes/likes are the most common activity, and it is proposed that during the PERFORMANCE CRISIS that outbound vote/like sharing be turned off by these overwhelmed servers.

pull request for draft:

https://github.com/LemmyNet/lemmy/compare/main...RocketDerp:lemmy_comment_votes_nofed1:no_federation_of_votes_outbound0

EDIT: LEMMY_SKIP_FEDERATE_VOTES environment variable

You're viewing a single thread.

6 comments

Thanks for doing all this.

Do we have any real numbers from a real server? How many votes are trying to be federated to how many servers?

Just ballparking some approximate numbers:

!technology@lemmy.world

15k subscribers

4000 subscribed servers

10 votes per subscriber per day

15000 * 4000 * 10 = 600,000,000 federated actions. That is around 7,000 per second 24/7 for one community.

IMO, this real time federation just doesn't scale. We need to start planning the specs for federation batching.

You've viewed 6 comments.