Skip Navigation
Fedigrow @lemm.ee
Blaze (he/him) @lemmy.dbzer0.com

Reminder that LW's delay with Aussie.zone is currently at 2.98 days

https://grafana.lem.rocks/d/bdid38k9p0t1cf/federation-health-single-instance-overview?orgId=1&var-instance=lemmy.world&var-remote_instance=aussie.zone

Parallel sending of federated activities to other instances. This can be especially useful for instances on the other side of the world, where latency introduces serious bottlenecks when only sending one activity at a time. A few instances have already been using intermediate software to batch activities together, which is not standard ActivityPub behavior, but it allows them to eliminate most of the delays introduced by latency. This mostly affects instances in Australia and New Zealand, but we’ve also seen federation delays with instances in US from time to time. This will likely not be enabled immediately after the upgrade, but we’re planning to enable this shortly after.

https://lemmy.world/post/23471887

43 comments

  • How is there a 3 day delay in federation?

    I am curious how this happens from a technical perspective.

    • Basically, the way Lemmy is designed, each instance has to tell each other instance what its users did (where relevant—no need to send to aussie.zone a post made in a community that there are 0 aussie.zone subscribers, for example). That includes posts, comments, and upvotes. And the way it's designed, the originating server (in this case, LW) has to send it to the receiving server (AZ), then the receiving server sends a confirmation back, and then the originating server can send the next one.

      Because LW is hosted in Germany, and AZ in Australia, there's a minimum amount of time thanks to the physical constraints of sending signals over that long distance. And double that because it's a return trip, and a small amount more for processing time. It ends up measuring in the hundreds of milliseconds. Which leaves you with a maximum of a few hundred thousand actions sent from LW to AZ per day. If LW users are doing more than this, then the delay will slowly grow. If they send less, the delay will shrink, or remain at near 0.

      Now, the most recent version of Lemmy actually lets you set it so that instead of sending just one at a time, you can have multiple threads, so you're sending multiple at a time. But LW only upgraded to this version a few days ago, and they didn't turn on this feature when they did so.

      • Wait, every single action is sent individually and the next action is sent upon confirmation of successful delivery?

        This is is wild.

      • Wait why didn’t they turn it on? When they upgraded I assumed this issue would be fixed.

    • Afaik only one item (post/comment) is synced at a time. So if there is a lot of latency between the instances, new content is produced faster than can be synced. But I'm sure someone will link the GH issue if I'm wrong

    • if the requests are all serialized, if the ping time is like 300ms, and if each request takes like 100ms of CPU time

      that means you only need 648,000 actions in queue to equal 3 days

      when you consider that even upvotes/downvotes of posts/comments count as actions, I could see it happening

      but the queue isn't completely serialized anymore, so maybe this number is still a bit unbelievable (EDIT: seems like LW has not yet enabled the feature for parallel sending)

      • A single action cannot take 100 ms of CPU time. This does not sound realistic.

        And why is this with aussie.zone only? This would be all instances, no?

  • I thought after lw upgraded the delays would go down :|

    • It should have. The backend added parallel syncing since a few versions back. By all accounts it should be going down.

      • The final bullet point of https://lemmy.world/post/23471887 suggests that that feature may not have been enabled yet.

      • Parallel sending of federated activities to other instances. This can be especially useful for instances on the other side of the world, where latency introduces serious bottlenecks when only sending one activity at a time. A few instances have already been using intermediate software to batch activities together, which is not standard ActivityPub behavior, but it allows them to eliminate most of the delays introduced by latency. This mostly affects instances in Australia and New Zealand, but we’ve also seen federation delays with instances in US from time to time. This will likely not be enabled immediately after the upgrade, but we’re planning to enable this shortly after.

        https://lemmy.world/post/23471887

  • Any idea how aussie.zone is hosted? Syncing can become bottlenecked as instances grow and discover more communities. If they're using a remote host they may need to upgrade or adjust their setup to hold sync with the larger instances.

43 comments