and further, it feels like Matrix just progressively becomes more of a mess, like piling on more infrastructure on top existing infrastructure, to sidestep some performance issues.

Essentially "Hey, let's pile on ANOTHER PostgreSQL database, on top of your existing Synapse installation, to hold state information for your client, so it's so much faster to sync up your rooms when you log into a new fresh client!"

Whereas, 9 years into the existence of a protocol, and the lead developer has to present at a developer conference as a 'groundbreaking change' that you can login and see your conversation history within a couple seconds finally: https://youtu.be/eUPJ9zFV5IE?t=601

Yep, it's similarly of why I've lost enthusiasm of Matrix, especially with my prior focus on XMPP over a decade prior. It was originally marketed as if Matrix was going to have full account portability, and I believe even be able to pop onto chats on remote servers if even your homeserver is down; but instead all we got was just another flashy webchat with a RESTful API and Double Ratchet support that federates. The reference client is a boat anchor of resources, compared to something like Conversations/Gajim in XMPP world. There's the reference server in Python, which has a Rust rewrite, while the focus of Dendrite was retconned to "ehh, this is more intended for embedded use, not really a full Synapse rewrite"; meanwhile there's an ecosystem of several highly-performant XMPP independent server implementations.

I jumped onto Matrix back in the days when it was the Vector.im client (before Riot rebranding, and then Element rebranding) and had rode that for some years, but instead I'm back exclusively to XMPP.