Not matrix?
XMPP is a good idea, but the wildly different levels of support among clients cause problems even back in its heyday
Matrix solves some of that, fully encrypted, chat history stored on the server in encrypted form, supports gateways to other services.
Those problems you speak of about XMPP are not really a concern anymore and haven't been for a while.
Matrix on the other hand is very difficult to implement, and currently there's only one (maybe two?) viable implementation choices. It is way over complicated, resource intensive, and has privacy issues.
Does it have privacy issues compared to XMPP which doesn't enforce the privacy extensions? I figure they are about the same there. Asking genuinely as I do not know other than Matrix might leak some metadata.
And quite frankly, I really wish we'd just agree on one or the other. Would love to host an instance and move some people to it but both are just stuck in this quasi-half used/half not state. And even people on here can't agree what should be "standard."
Xmpp definitely wins in privacy. What is there to privacy more than message content and metadata? Matrix definitely fails the second one, and is E2E still an issue for public groups? I don't remember if they fixed that.
XMPP being a protocol built for extensibility means it will be hard for it not to keep up with times.
On your point of picking one or the other, I'd say pick the one you like and bridges will help you connect to the other. But XMPP came way before matrix, and I believe they fractured the community instead of building it.
There's a good reason all the big techs built on top of xmpp (meta, Google, etc). It's a very good protocol and satisfies modern demands very well.
Honestly I just haven't looked at Matrix yet. Unfortunately like many of the privacy-centric protocols it's mostly used by people trying to hide something.
I don't know much about "mostly", but check out the channels on the server kde.org, where they do discussions regarding visual design, development, documentation and all that good stuff.
Sometimes, if you mostly find what you don't like, you might be looking at it from the wrong angle.
For instance, I found a few, very desirable communities on Reddit, so much that I am finding it hard to leave. And that is the few that I searched for. Only realised the toxic communities, when I read others' rants on it ^[and from the recommendations. Definitely don't checkout the Reddit recommended communities or you will get said toxic stuff.].