Yeah, my experience with Element and a Matrix.org account is that it's sluggish. However, it's been better at Beeper, so I'm uncertain whether it's intrinsic to Matrix or merely Matrix.org and/or Element's servers.
I don't believe that its existence causes more fragmentation than it remediates. https://news.ycombinator.com/item?id=36939482 explains why I consider Matrix fundamentally superior most (if not all) uses, although in practice it's because the clients (Element and FluffyChat primarily) are cross-platform and support a generally uniform set of features, in comparison to the aged (but glorious) Pidgin, and its counterparts.
Your hackernews post and the fact you mention Pidgin shows that you haven't used xmpp in the last 10 years. By the time Matrix was first released, xmpp had history sync.
Which is why I can't wrap my head around why a second protocol with no features that didn't already exist in XMPP took over.
I used it yesterday, via Pidgin. I'm rokejulianlockhart@xmpp.jp. Why else would I have referenced it? Don't tell me what I've done. That's not a way to have productive conversations.
Regardless, I can't provide any more technical insight than that - I know solely that the clients provide so much more functionality that irrespective of the protocol, it's better in practice. Fedora, openSUSE, the Bundeswehr, NATO, and Beeper - all chose Matrix over XMPP, not least partially because of Element (which they also all chose).