Matrix has de-facto centralization around Matrix.org where all metadata floats back through them in one way or another. Due to the expense of all the mirroring required (every message + attachment from every DM & chatroom of every user) as well as the Python back-end being slow, it’s in practice not good to self-host despite in theory Matrix being good. XMPP runs on a potato relatively & has so much feature overlap (with the ability to extend being inherent to the protocol)—the only issue is fragmentation of servers & clients (though Conversations Compliance is close-ish to a minimal standard).
I'm not very knowledgeable, but Matrix is only centralized, if you don't include non-matrix.org instances. You can safely chat locally without matrix.org, AFAIK.
If you chat with one user from Matrix.org or one user in your room is Matrix.org (or any org they host for) then all of that chat data will be synced with them. A room with any general traction will inevitably have someone from Matrix.org & there’s not a lot of options to choose from (I’ve seen a lot of instances shutdown because of the cost to sync all that data was too much).