If you interact with any other server on the network (meaning users on another server), all messages/attachments/metadata will be synced with all other servers by design.
Given the percentage of users on Matrix.org & servers they provide, some of your data will end up on Matrix HQ servers.
I want to set up XMPP gateways to do the same, but I would still feel comfortable federating since the network isn’t centralized around a single hub. The clients are a lot less weighty too.