Since I moved my server to ipv6 only federation broke. I'm guessing this server is acessible trough the cloudflare proxy, but the underlying server is unable to connect to mine
That is very possible. IPv6 only is still problematic. There are still loads of servers without or with broken IPv6 connectivity. You could put cloudflare on front of your server and get an IPv4 address that way.
After some sleep, I have to concede you this point, I enabled Cloudflare no my side. I've read a lot of criticism of Cloudflare here on Lemmy itself, but it seems to be the lesser evil.
I don't like it either but the facts are the facts. Transitioning between technologies is always hard, especially if it isn't necessary. Nat works too well.