Since I moved my server to ipv6 only federation broke. I'm guessing this server is acessible trough the cloudflare proxy, but the underlying server is unable to connect to mine
That is very possible. IPv6 only is still problematic. There are still loads of servers without or with broken IPv6 connectivity. You could put cloudflare on front of your server and get an IPv4 address that way.
I don't like it either but the facts are the facts. Transitioning between technologies is always hard, especially if it isn't necessary. Nat works too well.
This surprises me since Cloudflare has AAAA records listed for lemmy.world. Do you know your way around tcpdump? If so, it might be interesting to see what's happening with traffic to those addresses. My desktop doesn't have v6 right now (sigh) or I'd be testing that myself ...