Have you tried using nix on Gentoo? I've heard that this is the best of both worlds: you can use Gentoo to configure your system just so, and use nix for the big binaries, the unimportant userland stuff, and nix-shell.
For no reason, I always wanted to try Gentoo, I'll be back to Nixos soon... until I buy a new computer.
Using the binary makes you install thing you don't want. by doing that , there is no point of using Gentoo at all .