Gentoo. Great rolling release that is stable and had timely updates, but has the flexibility to configure my system down to the tiniest details, with a great and knowledgable community. I love source-based distros and Gentoo is definitely the best.
Yes, though there are some prebuilt binaries for large packages. I use gentoo on a desktop and updates don't take too long, minutes. Big updates that cause lot of packages to rebuild can take hours.