The previously fastest ran on Red Hat Enterprise Linux, the current fastest runs on SUSE Enterprise Linux.
The current third fastest (owned by Microsoft) runs Ubuntu. That's as far as I care to research.
Suoer-computing is a pain-in-the-ass, so my guess is some combination of SUSE picking up top talent that left other Linux vendors as IBM has been purchasing them, and SUSE just being willing to put in the extra work for the added brand recognition.
I can't imagine Supercomputers to use a mainstream operating system such as Ubuntu. But clearly people even put Windows on it, so I shouldn't be surprised...
They do use Ubuntu, Red Hat and SUSE mostly.
But for customers like that, the companies are of course willing to adjust the distro to their needs, with full support.
Microsoft uses their own Linux distro now.