Not OC, but per my last experience with it NVENC was way easier to work with.
You install the NVIDIA drivers, you install CUDA libs (in Fedora that's separate, at least) and it works.
For AMD, you need to figure out that you need the proprietary driver for AMF (which didn't have a proper installer for anything that wasn't Ubuntu the last time I tried it) or be stuck with the unfortunately not as good VAAPI. After that you usually had to hunt for guides on how to use the encoder in the program you want (OBS used to be a particular nightmare for it, hopefully it got better with time).
I hope things got and continue to get better, specially since I'm 100% going to get an AMD setup after my laptop eventually dies.