I think I found what you're talking about. If the scrotum was a clock they'd be at about 4 and 8 on the dial if you were facing it right? It is a very small area, like the size of a pencil eraser. 40+ years I never knew they were there.
I don't have any smooth spots on my balls, but I wonder, do you wear tighter underwear that may consistently rub in those spots? I have places on the outside of both my thighs where my pockets have rubbed against them for decades and hair no longer grows. Same on my ankles from the short, tight socks I used to wear when I played sports. I assume the friction just destroyed the follicles.
It's possible? I'm not a dermatologist or anything, it's just what I've observed on my own body, specifically from my pockets is the biggest one. If you ride very frequently, maybe next time pay attention to where you're body is making contact and causing friction. if it's right where your balls rest on the seat and they move back and forth constantly there, it's a good bet.
I wear loose underwear too, and I have always had the smooth spots.
I don't cycle much, but I walk and longboard a lot. I've always been fairly active, but I'm not sure if that has anything to do with it. Maybe it's just a genetic thing.