I've always hated "seldom". It just sounds so wrong to have an adverb not ending in -ly. It should clearly be seldomly. Or you can just say "rarely" which means exactly the same so idk why we need two words for the same thing.
I suppose well is common enough that it's become naturally integrated and I don't notice it. seldom is much rarer so hasn't hit that saturation point where I stop noticing it. And it just feels more egregious when you can say rarely instead, a much more common word.
Idk, I know it's a perfectly cromulent word but it just sounds wrong to me.