Well, for what it's worth, I have literally never heard anything like that from even the most misogynistic men I am or was acquainted with. Don't get me wrong, they definitely had some awful things to say about how a woman dresses, but that specific one was not one of them. The few times I've heard similar comments, it was a woman doing it to try embarrassing another woman.
Maybe it's not about the origin of each and every individual specific possible insult about a woman's appearance, but rather the whole concept of the importance of their appearance. If you all are catching my drift.
I'm not saying it's like this. I'm just asking "new" questions to make us all think. Maybe it's not like that. But it's food for thought perhaps.