It’s weird because it sets you up as the objective authority on what makes them attractive.
With another man, that just comes across as a neutral weird. With a woman, it comes across with a bunch of historical and cultural baggage tied to how a large number of men treat women, and automatically associates you with that group of misogynists.