Are people not allowed to find others attractive after they’re married or something?
There's a general taboo against oogling. But its often a question of finesse. Doing a loud wolf-whistle at someone as she passes you is bad form, and doubly so if your wife isn't wolf-whistling with you.
Also rudeness. Playing "hot or not" with every person that crosses your path is considered in poor taste. It objectifies and commodifies them, reducing them to visual spectacles in a way that can quickly become mean-spirited and oafish.
Also, depending on who your oogling, it can get creepy. Fixating on a particular age range or certain ethnic features (even unintentionally) is problematic for a whole host of reasons.