Actually, backing up a bit, this was all happening in the late 1700's in the midst of the Enlightenment and the French Revolution. There was a widespread rejection of anything tied to the old running classes of western Europe, and an embrace of utility over flair.
In Brummel's case he also apparently had a reputation for obsessing over details. So he went with the trend of avoiding garish, flamboyant clothes but would still show his class status by being super particular about the cut, fabric, stitching, etc. It caught on among the aristocracy. Modern men's formal wear is still very much stuck in that mindset.
This is interesting history that I've never heard of before. Thank you for sharing. I'm sort of glad for his influence, since I don't think I'd enjoy wearing all the crap worn by men in the 17th and 18th centuries.