In my hometown, we had "Mr. Noshirt". He'd walk everywhere in any weather either without a shirt on, or with a flimsy dress shirt unbuttoned the whole way. In Canada, so it got cold. Rumors were "couldn't feel cold" or the more likely "mental health problems."
I think you eventually get used to it whatever you wear
I tend to be the shorts and t shirt in the middle of winter guy, but when I put a hoodie on it's the most comfortable, perfect temperature you could imagine
As I get older, the cold bothers me more. Used to be, I could go outside and throw snowballs with my bare hands no problem. Nowadays, the temperature drops below 10, and suddenly my hands physically hurt if they get cold. It's not even related to how physically fit I am given I started a five-day-a-week gym habit eight years ago and am in some of the best shape of my life. It's just stupid aging as far as I can tell.