When I get cold I stay cold. It's like it soaks into my bones (I know that's not how heat works but that's what it feels like) no amount of added layers after that helps.
I do, it doesn't matter unless I'm just outside for a short time. Eventually wind gets through or I have to take a glove off for something and it's all downhill from there.