I feel like it'd be weird because it usually does smell neutral after a time. You notice smells off of your baseline but typically you're just unaware after a while.
I read somewhere it's likely an adaptation to help us notice dangerous smells. You're more successful at avoiding fire or poo (germs) etc when it stands out, which is easier when the baseline is neutral.