The worry that someone would get murdered and they'd pin it on me because I'm a fucking weirdo loner and wouldn't have an alibi because I was always at home by myself was a non-negligible part of why I started making an effort to socialize more.
Probably depends where you live. In a big city I don't think anyone would bat an eye, but in a small town in rural nowhere people would probably get upset right away.