I'm guessing he kept it all intentionally. He had the manifesto on him, probably expecting "accidental" suicide by cop in hopes that his message would continue and not be painted over by the media. Yeah, he could have ditched the gun, but again, perhaps he didn't want there to be any shadow of a doubt that he is guilty. This was an intentional sacrifice in hopes of making a change.
I can walk 1/2 a mile in any direction and find a body of water or deep woods where it would never be found. Also, I'd field strip it and chunk the parts in different places.
There's water in central park. Would've been nothing to chuck the pistol in a pond. Break it down a bit if you're extra. Slide in one stream, barrel in another, mag, grip, etc until you've disposed of it. Or trash cans at various bus stops on the way down to VA. Tbf it's really easy to back seat something like this. His brain must've been running a mile a minute, it's honestly impressive how well he did