Armémuseum, the Army Museum in Stockholm in Sweden does have a relatively large exhibit of guns. There is even a collection of war bounty cannons from Denmark on display outside.
Eskilstuna Stadsmuseum - Eskilstuna City Museum, has a collection of bayonets, swords, crossbows, spears, firearms, canons and machine guns with a special focus on weapons made in Eskilstuna.
I will also add Flygvapenmuseum in Linköping, it may not be handguns, but the Swedish Air Force museum is an excellent museum, showing off some truly unique artifacts like, the DC3 that was shot down by the Soviet Union over international waters in the Baltic while performing signals intelligence against them, several SAAB jet fighters, like Tunnan, Lansen, Draken, Viggen, Gripen and SK60, there are exhibits about Sweden and the cold war, flight simulators, futuristic architecture from 1968 used in the Swedish Air Force, and more. Highly recommended.
Thanks! I added the first two but for the sake of keeping the list short I'm going to try and limit the list to museums with a small arms collection. If I include every aviation, naval, or armor museum it'll be a lot more work. But I haven't been to Flygvapenmuseum, so if it does actually have at least a section dedicated to firearms or munitions I'll add it.
I did check out pictures of Flygvapenmuseum though and it looks very nice!