I feel you man I live in the country and all my shit is in plastic bins because of mice. Sorry you had to find out the hard way.
And yes you have to kill them or you will get infested. Use A LOT of snap traps. I have 10 in my attic and one in every cabinet. Poison baits hurt raptors and should be illegal tbh.