You can drink the tap water in most places in Iceland without boiling it (especially in Reykjavik), but if you're further into the countryside then I would check first!
The original (and best) soulBit - not associated with the furry porn guy