Ever see the Mythbusters episode where they lined up toothbrushes to see how far fecal matter traveled. Right near the toilet was the worst but nowhere was safe.
So yes washing hands helps but it's not the be all, end all solution.
I'm sure someone will disagree with me but I think protecting ourselves completely from infection isn't a good idea. Particularly after reading Immune (good but not the best resource) I think washing our hands all the time or using antibacterial soap is not a good choice for our individual health and immune system strength.
I'll agree on not using antibacterial soap all the time. There's plenty of soaps that aren't antibacterial but still help remove contaminants. People should definitely at least wash their hands with hot water. It's one thing to harbor trace amounts of bacteria, but no one should be spreading urine or fecal matter to every surface they touch.
One of the first things they taught in my micro bio class is that no single bacteria is bad. What is bad is when they are where they shouldn't be. This makes the point of hand washing more or less to remove bacteria that may be problematic being on your hands where you could spread them to vulnerable areas.
Regular hand soap is perfectly fine. It may dry your hands out from overuse, but as soon as you touch any part of your body (like your arms or face) your hands would be inoculated with your body's natural flora that resides on your skin.
Antibacterial soap would be the larger issue, as a lot of them have what is called a persistent effect. This persistent effect means that the antibacterial agent leaves traces of itself behind and will kill any bacteria that is susceptible to it. While good if used responsibly for people like surgeons to eliminate/reduce any stray bacteria during a procedure, this can lead to the same sort of scenario we have with antibiotics where some bacteria may develop a resistance to it, which could also include pathogenic varients/strains.
Washing hands a few times a day when using the restroom is infrequent enough to not be considered washing them all the time. There are hours to get some bacteria, but washing a few times a day reduces the volume so your body doesn't need to fight as hard.
Sometimes if I'm active enough, but the use of few was to indicate that it wasn't constantly all day. Like every couple of hours and not every 30.minutes or something when people complain that washing their hands every time is some kind of burden.
I dont live in a sterile environment. There is a lot of ground between bleaching your home every morning and licking every door handle in Central Station.
Washing your hands after using the bathroom or when handling food (especially for others) is important. Proper hygiene is important, and your immune system will still deal with plenty of microbes to keep it busy.