The main problem with polyamory is the jealousy. I have experienced jealousy maybe three to five times in life, because I was an only child and I have a very laid back temperament. I think if we start prioritizing quality of life more as a society, parents will be free to raise their children well with less insecurities, and maybe that would result in more people gravitating towards polyamory. But it's really not for everyone. Poly is hard work.
Jealousy is definitely the biggest sticking point for me. Every woman I've been with has always had a few other dudes sniffing around at any given time waiting for their chance. I on the other hand have gone years between relationships because I have a hard time meeting women that I'm interested in who also want to be with me. If I could easily find someone else to hook up with while my gf was out doing it I wouldn't really have that big of a problem but I would have a huge issue if I was stuck at home by myself while she was out potentially finding a replacement for me.
Also I think you have a good point that if we didn't have to invest so many resources into a relationship more people would probably be more okay with it.