Because one of the biggest issues with sex work, human trafficking, gets worse with legalization. Studies across Europe have shown that countries that outlaw prostitution see a decrease in human trafficking victims while countries that legalized or decriminalized it see an increase.
Unlike with drugs, you don't just create a way to increase the supply. A very small minority of women actually want to engage in sex work. And the few who do, usually envision the high class escort lifestyle. But working in a brothel charging $100 per client isn't something many want to do.
But legalizing prostitution increases demand. Which makes it more profitable for criminals to utilize human trafficking to fill that demand.
https://orgs.law.harvard.edu/lids/2014/06/12/does-legalized-prostitution-increase-human-trafficking/
One source of it.
It also doesn't help at all with protecting victims of human trafficking. Victims of human trafficking are already protected. But they don't step forward because of threats against their own well being and that of their families. Something that doesn't change just because their work technically is legal now.
Which leaves a small percentage of people who fall into financial hardship and consider prostitution as a method of overcoming said hardship. For them that might slightly improve their situation. But that still means exploiting vulnerable people and isn't people engaging in sex work because they want to. And it's even questionable if people in these scenarios would follow the legal way.
So while initially it might seem like legalizing it solves a lot of issues, it is more difficult than that.