If you want to only specify gender then that is the correct word to use. If someone wants to be derogative towards a certain group, the said person can twist the context of almost every known word so that it sounds rude and disrespectful. Does not mean everyone else should stop using that word in the correct context. I don't know what else to say.
It's equally, or more, correct to say "female/male people". It's just like "poor people" is ok, but "the poors" makes you sound like an asshat. Including "people" makes the difference.