According to this article, about 40% of men and 60% of women identify as feminist. That means only about 60% of feminists are women, at least in the States.
There are probably more women with internalized misogyny than men who are feminist. This is not to say that there are few male feminists, there is just a lot of women in the world and not everyone has caught up yet.
They hear the word and they think of misandry, primarily because so many misandrists think they're feminists and very loudly proclaim so. The problem is, the more you try to convince them feminism isn't what they think, the more they tune you out because if you talk about feminism so much, you must hate men enough to dole out your own hatred as payback. Which is exactly the preconception you're trying to address.
Idk, maybe feminism just needs a good ol' rebranding. I think it would at least win over a lot of the under 50 crowd.
I think "feminism" is an inherently biased label. The core concept is "egalitarianism". I think the "treat women better" mentality, in an age with much more gender equality than that in which "feminism" took root, has provided for a shift from "treat women equal to men" to "treat women better than men". I personally think "egalitarianism" is much better branding, across all identities.