There is no problem in the way you claim when it comes to limiting speech which is pro-Nazi or pro-Confederate. There is no question what such things are. Things like the march in Charlottesville or, as I mentioned, Stone Mountain, GA.
And they would say there is no question about trans or queer rights. You seem to be failing to consider how this would work from other's points of view. Just because you're right doesn't mean you're not outnumbered. You cannot change the status quo without necessarily being outside of it. Letting the state, with its monopoly on violence, enforce the status quo is counterproductive to the progress you and I both want. It is on us to use our speech to push for change and drown out the hateful speech.
You could make this argument about virtually any progressive idea. Of course political ideas can be used against you. That's not an argument for maintaining things as they are.
There are two different things that we're discussing here. The state and society. The state has a monopoly on violence and should not get to decide what people think and believe because of the monopoly on violence. Society, on the other hand, can and should make collective decisions on what is and isn't acceptable. We should all condemn hateful speech. We should take down confederate statues. We should advocate for change. What we shouldn't do is use the state's violence/force to do it. It has to be done by changing hearts and minds. It is our collective responsibility, not that of the state.
We absolutely should use the state to sandblast Stone Mountain. It's on private land so there's no other way to do it. It's an insult to every Black person in America, especially the ones in Atlanta.
I'm sorry but insults don't warrant state action. This seems to be a fundamental philosophical difference that we aren't going to come to agreement on.