The problem with those panels, in my opinion, isn't the defense of free speech itself, it's ending the conversation with a defense of free speech. This shifts the discussion away from how awful those other ideas are and instead distracts people with a debate over free speech itself.
If you truly want to support a free marketplace of ideas you have to be an active participant, you can't expect others to pick up the bullhorn for good in your place.