Universal and Transferable Attacks on Aligned Language Models - Carnegie Mellon University
Coverage:
Couldn't you just do a simple input classifier step to detect if there's nonsense strings in the user input and then not respond? You could even just use a simplistic algorithm to detect weird input strings.
Bing has a separate layer that attempts to step in to filter things, but false positives end up being pretty disruptive.