It looks like it goes away (I went into the settings for one contact, where there's a toggle to switch back to SMS/MMS).
Unfortunately, I can't figure out how to change it back to RCS for that conversation now.
Also, on a more personal note: the baby blue background is ugly AF. Like actually just heinous. I'd rather it wasn't there, you get a perfectly good indicator RIGHT ABOVE THE TEXT INPUT FIELD what it's about to send.
All I'm asking is for Google to bake visual voicemail into the Google Dialler for all phones but at least those with Google Fi because the stock app is awful