Tell them if you have to go back to the office, you quit.
Nah, never say "or I'll quit", you just say you won't go to an office. End of statement. You don't make any sort of threat or give them an ultimatum, just set your boundaries. Make them fire you.