They can make a stink or write you up or something. But if they're as short staffed as everyone else, it's the sort of thing you can tune out so long as someone higher up the chain recognizes they need you more than they need five minutes.
Think of all the productivity that could be done in those 5 minutes. I make a habit of never asking for permission i just do it, my colleagues know me for vanishing mid conversation and by the time they realise I'm already halfway home.