Violence, by definition, is an unjustified use of force. If a use of force is justified then it isn't violence.
For example, suppose you're walking across a bridge and you see someone about to jump to their death. So you run over, pull them back from the brink, knock them down, and sit on them. Have you committed an act of violence? I would say not.
On the other hand, suppose the person is just standing on a street corner waiting for the light to change. If you run over, pull them back from the curb, knock them down, and sit on them, that would in fact be an act of violence.
Where are you getting this definition from in the first place? Looking up an English definition for "violence" gives me this from Cambridge dictionary:
extremely forceful actions that are intended to hurt people or are likely to cause damage
None of the definitions I've seen on the first page of search results includes a requirement for it to be unjustified. If you stab someone in self defense, that's still violence.