Yes, but fewer people know that word, so it's less useful. And if you want to have a word to describe every specific version of "meat is bad" diets, you'd need as many words as there are people who avoid meat.
We use the word vegetarian to mean that we don't eat animals. Fish is an animal.<br>
we, vegetarians, don't eat fish and "vegetarian" is a useful word to mean exactly that.<br>
we don't stop using precise words just because "fewer people know that word"! What kind of a reasoning is this 🤦
Words are approximate. You can get a general idea of what a human is by saying "featherless biped", but you're not going to go around saying that a plucked chicken is a human but someone who had a leg amputated isn't. If someone generally doesn't eat animal products, but is okay with jellyfish, saying they're vegan will give a better understanding of them than saying they're not vegan.
We define vegan as someone who doesn't eat meat, in the sense that if you ask someone what it means that's what they'll say, but we don't strictly use it that way. There's just too many details to make a word for every possibility.