Anyone can stand up their own instance, subscribe to remote communities, and start receiving all the data necessary to show those communities. That includes posts, comments, and votes too.
Every instance operator is in control of a database containing all the activity for communities that instance's users are subscribed to. They can do whatever they like with that data. That's a consequence of how federation works.
The protocol as it stands today is also generally vulnerable to any malicious instance. A malicious Lemmy server could emit spam, send out bogus votes, or alter its users' comments after the fact (ahem, spez) and disseminate the modified versions. The main tool that other instances have to deal with a malicious instance is ... yup, defederating.
Ultimately, other federated services in Internet history have adopted different ways to deal with this problem:
IRC doesn't have a single federation; it has many federations ("IRC networks"), and server operators form peering relationships with one another based on mutual trust and agreement to uphold various rules. A given chat channel only exists on a single IRC network; you can't reach Libera.chat's #linux from a DALnet server. And occasionally a federation completely blows up — see e.g. the 2021 collapse of the Freenode network due to admin abuse.
Usenet pretty much floundered on spam mitigation because well-behaved servers didn't eject the malicious and ill-maintained ones.
Email has dozens or hundreds of different ways of dealing with bad instances (i.e. mail servers that emit spam), including published blocklists of known offenders' IP addresses. But even then, major mail servers depend on all manner of filters (including ML classifiers these days!) on top of straight IP blocking.
I think it is fine, since likes on twitter have been public. People just need to change their habits on upvoting and downvoting that they got used to when they were on reddit. So need to adjust to how people would go and downvote whatever comment they disliked and move on. Now that stuff is public, so maybe it can help against brigading?
I'm more concerned about the more toxic people having access to the names and profiles of people who downvote them. Reddit had a lot of crazies, and it seems like a good tool for targeted harassment. Not to mention, what's stopping them from having alt accounts on different instances and continuing even after they've been blocked or even banned on one account?
That is valid. It's part of why I wanted to make people know about how upvoting and downvoting works so they can be more mindful about how they use it.
It affects those who don't want their votes visible and for random stranger to track their activities. That's why this is better as an opt in for those who want them and have them be invisible for those who don't want to share it. Think of how you can set your Youtube playlists like favourites to private or unlisted.