How do I get respite from my elderly neighbor invading my space?

Every time I go outside to my yard, my elderly neighbor comes out and tries to talk with me. Every. Single. Time. Without fail. I go out to relax and do gardening and yard work, and his presence makes me increasingly anxious & leaves me feeling particularly uncomfortable/unsafe.

He is dealing with dementia, and has become increasingly vulgar and outspoken, and even made some vastly inappropriate remarks about his wife & my other neighbor while my child was right there.

His wife is tired. I know this. Christ, I would be, too. He was kind of insufferable even before the dementia got bad (think him talking about his shitty political views), but, god.. I am so tired of babysitting him while I try to unwind in my own yard. He'll talk over the fence or simply waltz past it and follow me around while I try to do stuff or as I'm trying to talk to my kid.

One thing I know is this: do not piss off your neighbors, because, chances are, you will probably be living next to them for a looooong time.

What can I do? I just want peace without feeling like I'm being accosted.