Matrix. With its bridges you can "wire-in" networks like WhatsApp, Slack, Signal, Telegram, Discord, iMessage, SMS, e-mail, ... and have a single app that interacts with them all. You can have a single group chat with users from all those networks participating and no one would be any the wiser.
From my experience (with Dendrite, not synapse, so keep that in mind), bridges create "fake" users to replicate your contacts on these platform as matrix users, and they are visible on the whole instance by all their users (but you might not be able to talk to them). Also, in puppeted mode (which is what you want to "replace" your app with matrix), only a single user can use the bridge at a time, so the other users cannot use it.
I'm on Signal (obviously not self hosted) and even if I really wanted to move to another platform be it self hosted or yet another privacy focussed one, I can't ask my friends and family to move to another platform again. I already asked them to move away from WhatsApp, can't do it again..
This is what I told most of my friend when they asked me to move to signal. Is is going to be a very shitty company managed by a shitty egocentric person and you are going to regret. But you will make people move and they won't do it again and won't understand the reasons
Nextcloud looks really great and it has a chat / video chat too, I want to give it a spin in the future, as it also allows you to self-host a lot of things that people usually outsource to Microsoft, Google or Apple.
I think you always need some amount of ram and cpu in a server... ;D
Well, it's a shame if those video calls aren't working nicely without some fiddling. I'll still set up a Nextcloud at some point, for all that other stuff like calendar, contacts, office, chat and file sync. Kinda enjoying the benefits of cloud synced data, but without a corporation hosting my information and selling me to the advertisement hyenas.
Matrix works, but it's way harder and more expensive to selfhost than for example XMPP, which can be hosted even on cheapest VPS or first RPi. I would definitely take the cost and "how hard is it to maintain in the long run" into consideration.
Mattermost also works and is pretty easy to selfhost, but it doesn't have federation.
Another option is always an email with delta.chat - I don't think it offers voice calling, but email is one of the most basic services one can host, and many automated solutions to help with that exist.
If you're already using Nextcloud, it has a chat w/ video chat as well.
Matrix / Synapse / Element.io is also pretty cool. The UX might not be on par with what some family expects though. I don't know if voice/video chat is built-in yet or not, but it was at least an option before.