Am a bit confused about these two and want to learn more. Are there recommended use cases? Which one is more secure? Is there something like the "better" one?
They could serve similar purposes. In terms of maturity nostr is younger. Here are the main differences from the point of view of nostr:
In nostr there is no registration, your identity is your public key that you generate by yourself (lose that and you cannot recover it). You can connect to a bunch of different nostr relays with the same key, or use different ones.
AFAIK nostr does NOT do end to end encrypted for group chat. But it does support end to end encryption for direct messages
nostr does not do video/audio calls
nostr does not host your images/files, you just put some URL in your messages
At its core nostr is a basic protocol where you send messages to a relay server and the relay passes them along to other people when they request them. And on top of those messages people implement extensions for features, full length posts, payments, etc. The are notions of followers and subscriptions (like twitter) but those are just tiny messages where you ask the relay for messages from person A or B. The list of specifications is here https://github.com/nostr-protocol/nips
Finally there are a few different nostr implementations for relays, clients and web interfaces. Some of them do not implement all the features, so you may need to shop around a bit if your are looking for some fancy features (check https://github.com/vishalxl/Nostr-Clients-Features-List).
Also some nostr highlights which I think don't have equivalent in matrix (but deserve nerd points)
message expiration dates - the relay removes them after the deadline
nostr has builtin proof of work to dissuade spam by forcing the client to do some computation before posting
you can do reposts across relays or share relay addresses to people in another relay
Ultimately you are trusting the relay server to hold your messages If the relay is not trustworthy, it could reveal those messages.
The only exception I know of are encrypted direct messages which are still held by the relay but are encrypted with the recipient's key. These messages still have a cleartext recipient id (so the server can deliver them).
So, if the relay is well behaved
messages are confidential between you and the relay
direct messages are only delivered to the recipient and are encrypted
most other messages are visible by anyone that can connect to the same relay
btw the relay can enforce a list of people that can connect (i.e. a private server) or just make it harder via proof of work (to discourage bots)
If the relay server is operated by the forces of evil, then the only thing you can assume is that direct message content is not visible, but they can see the message src/destination/timestamp.
I think the main motivation for nostr is censorship resistence - so if you are being blocked in one relay, you move to another - in terms of privacy/security it does not seem weaker than most other public message forums.