I can't speak for the OP, but in my case.... It was very much a resounding YES!!!!! If your using Pis as general home servers, like I was, you can very quickly run into resource constraints. I wound up replacing 5 Pi 4's with a pair of 4th gen Intel boxes and I'm still hit resource limitations from time to time. Though now it is more io related then ram or cpu.
You didn't blindly upgrade to the next Pi. Not upgrading or upgrading to something you know meets your demands are the right things to do. I assume upgrading to a Pi 5 is not that for most people
I started my home assistant journey with 4 smart globes, a Google home speaker with a pi 4. I learned how to use it and started integrating things I don't need hardware for, such as shopping/Todo lists, weather, Spotify, last.fm, calendars, basically anything I can learn without hardware. Right now I'm working on getting a shopping list sent to me when I enter the shops zone.
Then I bought a bunch of NFC tags (they're cheap) for medication reminders, kitty litter reminders/tracking. I have a music poster in my house I've stuck an NFC tag to and it opens up the album on Spotify, turns on Bluetooth and connects to my speaker.
I've slowly been adding more devices as I go along due to cost constraints. Not that smart home stuff is expensive, I just can't afford to do it all at once. Which also gives me time to consider/research smart devices before I buy.
I had a bunch of Pi 3Bs sitting around, so I made piholes for a few friends and family, I made a dedicated MAME emulator that I never have time to play, and I gave one to each of my kids to learn about computers and linux. I also use one for work as a linux test environment for our software, but the 3 hardware doesn't really keep up.
Since none of these require a Raspberry Pi to run, I would suggest using a mini PC (with an Intel N100 or similar) instead of a Pi 5. With all the accessories needed for the Pi, a mini PC can actually be cheaper and of course a lot more powerful. Since the Pi 5 is very power-inefficient, a mini PC can even be better in that regard too if that matters to you.
Especially for Jellyfin a PC with an Intel CPU with integrated GPU is awesome, since Jellyfin supports hardware transcoding with that.
Yes, here the question is if you really need to upgrade to 5, and if you really need, why not buy a dedicated NUC for example that would be a lot more powerful and extensible than the 5, while also consuming not too much more power.
I have 2 pi 4. One of them runs Vaultwarden as my self-hosted password manager. The other runs TPLink Omada SDN management software to manage my switch and WiFi APs.