Android custom rom security/privacy/trust

I've been a long time Android user and have been flashing custom roms on older phones when they end of life from their manufacturer to keep them up to date.

I started thinking.. how far should we trust custom roms?

There's a whole other debate how much you should trust the OEM roms as well but right now I'm focusing on custom roms.

Sure, they're open source but I'm not sure exactly how many eyes there are on the source code itself for a given rom. Many of them are "just" tweaks of some bigger more basic rom too, like Lineage OS for instance, then there's usually just one guy managing his particular rom.

Someone could theoretically add some nasties in there without people noticing if the code isn't vetted.

Sure, you could say that that's possible in all open source projects, like Linux Distros and so on, but there we have a ton of people working on the code so there's a much higher chance of bad stuff being found.

I'm not necessarily saying I don't trust Lineage OS or other roms, I was just hit by a train of thought and wanted to see what you guys think.

For my part I'd give more credibility to LOS than roms based on it that are managed by just one or a few persons for instance, but still.

I don't know. Was I suddenly hit by the paranoia stick or are these valid concerns?

Thoughts?