more derailed

Not at all, although admittedly my table is definitely more on the slapstick than the dramatic side.

everyone knows the reference

Absolutely, e.g. BitD referencing the game Dishonored, which some of my players hadn’t played.

I guess I should walk that recommendation back and say that if everyone is familiar with the media, and it fits the tone of the game you’re running, then references can be a powerful tool.

Being able to say something like "It's like ___ but with ___" isn't cheap, it's crucial. Being able to play out scenes at the table relies on everyone having a similar (-enough) idea in their head about what this world is like, and the amount they're going to be able to get from you in advance is very limited. That's why standard settings like the Sword Coast can feel so cliched - D&D runs on cliche. Short blurbs are better than detailed infodumps, and art is even better if you can find any.

I ran a game of Blades in the Dark for some friends a while back, which has a really cool setting, but because it was so out there it kept getting in the way - we'd be getting into a scene, and then someone would remember "Oh yeah, it's always night here" or "chickens don't exist here" or "Don't we all know that corpses turn into ghosts after a while? Maybe we shouldn't have left that guy there" and it would derail the whole session.

Also that's another reason letting your players fill in the gaps in your worldbuilding is so powerful - not only does it make them more invested in the world and story by giving them a feeling of ownership, which changes the way they behave towards NPCs and reduces the risk of them going full murderhobo on your beautiful creations, but also having information about the world be tied to cool moments at the table makes that information way stickier in everyone's minds. I'd go even further and say make sure you leave plenty of gaps for them to fill in - every detail you 'miss' is an opportunity.