Exactly this. It was only two generations ago when idTech was an open world engine, id can and have made it to do whatever they want and to suggest that despite Bethesda money (let alone MICROSOFT money) id couldn't make a better engine with similar development workflows as Creation is just dishonest to suggest.
It's a shame idTech is no longer released publicly. It would've been amazing to see what people could do with the beast of an engine that powered DOOM Eternal, especially modders.
I assume you're talking about Rage, which had an open world map, but no where near the level of simulation systems as a Bethesda game. In fact I remember back at the time most of us saying the map was pointless as it was just a way to travel between levels with nothing to do in it.
There are no "levels of simulation systems" in Starfield. NPCs don't even have schedules in this game, they literally just stand around in the same spot 24/7.
"keeping track of lots of variables" doesn't cost CPU time though, since nothing that isn't on the same map as you is ever relevant for anything. Their engine just fucking sucks.
Keeping track of variables doesn't use CPU time? Ok man. I'm all for hating on Bethesda's shitty engine but that's just not true. At the very least it does track what NPCs are doing off screen which is how they end up at your ship when you tell them to go there. They will actually walk to your ship if you don't get there first.
On the other hand it's basically guaranteed that Bethesda spent zero effort optimizing that. I bet it's the same code they ran for Skyrim.
You realise custom engines are built for specific game types right? iD Tech is great for creating high fedelity FPS games with linear levels and little environment interactivity. That's not what Bethesda make though.
They could do everything they usually do but better if they used Unreal. They don't need a custom engine. They just need an engine that isn't over 2 decades old with a bunch of shit taped to it to make it look modern. Not to mention, ID already did make a custom built engine that handles much of what Bethesda RPGs do when they made RAGE. They could have used that, with the only issue being learning it. Not sure what their turnover rate is like... maybe they're just too used to GameBryo/Creation to be able to switch now. It might take too long to learn anything new. Plus it would have to be able to have a toolset. If they didn't release those easy to use modding tools, there could be rioting in the streets.
As far as I know, Bethesda are unusual in modern Devs in that they have a small team for the size of game they make, but they have strong retention of staff so have huge amounts of institutional knowledge about how they do things. Shifting to a new engine would basically mean starting from scratch on a company level. Unlike Ubisoft or Activison, they can't just throw several thousend Devs at a game to brute force the development either.
But that's their biggest problem. There's no reason for them to have a small unchanging team. It's very very obvious that they never get an influx of new ideas. Starfield feels like it was made in 2016 and the optimization effort is comically bad. The writing is still mostly boring, campy and naive like it was written by a 15 year old Mormon. The facial animations are incrementally better than fallout but still noticeably worse than much older games like Witcher 3. I could go on.
It's not a bad game at all but it could've been so much better if Bethesda execs weren't greedy cheapasses and the dev team was open to changing their process.
This why Bethesda needs to be criticized instead of constantly getting fellated by fanboys. ES6 will be an outdated mess because Bethesda never sees any feedback except over the top praise for half-assing their games.
Fanboys downvote you but you are right, even if I love the fallout franchise, the same gameplay loop, the same engines, potato faces in 2023, outdated animations... etc, right now I would prefer Microsoft to force obsidian to take care of the next fallout, and ban Todd Howard for ever putting one foot in the dev took, even in the building. He can go fuck himself and his shit engine.
ID tech is nowhere near flexible enough for something like Starfield or even Skyrim. It's partially the reason why it's so efficient. It simply isn't fit for the task.
And the Bethesda developers are intimately familiar with Creation Engine, achieving the same level of productivity with something new will take a long time. Switching the engine is not an easy thing.
Not to say that Creation Engine isn't a cumbersome mess. It has pretty awful performance, stability and is full of bugs, but on the other hand it's extremely flexible which has allowed its games to have massive mod communities.
If Bethesda can't take the time to do it then who can? People act like they're some small time developer but they're not. They simply refuse to expand their dev team to do things like a redesign.
Creation engine is not going to hold up well for another 6 years, there's no way their cell loading system will be considered acceptable by the time ES6 comes out. The amount of loading screens in Starfield is insane for a modern game. This company needs new talent badly.
I know they don't want to switch, but it would be worth it to make the swap to something like unreal, even if it takes a few years of customization to get the open world stuff right. Creation Engine just feels so old.