People tend to ignore that spells are very difficult and intricate things. Unless you have innate or divinely inspired spell casting, every spell is simply too much to retain for regular people forcing spells to work for them. This goes for both components and the spell itself. If you get even the smallest detail wrong, it could be disastrous.
I think of it like an engineer or some such. Sure they have some approximate knowledge of most formulas they need for their work and they will know some by heart. But building a bridge from memory, just because they have been building bridges for twenty years won't be possible without proper preparation and work.
Also why wizards are the only class that can learn all spells. They are the only nerds on the block willing to put in the work.
I like to think of it like rubix cubes. Those guys that memorized the algorithm and can solve a rubix cube in seconds? That's what a wizard is doing for every spell, and each one is a different algorithm, shape, number of faces, etc.
A rubix is always the same cube.
Imagine if you changed what the cube looked like based on the wind, the trajectory of the object you're trying to hit, or the alignment of the stars, and if you fuck up it's exploding in your face. Also you have a troll charging at you trying to hit you, and you only have 6 seconds to solve it. And you need like 10 other types of puzzles memorized too.