This is probably going to be an unpopular opinion, but I'm more baffled why game devs continue to implement inventory limitations at all. I have yet to see a game that wouldn't be significantly improved by just giving the player infinite inventory space.
Now we bring in what it adds to the game. Player choice and agency. As the player, you get to pick your own path. In any game with an inventory, you are going to get to pick which things you keep and which you don’t. This means you have to actively make choices according to your playstyle. In Deus Ex the question boils down to, do you take the big fuck off GEP gun or do you keep your inventory lean for lockpicks, pistols, SMGs, shotguns, and assault rifles so you can use whatever ammo you come across. In the first Deus Ex, ammo is very scarce. Thus having room for ammo and the tools to use it is very important. a GEP gun only really works on 4-6 enemies and the ammo is huge as well as the gun. Taking up around a third of the inventory, maybe more, depending on if you get inventory upgrades. This is an active playstyle choice.
Yeah, this is the usual justification for limited inventory space, but I'm not buying it. I don't think it adds choice at all, quite the opposite. You could just as easily make the choice to not take the GEP gun even if you had infinite inventory space, so contrary to what you said, the limitation doesn't actually add any choice. What it does do is remove the choice of taking the gun anyway "just in case" even if you're on a stealthy playthrough, and that has the knock-on effect of removing a whole bunch of other choices later down the line. You can't pull a "surprise, motherfucker" moment on some hapless enemy. You can't change your playstyle later if you discover that stealth really isn't as much fun as you thought it would be. And perhaps most importantly, you can't choose to keep playing stealthily despite having a big fuckoff gun. Choice only exists when alternatives exist. By taking those alternatives away and locking the player into one playstyle, you're not adding choice, you're diminishing it.
This is basically the same discussion that occurred around Skyrim, where Bethesda made the controversial design decision of getting rid of character classes in favor of completely free-form character leveling. Some people argued that this meant that you couldn't choose your playstyle anymore, and I always found that logic to be completely backwards. Of course you can still choose your playstyle, but now you make that choice by actually playing the game in a particular way rather than by clicking stuff in a menu at the beginning. Surely it's self-evident that this is better...?
Inventory limitations (and character classes) are therefore just one aspect of the larger question of whether or not to lock the player into their choices or allow them to play a jack-of-all-trades or change their playstyle later. Having played a lot of games from either side of that spectrum, I'm very firmly in favor of the latter. It's easy to point to Deus Ex, regarded by many as the best game ever made, as an example of how to do it right. But the reality is that most games aren't that good and don't offer multiple equally well fleshed out playstyles to choose from. What if the game is badly balanced and your chosen playstyle ceases to be viable halfway through? What if it turns out that the playstyle simply isn't as fun as it initially seemed? If the game falls into the former category of locking you into your choices, you're screwed and have to start over. That's just frustrating and disrespectful of the player's time. Game devs would be wise to eat their humble pie and allow a lot of leeway to the player not just because the player may have made some bad choices when playing the game but also because the devs themselves may have made some bad choices when making it. Only a perfect game is justified in being unforgiving, and precious few are.
Now you might say, "But Sordid, if you allow the players to have everything, they're going to optimize the fun out of the game and play an OP jack-of-all-trades!" And my response to that is, "So what?" It's their choice, let them. If adding choice were the goal as you claim, it should be a no-brainer. Yeah, the temptation to optimize is strong, and sticking to your chosen playstyle can be difficult at times, but nothing worthwhile is easy. And some people genuinely enjoy being an OP jack-of-all-trades. Why take that choice away from them when it doesn't add any choice to anyone else?
You're right about some of the effects that limited inventory space has, like forcing the player to prioritize what they take or to vary their approach instead of just using one gun throughout the entire game, but that's kinda my point. I was responding to your statement that limited inventory adds player choice, which is not true. These effects are the opposite, they're restrictions on the player's choice, specifically on the player's ability to choose something that might be considered boring or overpowered. I don't think such restrictions are necessary, but more to the point, I think players deserve to know the true nature and purpose of them. Don't go around saying that limited inventory space adds choice. Tell your players the truth, that it's a leash that restricts their choice and that you're keeping them on because you don't trust them not to ruin the game for themselves. And that you're withholding that choice from those who would actually enjoy that playstyle.
You're also correct that skill systems are another significant factor in locking the player into their choices. That's what I meant when I mentioned character classes and praised Skyrim's lack of them. You're right that switching playstyles in Skyrim requires some effort, but not nearly as much as leveling up in the first place. A high-level character has all kinds of equipment, abilities, and resources they can use to speed up the process (e.g. just using their wealth to pay for skill training), and I really think Bethesda is onto something with this system. It doesn't let you be great at everything right from the start (too OP), nor does it rigidly lock you into a single playstyle (too restrictive). Other games have also implemented skill systems that achieve a similar effect (Kenshi, Dark Souls 2).
As for the question of how to handle inventory itself, here's the thing. The issue I have with limited inventory space is that most games slavishly implement it without giving a lot of thought to what happens when you run out. Because that's the whole point, right? Your inventory capacity is limited so that you can run out, either of inventory capacity itself or of the resources that you carry in your inventory. At which point you need to stop doing what you're doing and replenish, and the usual ways of handling that are just uninspired and annoying: If you run out of inventory space, you need to make a trip to town and sell your vendor trash before you get back to killing mobs. If you run out of ammo, you need to stop fighting and go pick up some more. Both of those feel like a chore, they're a needless, boring interruption of gameplay.
And it's not an easy problem to solve, very few games even make an attempt. Some adjust resource drops to suit your needs, dropping you ammo/resources that you're short on and not dropping stuff that you have plenty of. That works, but it cheapens the experience, because the entire resource economy is fake. The new Dooms have the right idea of making the player engage enemies even more aggressively instead of backing off to replenish ammo, but even there the glory kills are basically short first-person cutscenes that interrupt the gameplay (which is the reason I haven't played those games, I find the idea of having to frequently do finishers with canned animations off-putting). The best solution I've seen, which unfortunately doesn't seem to be very popular, is Mass Effect 1's overheat meter, which is basically equivalent to regenerating ammo. It prevents you from just holding the button down as you might if you had an enormous/infinite ammo reserve, but it also doesn't force you to stop fighting when you run out and have to wait for your gun to cool down, since you can use that time to reposition, throw grenades, use abilities, etc. It's a shame this system hasn't caught on.
Similar issues exist around vendor trash hauling. Some games manage to do away with it completely. For example, The Saboteur and The Bard's Tale both show you what loot you found and then immediately convert it into money as if you'd sold it to a vendor, only you didn't actually have to make the trip. It's basically just a currency drop with some flavor text, and the games are in no way worse off for not forcing the player to periodically visit a vendor. But that only works in these games because actual usable items and weapons are very few and very simplistic, it's a much tougher nut to crack in games with more complex itemization. Path of Exile actually uses inventory management issues as part of its monetization model, where buying stash tabs is technically optional, but inventory management is hell without them. The Nioh series, which is a soulslike with ARPG-esque itemization, is a prime example of devs not giving any thought to what happens when you run out. They almost give you an unlimited inventory but not quite. You can carry 600 items on you and store 4000 more in your stash, which is a lot. You won't have to worry about inventory limitations for a long, long time, quite possibly all the way to endgame. But what happens when you do finally run out of space? You probably just give up on the game outright, because now it expects you to manually sort through almost five thousand items one by one, and fuck that, right? Unlike the ammo problem, which IMO is solved quite well by ME1, I can't recall a game that has a really good solution to the vendor trash problem.
Oh, and as for players finding your game too easy and boring due to unlimited inventory and leaving bad reviews after playing for five minutes, I don't think that's a realistic concern. You can make games challenging and interesting just fine without rigidly locking players into their choices. Skyrim's unlimited leveling doesn't seem to have made a dent in its success, and Nioh also isn't regarded as easy despite having an inventory limit that you're unlikely to reach for a few hundred hours.
I'm glad I was able to give you some food for thought. For my part, I can see how some games might benefit from inventory limitations. Certainly if you're making any kind of realistic simulation, infinite inventory space would be inappropriate. Allowing the player to amass huge stockpiles of consumables is probably also not a good idea (though this issue can also be attacked from the other direction, by simply not giving the player too many consumables in the first place). After further consideration, it seems clear to me a distinction should be drawn between inventories used to hold items the player uses, such as weapons and consumables, and inventories used to hold vendor trash. I still firmly believe that the latter should be unlimited and the player should not be forced to interrupt the fun they're having slaying monsters just to make room for more loot. It doesn't help that most games don't in fact make such a distinction and just have one inventory for everything (Kingdoms of Amalur being the only exception I can think of at the moment).
If you have infinite inventory space, then you need a way to navigate through infinite items. Towards the end of the game, a player could easily have nearly every item in the game. For some games, that would be fine, but for many, that would make the list of items prohibitively long. Filtering and searching would help, but if you're looking for an item that you forgot the name of, a search doesn't necessarily do much.
Then there's balance reasons. Some games use their inventory system to limit the player, making sure they don't start a level with enough health potions and grenades to cheese every fight.
In survival games, a finite inventory sets the gameplay loop: you go exploring/mining and then return to base, drop off your stuff and head out again. It makes your base valuable, if only because that's where you keep most of your resources and moving would be hard. It also gives the player a break from one task. I played a Minecraft mod that gave me an effectively infinite inventory. I went mining for so long that it started to feel like an awful slog. Because my mine shafts went on too long, getting back was itself a hassle. When I reverted back to a more typical inventory size, I could feel how a full inventory breaks up the grind and prevents mining from getting out of hand.
Filtering and searching would help, but if you’re looking for an item that you forgot the name of, a search doesn’t necessarily do much.
Keywords are the answer. If the player forgets whether the healing item is called "medkit", "first aid", or "bandage", just let them search for "heal".
Some games use their inventory system to limit the player, making sure they don’t start a level with enough health potions and grenades to cheese every fight.
That's better solved by just not giving the player that many to begin with. Some games adjust their item drops based on what you already have, so if you're running low on pistol ammo, they'll give you some. If you already have five grenades, you won't find any more until you use some. That achieves the same goal without limiting your ability to pick up different types of items.
In survival games, a finite inventory sets the gameplay loop: you go exploring/mining and then return to base, drop off your stuff and head out again. It makes your base valuable, if only because that’s where you keep most of your resources and moving would be hard.
That's exactly one of the things that I hate about limited inventory space. It's effectively the same as hauling vendor trash, and that's something that I view as an unnecessary interruption of gameplay. "You've had fun killing mobs for five whole minutes, time to do a chore!" Yeah, no, that can fuck right off. All the way off.
I played a Minecraft mod that gave me an effectively infinite inventory. I went mining for so long that it started to feel like an awful slog.
Plus you can put the items into categories. Fallout/Skyrim games still have a weight limitation but even in the middlegame most players are carrying hundreds of items. The devs helped alleviate frustration by categorizing everything. Need health? Go to the "aid" tab. Need new armor? Click the "armor tab." Etc
I love how modded minecraft has infinite different options for inventory management. Enderchest in a pouch, automatic pickup and sorting, intelligent sorting buttons, autopopulating hotbars, auto delete, entire mods like AE2 just to organize your chest system with a searchbar. It's a nice commentary on how difficult balancing a game around collecting limited resources and accessing them can be. The fact that certain people like certain inventory mods is telling.