Alchemist's Fire is basically the Molotov Cocktail of 5e, so I'd just use that.
1d4 Fire damage per round unless they take an action to put the fire out seems pretty reasonable to me. Puts it on par with a shortsword at the very least.
Fireball's damage is insane (the designers intentionally made it deal more damage than spells of the same level "because it's an iconic spell"), so I wouldn't really use it as a baseline for balancing anything, personally.
Alchemist's Fire also exists in 2.0, 2.5, 3.0, and 3.5. I dunno about WoW with dice 4th edition, or 1st edition. My parents had both the red and blue boxes, but we read the rules and immediately went for AD&D cause that was more flexible when we started the family game in Dragonlance.
How many videos have you seen of guys crit fail throwing a brick through a window and hitting themselves? That shits real life and understandable when you’re terrible at what you’re doing.
Sure, but the question was "how you you as a DM rule this?" Unless you're running a game where the players are all commoners, it's safe to assume there's a less than 1 in 20 chance of hitting yourself with a thrown weapon
You should take your own advice, your players who play martials will have a lot more fun if they aren't dealing damage to themselves one in every twenty hits