LLMs can’t reason — they just crib reasoning-like steps from their training data

"sigh"

(Preface: I work in AI)

This isn't news. We've known this for many, many years. It's one of the reasons why many companies didn't bother using LLM's in the first place, that paired with the sheer amount of hallucinations you'll get that'll often utterly destroy a company's reputation (lol Google).

With that said, for commercial services that use LLM's, it's absolutely not true. The models won't reason, but many will have separate expert agents or API endpoints that it will be told to use to disambiguate or better understand what is being asked, what context is needed, etc.

It's kinda funny, because many AI bros rave about how LLM's are getting super powerful, when in reality the real improvements we're seeing is in smaller models that teach a LLM about things like Personas, where to seek expert opinion, what a user "might" mean if they misspell something or ask for something out of context, etc. The LLM's themselves are only slightly getting better, but the thing that preceded them is propping them up to make them better

IMO, LLM's are what they are, a good way to spit information out fast. They're an orchestration mechanism at best. When you think about them this way, every improvement we see tends to make a lot of sense. The article is kinda true, but not in the way they want it to be.