Exactly, it's not something designed to output facts, it's designed to output the most likely set of words.

Piracy won't be dying any time soon. Every software has its flaws. Sure, pirates will hit a couple rough waves, maybe a storm or two. But piracy will continue.

Amateur! I write my code down on a piece of paper, scan it in, send it to my computer through email, then make a custom-built AI read the paper and print it in the terminal!

Ok? I feel like people don't understand how these things work. It's an LLM, not a superintelligent AI. It's not programmed to produce the truth or think about the answer. It's programmed to paste a word, figure out what the most likely next word is, paste that word, and repeat. It's also programmed to follow human orders as long as those order abide by its rules. If you tell it the sky is pink, then the sky is pink.

put that map away, we need to keep the secret country hidden, you never know who could be reading this