If I understand LLMs right, they have a maximum prompt length, but can be trained on any amount of text data.
The only way to add knowledge that doesn’t fit into a prompt, is to put it in the training data then re-train.
But, you could describe some sort of algorithm that it can use to sleuth out data using API calls, and it would then have access to lots more up-to-date data than can fit in a prompt. Except the body of the response would all have to become part of a prompt.
But the whole dataset it has access to doesn’t have to be mentioned in the conversation, so doesn’t have to be part of the prompt. Ultimately you don’t want your AI assistant telling you everything it knows in each interaction, just to access some slice of your data world, make changes to it, then eventually get you an answer or a report.
My sister had a marketing gig that she got let go of, because she was so good at selling her product that the company said they established strong markets in her area and didn't need the publicity anymore.
Exactly that: too good at her job.