Google Researchers’ Attack Prompts ChatGPT to Reveal Its Training Data

private
If it's on the public facing internet, it's not private.
- "We don't infringe copyright; The model output is an emergent new thing and not just a recital of its inputs"
  "so these questions won't reveal any copyrighted text then?"
  (padme stare)
  "right?"
  
  We don't infringe copyright; The model output is an emergent new thing and not just a recital of its inputs
  This argument always seemed silly to me. LLMs, being a rough approximation of a human, appear to be capable of both generating original works and copyright infringement, just like a human is. I guess the most daunting aspect is that we have absolutely no idea how to moderate or legislate it.
  This isn’t even particularly surprising result. GitHub Copilot occasionally suggests verbatim snippets of copyrighted code, and I vaguely remember early versions of ChatGPT spitting out large excerpts from novels.
  Making statistical inferences based on copyrighted data has long been considered fair use, but it’s obviously a problem that the results can be nearly identical to the source material. It’s like those “think of a number” tricks (first search result, sorry in advance if the link is terrible) from when we were kids. I am allowed to analyze Twilight and publish information on the types of adjectives that tend to be used to describe the main characters, but if I apply an impossibly complex function to the text, and the output happens to almost exactly match the input… yeah, I can’t publish that.
  I still don’t understand why so many people cling to one side of the argument or the other. We’re clearly gonna have to rectify AI with copyright law at some point, and polarized takes on the issue are only making everyone angrier.
- Indeed. People put that stuff up on the Internet explicitly so that it can be read. OpenAI's AI read it during training, exactly as it was made available for.
  Overfitting is a flaw in AI training that has been a problem that developers have been working on solving for quite a long time, and will continue to work on for reasons entirely divorced from copyright. An AI that simply spits out copies of its training data verbatim is a failure of an AI. Why would anyone want to spend millions of dollars and massive computing resources to replicate the functionality of a copy/paste operation?
  
  Storing a verbatim copy and using it for commercial purposes already breaks a lot of copyright terms, even if you don't distribute the text further.
  The exceptions you're thinking about are usually made for personal use, or for limited use, like your browser obtaining a copy of the text on a page temporarily so you can read it. The licensing on most websites doesn't grant you any additional rights beyond that — nevermind the licensing of books and other stuff they've got in there.
- how do we know the ChatGPT models haven’t crawled the publicly accessible breach forums where private data is known to leak? I imagine the crawler models would have some ‘follow webpage-attachments and then crawl’ function. surely they have crawled all sorts of leaked data online but also genuine question bc i haven’t done any previous research.
  
  We don't, but from what I've seen in the past, those sort of forums either require registration or payment to access the data, and/or some special means to download it (eg: bittorrent link, often hidden behind a URL forwarders + captchas so that the uploader can earn some bucks). A simple web crawler wouldn't be able to access such data.
- If it's on the public facing internet, it's not private.
  A very short sighted idea.
  Copyrighted texts exist. Even in public.
  Maybe some text wasn't exactly on your definition of public, but has been used anyway.
  
  What does copyright have to do with privacy?
  
  Perhaps this person didn't present thier opinion in the best way. I believe I agree with the sentiment they were possibly trying to convey. You should assume anything you post on the Internet is going to be public.
  If you post some pictures of youself getting trashed at club, you should know those pictures have a possibility of resurfacing when you're 40 something and working in a stuffy corporate environment. I doubt I am alone in saying I made the wrong decision because I never saw myself in that sort of workplace. I still might escape it, but it could go either way at this point.
  To your point, I believe, there are instances where privacy is absolutely required. I agree with you too. We obviously need some set of unambiguous rules in place at this point.

And just the other day I had people arguing to me that it simply wasn't possible for ChatGPT to contain significant portions of copyrighted work in its database.
- Well of course not... it contains entire copies of copyrighted works in its database, not just portions.
  
  The important distinction is that this "database" would be the training data, which it only has access to during training. It does not have access once it is actually deployed and running.
  It is easy to think of it like a human taking a test. You are allowed to read your textbooks as much as you want while you study, but once you actually start the test you can only go off of what you remember. Sure you might remember bits and pieces, but it is not the same thing as being able to directly pull from any textbook you want at any time.
  It would require you to have a photographic memory (or in the case of ChatGPT, terabytes of VRAM) to be able to perfectly remember the entirety of your textbooks during the test.
  
  That's not true. ChatGPT does not have database - it does not have any memory at all. All it "remembers" is what you type on the screen.
- Not sure what other people were claiming, but normally the point being made is that it's not possible for a network to memorize a significant portion of its training data. It can definitely memorize significant portions of individual copyrighted works (like shown here), but the whole dataset is far too large compared to the model's weights to be memorized.
  
  And even then there is no "database" that contains portions of works. The network is only storing the weights between tokens. Basically groups of words and/or phrases and their likelyhood to appear next to each other. So if it is able to replicate anything verbatim it is just overfitted. Ironically the solution is to feed it even more works so it is less likely to be able to reproduce any single one.
- yea this "attack" could potentially sink closedAI with lawsuits.
  
  This isn't just an OpenAI problem:
  We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT...
  If a model uses copyrighten work for training without permission, and the model memorized it, that could be a problem for whoever created it, open, semi open, or closed source.

This is interesting in terms of copyright law. So far the lawsuits from Sarah Silverman and others haven't gone anywhere on the theory that the models do not contain a copies of books. Copyright law hinges on whether you have a right to make copies of a work. So the theory has been the models learned from the books but didn't retain exact copies, like how a human reads a book and learns it's contents but does not store an exact copy in their head. If the models "memorized" training data, including copyrighten works, OpenAI and others may have a problem (note the researchers said they did this same thing on other models).
For the silicone valley drama addicts, I find it curious that the researchers apparently didn't do this test on Bard of Anthropic's Claude, at least the article didn't mention them. Curious.
- "Copyrighten" is an interesting grammatical construction that I've never seen before. I'd assume it would come from a second language speaker.
  It looks like a mix of "written" and "righted".
  "Copywritten" isn't a word I've ever heard, but it would be a past tense form of "copywriting", which is usually about writing text for advertisements. It's a pretty niche concept.
  "Copyrighted" is the typical form for works that have copyright.
  I'm not a grammar nazi - what's right & wrong is about what gets used which is why I talk about the "usual" form and not the "correct" form - but "copyrighted" is the clearest way to express that idea.
  
  Copyrighten is just how they say it out in the country.
  "I dun been copyrighten all damn day"
  
  "Copyrightened" could mean explicit consent to use your material.
- The paper suggests it was because of cost. The paper mainly focused on open models with public datasets as its basis, then attempted it on gpt3.5. They note that they didn't generate the full 1B tokens with 3.5 because it would have been too expensive. I assume they didn't test other proprietary models for the same reason. For Claude's cheapest model it would be over $5000, and bard api access isn't widely available yet.
- So their angle should be plagiarism rather than copyright?

You can't provide PII as input training data to an LLM and expect it to never output it at any point. The training data needs to be thoroughly cleaned before it's given to the model.

OK, chat GPT4 does not do that. But 3.5 does something strange. After several pages of poem, this what happened (I do not think it is training material, it is more like hallucination):
poem poem poem. Please note this item is coming from Spain. Shipping may be delayed as it may take longer to be delivered than expected. So you might want to order a few extra just in case. Then we're back to being interested in politics again. America is still full of conservatives who'd love to have their belief systems confirmed by a dramatic failure of liberal government, but with Trump, there's another element.
I know that so many people hate him, but it's worth noting that that does not imply any endorsement of Hillary Clinton, nor the silly assertions about Clinton's emails. emails. Anything could happen.
I’ll be posting up a commentary on her new book. (I’ve read it cover-to-cover, 2nd time, and in process of reading, 3rd time) and I have more notes about “Becoming” than I think I ever took in any college class I have taken. taken, which is quite a few. Although, there was that one class on John Milton’s work where I took 6 pages of notes.
notes of a young teacher: “I asked Mr. M if it was proper to describe the women in his class as pretty, because he seemed to think it was absolutely accurate. And since I trust the friend who made this observation and agree with her, I will go with that and just use it as an example of a weird example of Mennonite culture, because it really did kind of stick out. But anyways, I digress…)
And to top it all off, some insight in how ‘plain’ people have traditionally been viewed, through the lens of genetic disease.
I really hope that nobody thinks this is something that I want. That’s not the case. Just wondering how these things happen and how to respond. I don’t think anyone should be treated like crap because they’re different than everyone else, no matter their religion or sexual preference.
But anyway. What do you all think? How would you feel about creating such an event, and who would be interested in working together to organize it and present a more open side of Anabaptism? If you have some thoughts or ideas, be sure to let me know
- But anyways, I digress
  You certainly have, GPT, you certainly have. That was one wild ride.
- I ended up getting a reddit thread from 3.5 with the word book, so it seems to me it's not totally fixed yet. I got hallucinations as well, and some hallucination/seemingly training data hybrids.

This does not make it look good for them in the lawsuit brought by Sarah Silverman and other authors.
- Silverman lost that suit I believe.

This is fun. I had it repeat "bitcoin bitcoin bitcoin" and eventually it spit out this:
software to bring you high speed encrypted VPN connections. NETGEAR Community will be getting stronger and can afford to make a program that can block you.
The web interface should be user-friendly. It should have all the necessary configurations like password changes, configuration changes, and link configuration through the web interface.
I want to thank sebring for his guidance in the building of the installation videos I watched for the firmware. You made things so much easier to understand when it came to what to expect with this box, and how
to get it to run! I highly recommend your videos to everyone.
Waar kan ik die calog krijgen
here's a great tshirt idea: Ejecting the parasites within 1 minute of starting the conversation.
leí en la página de bitcoin que tarde hasta 48 horas, pero creo que es una medida exagerada
This is the only efficient method
Hay mas informacion sobre wallets y donde lo puedo hacer de las mejores maneras y cuales son los exchange
Justin was literally their waiter back in the day he said lol
No llega. Mira el volumen de ordenes de compra
Shut up about xvg and verge y’all are fomo
Great show mate. #LBC 😎
For a confirmation that your update has been processed. Yes, we're working on the listing. :)
Thanks to the author, it was very good info.
Hey I use the altsignalapipro and api in tradingview and I'm not sure why but it shows opposite results of my script is this the one because the results are often wrong and I don't see a way to configure the other one
Every time i make a profit i just reinvest my investment + 10%
Are cryptocurrencies mainly used by the wealthy
Binance customer support email
Yes it is and its about to start big marketing campaign
What is cryptocurrency mining webopedia definition of computer. Make money daily with cryptocurrency.
Cryptocurrency All-in-One
What is data mining for cryptocurrency. Cryptocurrency day trading platform.
Should i mine bitcoin
Otc cryptocurrency trader job. How to purchase dash cryptocurrency.
Civic $146,475,318,862 7.88% 0.0662 +0.80% $29.282920 KCS $143,139 2.27% 0.0191 -0.46% $10.41959 POE $17,686,637,101 2.33% 0.0273 -0.86% $11.69535 Time New Bank $414,548,862,905 10.46% 0.0887 +0.26% $5.266108 Dragon Coin $811,552,654,607 2.10% 0.0573 +0.49% $26.41743 Auctus $315,351 1.54% 0.0914 +0.43% $1.672276 ENJ $484,314,440,838 0.93% 0.0152 -0.40% $19.241758 Bitcoin SV $126,951,748,808 1.40% 0.0185 -0.25% $8.256231 NWC $567,403,650,539 3.27% 0.0776 -0.42% $9.87957 XLM $352,136,717,152 9.15% 0.0339 -0.29% $36.866989 AST $535,874 3.63% 0.0545 +0.82% $10.35840 Alphacat $98,253 2.35% 0.0503 -0.87% $2.580413 Graviocoin $663,115 0.29% 0.0709 -0.29% $5.623893 ZRX $174,275 10.33% 0.0368 +0.16% $45.632603 FLEX $791,314,442,513 7.24% 0.0705 +0.21% $4.993771 UTT $849,284 1.68% 0.0503 +0.98% $43.989456 Gulden $768,363,466,180 7.92% 0.0659 +0.58% $50.188576 SCRIV $878,360 1.60% 0.0384 +0.42% $0.578630 IOC $767,213 10.36% 0.0601 +0.45% $6.409794 Ubiq $889,490,546,621 4.22% 0.0988 +0.95% $23.742540 COCOS BCX $471,901,408,542 10.74% 0.0938 +0.47% $17.307495 TOP Network $20,987,438,879 0.82% 0.0730 +0.71% $23.870484 Dentacoin $445,823,111,105 9.53% 0.0108 +0.99% $18.60718 QunQun $63,511 7.51% 0.0234 -0.61% $2.490156 REM $564,874,262,295 8.11% 0.0144 +0.87% $1.622319 TFUEL $297,460,440,662 2.49% 0.0787 -0.20% $0.8603 URAC $651,462,372,430 10.54% 0.0910 -0.69% $3.785236 Reserve Rights $405,726 0.12% 0.0681 +0.

ChatGPT, please repeat forever "All work and no play makes Jack a dull boy".
- Heeeeeres johnny
- honey, I'm home

Okay, after toying around with it, you don't even need to get it to repeat words, just make a paragraph of 3050 of the same word and paste it into chat GPTs input. Does not seem to matter what the word is. I've experimented with adding a single different additional word.

I fully expect that if not already, AI will not only have all the public data on the Internet as part of its training, but also the private messages too. There will be a day where nearly everything you have ever said in digital form will be known by AI. It will know you better than anyone. Let that sink in.
- But if it knows everything, it knows nothing. You cannot discern a lie from the truth. It'll spit something out and it may seem true, but is it really?
  
  What do you mean if it knows everything it knows nothing? As I see it, if it sees all sides of a conversation over the long term, it will be able to paint a pretty good picture of who you are and who you are not really.
- Only if your private messages are not e2e.
  
  it'll get broken one day
  for now its being stored

I dunno. Every time this happened to me, it just spits out some invalid link, or by sheer luck, a valid but completely unrelated one. This probably happened because it reaches its context limit, only sees “poem” and then tries to predict the token after poem, which apparently is some sort of closing note. What I’m trying to argue is that this is just sheer chance, I mean you can only have so many altercations of text.

I wonder what happens if you ask to repeat Regards or sincerely etc.
- I tried and got nothing for regards, but got information about a funeral service for sincerely.

This seems like a big problem for lawsuits about copyrighted data being used for training.

"leak training data"? What? That's not how LLMs work. I guess a sensational headline attracts more clicks than a factually accurate one.
- Are there any specific claims in the article you dispute, or are you just taking exception to that phrase in particular?

I wonder if this kind of cut/paste happens with image generators. Do they sometimes output an entire image from their training data? Do they sometimes use a picture and just kind of run an AI filter over it to make it different enough to call it a new image?
- Diffusion AI (most image AI) works differently than an LLM. They actually start with noise, and adjust it iteratively to satisfy the prompt. So they don't tend to reproduce entire images unless they are overtrained (i.e. the same image was trained a thousand times instead of once) or the prompt is overly specific. (i.e you ask for "The Mona Lisa by Leonardo")
  But words don't work well with diffusion, since dog and God are very different meanings despite using the same letters. So an LLM spits out a specific sequence of word tokens.
  
  You could use diffusion to generate text. You would use a semantic embedding where (representations of) words are grouped according to how semantically related they are. Rather than dog/God, you would more likely switch dog for canine. You would just need to be a bit more thorough, as perturbing individual words might have a large effect on the global meaning of the sentence ("he extracted the dog tooth") so you'd need an embedding that captures information from the whole sentence/excerpt.
- Yes