Skip Navigation
Remote View
Technology @lemmy.world
ElectroVagrant @lemmy.world

This new data poisoning tool lets artists fight back against generative AI

A new tool lets artists add invisible changes to the pixels in their art before they upload it online so that if it’s scraped into an AI training set, it can cause the resulting model to break in chaotic and unpredictable ways.

The tool, called Nightshade, is intended as a way to fight back against AI companies that use artists’ work to train their models without the creator’s permission.
[...]
Zhao’s team also developed Glaze, a tool that allows artists to “mask” their own personal style to prevent it from being scraped by AI companies. It works in a similar way to Nightshade: by changing the pixels of images in subtle ways that are invisible to the human eye but manipulate machine-learning models to interpret the image as something different from what it actually shows.

130 comments

  • Here's the paper: https://arxiv.org/pdf/2302.04222.pdf

    I find it very interesting that someone went in this direction to try to find a way to mitigate plagiarism. This is very akin to adversarial attacks in neural networks (you can read more in this short review https://arxiv.org/pdf/2303.06032.pdf)

    I saw some comments saying that you could just build an AI that detects poisoned images, but that wouldn't be feasible with a simple NN classifier or feature-based approaches. This technique changes the artist style itself to something the AI would see differently in the latent space, yet, visually perceived as the same image. So if you're changing to a different style the AI has learned, it's fair to assume it will be realistic and coherent. Although maaaaaaaybe you could detect poisoned images with some dark magic tho, get the targeted AI then analyze the latent space to see if the image has been tampered with

    On the other hand, I think if you build more robust features and just scale the data this problems might go away with more regularization in the network. Plus, it assumes you have the target of one AI generation tool, there are a dozen of these, and if someone trains with a few more images in a cluster, that's it, you shifted the features and the poisoned images are invalid

    • Trying to detect poisoned images is the wrong approach. Include them in the training set and the training process itself will eventually correct for it.

      I think if you build more robust features

      Diffusion approaches etc. do not involve any conscious "building" of features in the first place. The features are trained by training the net to match images with text features correctly, and then "just" repeatedly predict how to denoise an image to get closer to a match with the text features. If the input includes poisoned images, so what? It's no different than e.g. compression artifacts, or noise.

      These tools all try to counter models trained without images using them in the training set with at most fine-tuning, but all they show is that models trained without having seen many images using that particular tool will struggle.

      But in reality, the massive problem with this is that we'd expect any such tool that becomes widespread to be self-defeating, in that they become a source for images that will work their way into the models at a sufficient volume that the model will learn them. In doing so they will make the models more robust against noise and artifacts, and so make the job harder for the next generation of these tools.

      In other words, these tools basically act like a manual adversarial training source, and in the long run the main benefit coming out of them will be that they'll prod and probe at failure modes of the models and help remove them.

      • Just to start with, not very experienced with neural networks at all beyond messing with openCV for my graduation project.

        Anyway, that these countermeasures expose "failure modes" in the training isn't a great reason to stop doing this, e.g. scammers come up with a new technique, we collectively respond with our own countermeasures.

        If the network feedbacks itself, then cool! It has developed its own style, which is fine. The goal is to stop people from outright copying existing artists style.

    • Haven't read the paper so not sure about the specifics, but if it relies on subtle changes, would rounding color values or down sampling the image blur that noise away?

      • Wondering the same thing. Slight loss of detail but still successfully gets the gist of the original data.

        For that matter, how does the poisoning hold up against regular old jpg compression?

        Eta: read the paper, they account for this in section 7. It seems pretty robust on paper, by the time you've smoothed out the perturbed pixels, youve also smoothed out the image to where the end result is a bit of a murky mess.

  • I remember in the early 2010s reading an article like this one on openai.com talking about the dangers of using AI for image search engines to moderate against unwanted content. At the time the concern was CSAM salted to prevent its detection (along with other content salted with CSAM to generate false positives).

    My guess is since we're still training AI with pools of data-entry people who tag pictures with what they appear to be, so that AI reads more into images than their human trainers (the proverbial man inside the Iron Turk).

    This is going to be an interesting technology war.

  • Ooo, this is fascinating. It reminds me of that weird face paint that bugs out facial-recognition in CCTV cameras.

    • Or the patterned vinyl wraps they used on test cars that interferes with camera autofocus.

  • I generally don't believe in intellectual property, I think it creates artificial scarcity and limits creativity. Of course the real tragedies in this field have to do with medicine and other serious business. But still, artists claiming ownership of their style of painting is fundamentally no different. Why can't I paint in your style? Do you really own it? Are you suggesting you didn't base your idea mostly on the work of others, and no one in turn can take your idea, be inspired by it and do with it as they please? Do my means have to be a pencil, why can't my means be a computer, why not an algorythm? Limitations, limitations, limitations. We need to reform our system and make the public domain the standard for ideas (in all their forms). Society doesn't treat artists properly, I am well aware of that. Generally creative minds are often troubled because they fall outside norms. There are many tragic examples. Also money-wise many artists don't get enough credit for their contributions to society, but making every idea a restricted area is not the solution. People should support the artists they like on a voluntary basis. Pirate the album but go to concerts, pirate the artwork but donate to the artist. And if that doesn't make you enough money, that's very unfortunate. But make no mistake: that's how almost all artists live. Only the top 0.something% actually make enough money by selling their work, and that's is usually the percentile that's best at marketing their arts, in other words: it's usually the industry. The others already depend upon donations or other sources of income. We can surely keep art alive, while still removing all these artificial limitations, copying is, was and will never be in any way similar to stealing. Let freedom rule. Join your local pirate party.

    • I generally don’t believe in intellectual property, I think it creates artificial scarcity and limits creativity. Of course the real tragedies in this field have to do with medicine and other serious business.

      But still, artists claiming ownership of their style of painting is fundamentally no different. Why can’t I paint in your style? Do you really own it? Are you suggesting you didn’t base your idea mostly on the work of others, and no one in turn can take your idea, be inspired by it and do with it as they please? Do my means have to be a pencil, why can’t my means be a computer, why not an algorythm?

      Limitations, limitations, limitations. We need to reform our system and make the public domain the standard for ideas (in all their forms). Society doesn’t treat artists properly, I am well aware of that. Generally creative minds are often troubled because they fall outside norms. There are many tragic examples. Also money-wise many artists don’t get enough credit for their contributions to society, but making every idea a restricted area is not the solution.

      People should support the artists they like on a voluntary basis. Pirate the album but go to concerts, pirate the artwork but donate to the artist. And if that doesn’t make you enough money, that’s very unfortunate. But make no mistake: that’s how almost all artists live. Only the top 0.something% actually make enough money by selling their work, and that’s is usually the percentile that’s best at marketing their arts, in other words: it’s usually the industry. The others already depend upon donations or other sources of income.

      We can surely keep art alive, while still removing all these artificial limitations, copying is, was and will never be in any way similar to stealing. Let freedom rule. Join your local pirate party.

      Reformatted for easier readability.

    • As an artist I agree. People are being so irrational with this.

130 comments