Can't.
Adversarial efforts are how some networks get trained.
this post was submitted on 11 Apr 2025
42 points (95.7% liked)
Solarpunk
6445 readers
13 users here now
The space to discuss Solarpunk itself and Solarpunk related stuff that doesn't fit elsewhere.
Join our chat: Movim or XMPP client.
founded 3 years ago
MODERATORS
I actually spent a few years of my life writing a whole software project for exactly this purpose, and I still think that it's the only practical way to solve this problem.
Called "Aletheia", it ditches the idea that software can spot a fake entirely and instead provides a way to guarantee the author of a piece of media. If you trust the source, you can trust the image/video/document/whatever.
If you're curious, here are a few relevant links:
If you trust the source, you can trust the image/video/document/whatever.
I think this is key. There have always been sources that have been incredibly trustworthy that people choose to ignore. Like AP News or Reuters for example. Sometimes they might make mistakes but as long as they keep fixing them in a timely manner that's what's important.
Yeah it's still possible they could post an AI image - but that's why you compare sources.
Besides that would ruin the credibility they've spent decades building.
Read the beginning of the white paper two - great work! I can tell you put care into this.
Is this similar to how the NFT was supposed to work?
Thanks! And no, this is absolutely nothing like NFTs.
NFTs require the existence of a blockchain and are basically a way of encoding a record of "ownership" on that chain:
Alice owns this: https://something.ca/...
If the image at that URL changes (this is called a rug pull) or a competing blockchain is developed, then the NFT is meaningless. The biggest problem though is the wasted effort in maintaining that blockchain. It's a crazy amount of resources wasted just to establish the origin.
Aletheia is much simpler: your private key is yours and lives on your computer, and your public key lives in DNS or on your website at a given URL. The images, videos, documents, etc. are all tagged with metadata that provides (a) the origin of the public key (that DNS record or your website) and a cryptographic proof that this file was signed by whomever owns the corresponding private key. This ties the file to the origin domain/site, effectively tying it to the reputation of the owners of that site.
The big benefit to this is that it can operate entirely offline once the public keys are fetched. So you could validate 1 million JPEG images in a few minutes, since once you fetch the public key, everything is happening locally.
Sounds much more similar to something like NOSTR. I've always like the simplicity of using a keypair as an identity.
Thanks for the explanation!
You should never have trusted images before generative AI either. Trace the source and only trust the image if the source is legitimate.
Does anyone know of any proof of legitimacy projects that are gaining traction? I can imagine news orgs being the first to be hit by this problem. Are they working on anything?
In short, no. There's no solution to this crisis of legitimacy and no way to validate that a work came from human hands that can't be counterfeited. Short of a Butlerian Jihad, it won't be going away either. I'm looking forward to the inevitable bubble pop that will kill the overinvestment in generative tech but the models will still be around even without big companies wasting billions to train new ones.
In short, it's long past time to start treating every day like April Fools Day.
Butlerian Jihad it is then!
Time lapses of drawing the image would be near impossible to replicate currently.
Only a matter of time and they're already good enough to fool the gullible.
My understanding is that the generator would have to make each frame from scratch and also keep track of the progress of the drawing.
They may have trained on a few Timelapse drawings, but that dataset is much smaller then the photographs and artworks in the models.
I’m sure it could happen, but I’m not sure there will be enough demand to bother.
Universal Basic Income is really the only answer. So we can make art for fun instead of as a means to survive.
The problem is Goodhart's Law; "Every measure which becomes a target becomes a bad measure". Implementing a verification system that depends on video evidence creates both an incentive to forge such videos and a set of labeled training data that grows more readily available as the system sees more use. The adversarial generative network is literally designed to evolve better scams in response to a similarly-evolving scam detector, there's no computational way around the need to have people involved to make sure things are what they're claimed to be.
Universal Basic Income would be a good start, but the fundamental problem is money as the primary organizing force of society.
Negative proof: the AI company signs it with their watermark.
Positive proof: the photographer signs it with their personal key, providing a way to contact them. Sure, it could be a fake identity, but you can attempt to verify and conclude that.
Cumulative positive and negative proof: on top of the photographer, news organizations add their signatures and remarks (e.g. BBC: "we know and trust this person", Guardian: "we verified the scene", Reuters: "we tried to verify this photo, but the person could not be contacted").
The photo, in the end, would not be just a bitmap, but a container file containing the bitmap (possibly with a steganographically embedded watermark) and various signatures granting or withdrawing trust.
Isn't that more like trusting your source though, which media companies either do or don't do already.
It would be a method of representing trust or distrust in a structured way that's automatically accessible to the end user.
The user could right-click an image, pick "check trust" from a menu, and be presented with a list of metainfo to see who has originally signed it, and what various parties have concluded about it.
The final output image is just a grid of pixels, just like any other image. Assuming the image has no metadata or has been stripped of metadata, how do you really tell the difference in the first place?
Sure, you can look for JPEG artifacts and chromatic noise and all, but it's pretty easy to overlay that on top of an AI generated image to make it appear more legitimate at a passing glance.
I really don't know a good answer to your question right now, but I'm definitely interested in whatever useful answers others might offer...
I didn't think this is really feasible.
I've heard of efforts (edit: this is the one https://c2pa.org/ - I haven't read it at all so I don't know if it overlaps with my ideas below at all) to come up with a system that digitally signs images when they are taken using a tamper resistant TPM or secure enclave built into cameras, but that doesn't even begin to address the pile of potential attack vectors and challenges.
For example, if only cameras can sign images, and the signature is only valid for that exact image, then editing the image in any way makes the signature invalid. So then you'd probably need image editors to be able to make signatures or re-sign the edit, assuming it's minor (crop, color correct) but you'd need a way to prevent rogue/hacked image editors from being able to re-sign an edit that adds AI elements. So unless you want image editors to require you to have a TPM that can verify your edit is minor / not adding AI, then the image editor would be able to forge a signature on an AI edit.
Assuming you require every image editor to run on a device with a TPM in order to re-sign edits, there's also the problem of how you decide which edits are ok and which are too much. You probably can't allow compositing with external images unless they are also signed, because you could just add an AI image into an originally genuine image. You also probably couldn't stop someone from using macros to paint every pixel of an AI image on top of a genuine image using the pencil tool at 1px brush size, so you would need some kind of heuristic running inside the TPM or TEE that can check how much the image changed - and you'd have to prevent someone from also doing this piecewise (like only 1/10 of overlaying an AI image at a time so that the heuristic won't reject the edit), so you might need to keep the full original image embedded in the signed package so the final can be checked against the original to see if it was edited too much
You might be able to solve some of the editing vulnerabilities by only allowing a limited set of editing operations (like maybe only crop/rotate or curves), if you did that then you could not require a TPM to edit if the editing software doesn't actually create a new signature but just saves the edits as a list of changes along side the original signed image. Maybe a system like this where you can only crop/rotate and color correct images would work for stock photos or news, but that would be super limiting for everyone else so I can't see it really taking off.
And if that's not enough, I'm sure if this system was made then someone would just mitm the camera sensor and inject fake data, so you'd need to parts pair all camera sensors to the TPM, iPhone home button style (iiuc this exact kind of data injection attack is the justification for the iPhone home button fingerprint scanner parts pairing).
Oh, and how do you stop someone from using such a camera to take a picture of a screen that has an AI image on it?
Much of these problems can be solved by introducing a signature chain:
- Company A created the image
- Company B resized it
In this example, "Company A" can be a reliable news source, and "Company B" could be an aggregator like Mastodon or Facebook. So long as the chain is intact, the viewer can decide whether they trust every element in the chain and therefore trust the image.
This even allows people to use AI for responsible editing, because you're attacking the real problem: the connection between the creator (in whom you may or may not vest a certain amount of trust) and the media you're looking at.
I think you might be assuming that most of the problems I listed are about handling the trust of the software that made each modification - in case you just read the first part of my comment. And I'm not sure if changing the signature to a chain really addresses any of them besides having a bigger "hit list" of companies to scrutinize.
For reference, the issues I listed included:
- Trusted image editors adding or replacing a signature cannot do so securely without a TPM - without it someone can memory edit the image buffer without the program knowing and have a "crop" edit signed by Adobe which replaces the image with an AI one
- Needs a system to grade the "types" of edits in a foolproof way - so that you can't bypass having the image marked as "user imported an external image" by painting the imported images pixels over the original using an automated tool for example
- Need to prevent MITM of camera sensor data that can make the entire system moot
- You cannot prevent someone from taking a picture of a screen with Ai image
There are plenty of issues with how even a trusted piece of software allows you to edit the picture, since trusted software would need to be able to distinguish between a benign edit and one adding AI. I don't think a signature chain changes much since the chain just increases the number of involved parties that need to be vetted without changing any of the characteristics of what you are allowed to do.
I think the main problem with the signature chain is that is that the chain by itself doesn't allow you to attribute and particular part to and party in the chain. You will be able to see all the responsible parties but not have any way of telling which company in the chain could be responsible for signing a modification. If the chain contains Canon, gimp, and Adobe, there is no way to tell if the AI added to the image was because the canon camera was hacked or if gimp or Adobe has a workaround that allowed someone to replace the image with an AI one. I think in the case of a malicious edit, it makes less sense to allow the picture to retain the canon signature if the entire image could be changed by Adobe, essentially putting Canon's signature reputation on the line for stuff they might not be responsible for.
This would also bring a similar problem to the one I mentioned where there would need to be a level of trust for each piece of editing software - and you might have a world where gimp is out because nobody trusts it, so you can say goodbye to using any smaller developers image editor if you want your image to stay verified. That could be a nightmare if providers such as Facebook or others wanted to use the signature chain to prevent untrusted uploads, it would penalize using anything but Adobe products for example.
In short I don't think a chain changes much besides increasing the number of parties you have to evaluate complicating validation, without helping you attribute malicious edit to any party. And now you have a situation where gimp for example might be blamed for being in the chain when the vulnerability was from Adobe or Canon. My understanding of the question is that the goal is an automatic final determination of authenticity, which I think is infeasible. The chain you've proposed sounds closer to a "web of trust" style system where every user needs to create their own trust criteria and decide for themselves what to trust, which I think defeats the purpose of preventing gullible people from falling for AI images.
I think you're misunderstanding the purpose behind projects like c2pa. They're not trying to guarantee that the image isn't AI. They're attaching the reputation of the author(s) to the image. If you don't trust the author, then you can't trust the image.
You're right that a chain isn't fool-proof. For example, imagine if we were to attach some metadata to each link in the chain, it might look something like this:
| Author | Type | |
|
| | Alice the Photographer | Created | | AP photo editing department | Cropping | | Facebook | Resizing/optimisation |
At any point in the chain, someone could change the image entirely, claim "cropping" and be done with it, but what's important is the chain of custody from source to your eyeballs. If you don't trust the AP photo editing department to act responsibly, then your trust in the image they've shared with you is already tainted.
Consider your own reaction to a chain that looks like this for example:
| Author | Type | |
|
| | Alice the Photographer | Created | | AP photo editing department | Cropping | | Infowars | Cropping | | Facebook | Resizing/optimisation |
It doesn't matter if you trust Alice, AP, and Facebook. The fact that Infowars is in the mix means you've lost trust in the image.
Addressing your points directly:
- I'm not sure how a TPM applies to this as I haven't dug deep into c2pa other than the quick review I did this morning. I'm more interested in the high-level: "can we solve this by guaranteeing the origin" question, and I think the answer to that is yes. See my other comment for my own take on this.
- I don't think we need any sort of controls on defining the types of edits at all. If AP said they cropped the image, and if I trust AP, then I trust them as a link in the chain.
- Worrying about MITM attacks is not a reasonable argument against using a technology. By the same token, we shouldn't use TLS for banking because it can be compromised.
- Absolutely, but you can prevent someone from taking a picture of an AI image and claiming that someone else took the picture. As with anything else, it comes down to whether I trust the photographer, rather than what they've produced.
I think you are misunderstanding my mention of C2PA, which I only mentioned offhand as an example of prior art when it comes to digital media provenance that takes AI into account. If C2PA is indeed not about making a go/no-go determination of AI presence, then I don't think it's relevant to what OP is asking about because OP is asking about an "anti-ai proof", and I don't think a chain of trust that needs to be evaluated on an individual basis fulfills that role. I also did disclaim my mention of C2PA - that I haven't read it and don't know if it overlaps at all with this discussion. So in short I'm not misunderstanding C2PA because I'm not talking about C2PA, I just mentioned it as an interesting project that is tangentially related so that nobody feels the need to reply with "but you forgot about C2PA".
I’m more interested in the high-level: “can we solve this by guaranteeing the origin” question, and I think the answer to that is yes
I think you are glossing over the possibility that someone uses Photoshop to maliciously edit a photo, adding Adobe to the chain of trust. If instead you are suggesting that only individuals sign the chain of trust, then there is no way anyone will bother looking up each random person who edited an image (let alone every photographer) so they can check if it's trustworthy. Again I don't think that lines up with what OP is asking for. In addition, we already have a way to verify the origin of an image - just check the source AP posting an image on their site is currently equivalent to them signing it, so the only difference is some provenance, which I don't think provides any value unless the edit metadata is secured as I mention below. If you can't find the source then it's the same as an image without a signature chain. This system can't doesn't force unverified images to have an untrustworthy signature chain so you will mostly either have images with trustworthy signature chains that also include a credit that you can manually check or images without a source or a signature. The only way it can be useful is if checking the signature chain is easier than checking the website of the credited source, which if it requires the user to make the same determination I don't think it will move the needle besides making it marginally easier for those who would have checked for the source anyway to check faster.
I don’t think we need any sort of controls on defining the types of edits at all.
I disagree, the entire idea of the signature chain appears to be for the purpose of identifying potentially untrustworthy edits. If you can't be sure that the claimed edit is accurate, then you are deciding entirely based on the identity of the signatory - in which case storing the edit note is moot because it can't be used to narrow down which signature could be responsible for an AI modification.
If AP said they cropped the image, and if I trust AP, then I trust them as a link in the chain
The thing about this is that if you trust AP to be honest about their edits, then you likely already trust them to verify the source - this is something they already do so it seems the rest of the chain is moot. To use your own example, I can't see a world where we regularly need to verify that AP didn't take the image that was edited by Infowars posted on facebook, crop it, and sign it with AP's key. That is just about the only situation where I see the value in having the whole chain, but that's not solving a problem we currently have. If you were worried that a trusted source would get their image from an untrusted source, they wouldn't be a trusted source. And if a trusted source posts an image where it gets compressed or shared, it'll be on their official account or website which already vouches for it.
Worrying about MITM attacks is not a reasonable argument against using a technology. By the same token, we shouldn’t use TLS for banking because it can be compromised
The difference with TLS is that the malicious parties are not in ownership of the endpoints, so it's not at all comparable. In the case of a malicious photographer, the malicious party owns the hardware to be exploited. If the malicious party has physical access to the hardware it's almost always game over.
Absolutely, but you can prevent someone from taking a picture of an AI image and claiming that someone else took the picture. As with anything else, it comes down to whether I trust the photographer, rather than what they’ve produced.
Yes and this is exactly the problem, it comes down to whether you trust the photographer, meaning each user needs to research the source and make up their own mind. The system would have changed nothing from now, because in both cases you need to check the source and decide for yourself. You might argue that at least with a chain of signatures the source is attached to the image, but I don't think in practice that will change anything since any fake image will lack a signature just as how many fake images are not credited. The question OP seems to be asking is about a system that can make that determination because leaving it up to the user to check is exactly the problem we currently have.
This relies on everyone maintaining the chain though which there is nothing to force them into doing so.
Absolutely, but that's not really the point. If you remove the chain, then the file becomes untrusted. We're talking about attaching trust to an image, and a signature chain is how you ensure that that trust.
Couldn't you just start a new chain though from any point?
Yes, but starting a new chain would necessarily reallocate the ownership. So if reuters.com created a real image and then Alex Jones modified it, stripped the headers, and then re-created them, then the image would no longer appear to be from Reuters, but rather from infowars.com.
Being able to "prove" that something is AI generated usually means that:
A) The model the generated it leaves a watermark, either visually or hidden
B) The model is well known enough that you can deduce a pattern and reference what you're checking with that pattern.
The problem with the former is that you are trusted this corporations (or individuals training their own models, or malicious actors) to do so.
There are also problems with the latter: The models are constantly iterating and being patched to fix issues that people notice (favoring certain words, not being able to draw glasses of liquids full to the brim, etc)
Also, if the image or work was made using a niche or not well-documented AI then it probably wouldn't be a pattern that you're checking.
Also also, theres a high false positive rate, because it's just pattern matching mostly.
Perhaps a trusted certificate system (similar to https) might work for proving legitimacy?
Certificates like that can only guarantee that the work was published by someone who is the person they claim to be, it can't verify how that content came to be in their possession.
Hmm, I see. Surely wouldn't that be enough if they proclaimed they would only sign real photographs though?
Anyone can make such a promise. Verifying that they have followed through with it is not a technical challenge, it's a socioeconomic issue.
https://contentauthenticity.org/how-it-works
The page is very light on technical detail, but I think this is a system like trusted platform modules (TPMs), where there is a hardware root of trust in the camera holding the private key of an attestation certificate signed by the manufacturer at the time of manufacture, and it signs the pictures it takes. The consortium is eager for people to take this up ("open-source software!") and support showing and appending to provenance data in their software. The more people do so, the more valuable the special content-authenticating cameras become.
But TPMs on PCs have not been without vulnerabilities. I seem to recall that some manufacturers used a default or example private key for their CA certificates, or something. Vulnerabilities in the firmware of a content-authenticating camera could be used to jailbreak it and make it sign arbitrary pictures. And, unless the CAI is so completely successful that every cell phone authenticates its pictures (which means we all pay rent to the C2PA), some of the most important images will always be unauthenticated under this scheme.
And the entire scheme of trusted computing relies on wresting ultimate control of a computing device from its owner. That's how other parties can trust the device without trusting the user. It can be guaranteed that there are things the device will not do, even if the user wants it to. This extends the dominance of existing power structures down into the every-day use of the device. What is not permitted, the device will make impossible. And governments may compel the manufacturer to do one thing or another. See "The coming war on general computation," Cory Doctorow, 28c3.
What if your camera refused to take any pictures as long as it's located in Gaza? Or what if spies inserted code into a compulsory firmware update that would cause a camera with a certain serial number to recognize certain faces and edit those people out of pictures that it takes, before it signs them as being super-authentic?
Camera companies have been working on this. They have been trying to create a system that makes it possible to detect if an image has been tampered with https://www.lifewire.com/camera-makers-authentication-prevent-deepfakes-8422784
However this signature probably just uses assymetric encryption which could mean that the signing key on the device could be extracted and abused.
Yes, several large AI companies themselves are "watermarking" their images. https://www.nytimes.com/2024/02/08/business/media/google-ai.html