It's hard to see this as much anything but snake oil. While the technique allegedly allows you to target multiple models, it still requires you to know what adversarial model you're targeting before the fact. If the adversary sees there's visible artifacts they'll just change a few parameters or use a different model and be good to go none the less. In short, while this might be academically interesting, I can't help but feel that this is a futile field to work in. A good way to see why is to look at it this way: a skilled human can manually manipulate an image so that it isn't possible to tell that the image has been manipulated. It is then a matter of teaching the model to do the same, which might take time or be computationally prohibitive for now, but eventually it will be done.
A few weeks ago, there was a post here about that guy who deepfaked female profile pictures. He showed that the visible artifacts go away by merely scaling up the number of parameters in the model. Give it a few years and these fake images will be impossible to detect unless we agree on some way of cryptographically signing images from real cameras.
Cryptographic keys embedded in hardware worked really well for the DRM industry. Eventually someone will work out how to extract the key, then create malware to spread around android devices to send back millions of valid keys. What do you do then? Blacklist millions of real peoples cameras and prevent them from using the internet?
You only need a method of key invalidation and renewal, that's all. Ssl certificates have been facing this problem for years. The threat model is equal to someone infecting millions of devices and then sending back banking data, so it's not like people aren't working on mitigating that stuff.
>Ssl certificates have been facing this problem for years.
That's an entirely different, and much easier problem. In the case of SSL you are not worried about an attacker obtaining a certificate for any host, just the ones you care about. E.g. an attacker getting a certificate for facebook.com would be catastrophic, but an attacker getting a certificate for a website that no one uses would be a non-issue.
For the case of avoiding deepfakes, you need to avoid the attacker extracting a key from any of the millions of cameras that are sold every year.
Surely the threat model remains the same. being able to forge photos from a camera held by the LAPD forensics lab, or Reuters would be more "damaging" than having my camera hardware key.
I think there is scope for a simple self regulation here to start us off. I would love to see Reuters or the BBC start to publish their raw footage with the hashes. It is a question of starting the ball rolling
Not really because this is being proposed as a solution to solve people using fake photos on dating apps which means it has to work on every single consumer device to actually work.
You just need to couple the keys to something tangible. But the truth is that every system can by bypassed if you spend enough effort. The only question is when it becomes enough hassle so that it doesn't disrupt ordinary people's lives. Not even facebook cared about enforcing https until someone made a browser plugin that let everyone steal cookies.
So now we are essentially bricking the devices of real users? Without doing anything meaningful to real attackers who just grab the next key out of their list of 10 million stolen from insecure androids.
What? No. The signature needs to be added by the image sensor as it gathers data.
It shouldn't be easy to extract it and that's it. It's even less difficult than keeping the DVD/Blueray keys secure because each device has a separate key, so if a line of devices gets compromised easily it's easy to spot.
Then you put a legal framework around what can be presented by media, the requirement for signature collection and so. And one of problems with photo/video authenticity is essentially solved.
I'm not hopeful that the latter will do much. Just do your deepfake, point a camera at your screen and press the shutter button to get it cryptographically signed..?
Ok, so that's tongue in cheek and we'd see some artifacts there but the general principle works. You could intercept the signal from the CCD, or just extract the signing key from the camera's ROM, etc etc.
You can already do that today to fool basic image forensics. Security is never absolute, but if you make the hassle big enough it might suffice to protect the general population. Just like almost all smartphones don't let you mess with the wifi card or spoof your mac address unless you root them. If that wasn't the case, you'd see way more hostapd-wpe attacks these days, since every kid could do it.
In the case of image trustworthiness, it might actually be better for it to be accessible to the general population, if it's doable at all. Then the public knows that images are unreliable evidence.
The point is that an image of consequence, if real, would have a signed raw copy available which people could check to verify that there was in fact a real image that showed what a given edited image purported to show. If an image is being circulated but no one as been able to verify that a signed version exists, that image remains untrustworthy. This wouldn't stop people from editing images for aesthetic value.
I guess there'd have to be certified editors, which would be trusted to allow only "safe" operations on images (crop, white balance, levels, etc) and would sign the resulting jpegs themselves alongside the signature from the camera.
It doesn't really seem likely at all to work in practice.
Editors include deep fake technology now. Removing people from a scene has been a common technique for a long time and generative models just have made this easier (e.g. usage: you want the background/scenery but it is a well traveled tourist destination so you can't get a photo without people in it.[0])
[0] Normally you accomplish this by taking several photos and combining them
Sure, you'd have to have a "safe mode" or something for Photoshop that would lock you out of almost everything that could be used to alter the semantic meaning of an image. Edits made in that mode would be considered "safe" and Photoshop would be trusted to sign the resulting jpegs.
I think the solution is for the editor to have a hard limit on how many edits it will apply to one image. You'd need to write into the output file how many edits had been performed before signing it and then refuse to process the edited image any further if it's too high.
Any scheme like this would be eventually broken somehow, so it's altogether a bad idea imho
My new startup Tovera[0] is working on a verification approach that (with some tweaking and user feedback) should be usable by anyone without some of the pesky and potentially privacy concerning aspects of a key/cert based approach. More in my comment here:
Imagine a cryptocurrency like log of data verifying images. Essentially, all cameras will be internet connected with a process to force creation of an NFT at the moment users press the photo button which includes the entire data of the image. Any image without that tech will be considered unreliable. Of course that would have implications that no photo could ever be deleted, I’m guessing this system would be government based and they would be just fine with that.
This is exactly what my new startup Tovera[0] does. Process is roughly:
1) Upload image with web-based user dashboard.
2) We strip existing metadata (for privacy and all of the reasons everyone else does).
3) Our API generates a unique identifier for the asset (UUIDv4 in hex).
4) The unique identifier is embedded in the image with XMP.
5) A SHA256 checksum of the entire file is generated.
6) Via the API the checksum is associated with the unique identifier (along with some other stuff).
7) (Optionally) the unique identifier and checksum are added as JSON to IPFS via a pinning service and (essentially) an NFT is minted for the JSON verification data on the Polygon blockchain.
8) The user gets sharable[1] and iframe embeddable[2] links and the ability to post directly to various social media networks, etc.
When the links are viewed our Javascript reads the unique id, fetches the stored checksum from the API, and generates a new checksum of the image in the browser. If the checksums match a clickable icon appears in the top right of the image with additional information about the image, links to the IPFS and blockchain links, etc. Users can change the additional metadata at anytime and it updates instantly.
Long term goals are hosted javascript verification library, browser extensions, mobile SDKs, potential browser/OS integrations, native plugins for popular authoring/editing applications, and so on.
What can crypto verify here exactly? What stops me from:
A. Writing my own software "camera" that does the same gyrations a real camera would do to sign its raw image, but applied to an arbitrary unverified image file on my computer?
or
B. Printing out an unverified picture and taking a picture of that picture (with good lighting etc so that it is not obvious), with a camera that makes it verified "real"?
Nothing stops smartphone or camera manufacturers from adding something like a TPM to the CMOS. In fact, Apple already does sign its hardware components to prevent third party repairs.
This is fine, we will just train another model to detect which model is being targeted.
Then if there is an adversarial model built for that, we will just create a model to detect images that have been changed to hide the model they were built with.
Then we will build a model for the adversarial AI on top of that and so on.
> I can't help but feel that this is a futile field to work in.
Considering that training a model costs thousands of dollars, and that people reuse models instead of training from scratch, I somewhat disagree with this in general, for now.
But this specific technique implies the attacker sees the result, which means they can try different models, until one does not produce artifacts.
Models come in a broad spectrum and while it's true that gpt-3 costs a ton, many do not. Any mobile ready snapchat esque real time deep fake filter will typically be super fast and cheap to train (since the model has to be lightweight to run on a phone). There's also transfer learning from an expensive model which could easily be modified to remove whatever 'architecture flagging' is built into the structure.
> First, we show how an image owner with access to an inpainting model can augment their image in such a way that any attempt to edit it using that model will add arbitrary visible information. We find that we can target multiple different models simultaneously with our technique. This can be designed to reconstitute a watermark if the editor had been trying to remove it. Second, we show that our markpainting technique is transferable to models that have different architectures or were trained on different datasets, so watermarks created using it are difficult for adversaries to remove. Markpainting is novel and can be used as a manipulation alarm that becomes visible in the event of inpainting.
Yes, but if the "fake detection" technique is computationally much faster than the generation technique, having access to a better oracle is not so helpful to the deepfake creators, who are bottlenecked by the other phase.
I would interested in a way to automatically edit real photos of people in such a way that it looks the same but now tests positive for being a deepfake.
Tangentially reminds me of the film Surrogates, where people only went out in their surrogate, a remote controlled android - in some cases, a younger, better looking version of people. In other cases something completely different of course.
The whole point of GANs is to train a model until it can defeat detection by another model. Virtually any technique for detection that can be reproduced could just be incorporated into the training loop and rendered almost useless.
> An image owner can modify their image in subtle ways which are not themselves very visible, but will sabotage any attempt to inpaint it by adding visible information determined in advance by the markpainter.
Hmm... so if I ask it to add an apple to the scene it just looks for an apple and says "yep, found an apple".
That is not an algorithm to detect fakes, it is an algorithm to detect predetermined edits.
Steganography/watermarking has been not only known but used for practical purposes for a long time.
no i think that sentence means you can still use photoshop manually to laboriously edit stuff… but the super convenient AI editing will fail, meaning 99,9% of the population won‘t be able to create deepfakes
Watermarking can be used to find portions of the photo that have been tampered with.
It is actually pretty simple. And using AI for this is pretty stupid in my opinion.
You just disperse a little bit of additional signal in the photo, maybe divide it into lots of blocks (they don't need to be rectangles and they can overlap). Think of it as every small piece of the photo having its own signature embedded.
If you don't know how the signature was embedded or what key was used for it, you are going to disturb the signature and your editing attempt will be foiled. Not only that, but you will be able to tell which parts of the picture were touched and how so you can tell whether the picture was just cropped, brightness changed, or something else happened to it.
Actually, it is probably even worse than having deepfakes proliferated. In this case people would expect the photos could have been manipulated. On the other hand if only handful of players can do this it can be used with much more impact.
scenario: you want to resell 50000 istock phots, but they are watermarked, making the free high resolution versions valueless.
solution: you use a denoising filter to reconstruct the watermark pixels to plausible original values. Profit!
this: instead of just simple obvious watermarks, you can instead encode visually indistinct fake-noise that deliberately confuses denoising neural networks.
They claim “We find that we can target multiple different models simultaneously with our technique.”, ie. it is reasonably generic.
The OP's title is incorrect. This doesn't detect deepfakes, it serves for people to watermark their images in a way that are hard to remove by conventional ML approaches.
> ...a photo agency that makes stock photos available on its website with copyright watermarks can markpaint them in such a way that anyone using common editing software to remove a watermark will fail; the copyright mark will be markpainted right back. So watermarks can be made a lot more robust.
I fail to see how this can actually combat against someone with the patience to manually paste out copyright marks using something like a clone brush, or even a normal brush ?
If someone wants to steal your work, they will. There's no sure-fire way to stop them.
It won't defeat manual edit, nor do the authors claim it. They do however make a solid point in saying it will defeat cheap automated manipulation / deepfakes at scale that would otherwise be easy with inpainting features.
Anything that can help us differentiate authentic images from manipulated ones, especially when it comes to news, is imo a welcome addition in the arsenal of societies attached to the concept of verifiable facts.
I'd say cropping an image leaves the image 'authentic'. Even if it changes the meaning. It's when you start painting out objects that you are making a 'non-authentic' picture.
Couldn't reputable photo/video authors just publish content as an NFT? If the NFT is not traced back to a reputable source, then the content is deemed unreliable.
You mean devices could sign the images they create and then wherever someone publishes the picture you'd be able to validate the signature using that device's (published somewhere) public key? That would be cool indeed.
Except it wouldn't change anything. Facebook doesn't care about fakes and people wouldn't care whether the picture someone shared hasn't been tampered. Also whoever manipulated the picture could just sign it again probably.
Some people do care about authenticity though, hence the genesis of the OP's work to try and authenticate and it's why people still read news papers vice trusting their clickbait Google "news" feed. I'm just saying that I'd trust some encryption cert (whatever it is) traced back to a known more reputable source (say Financial Times and the like) than an algorithm that is just guessing that a photo/video is an un-altered genuine original vice a fake.
The challenge is establishing an author's credibility in the first place. If I trace a cert back to joe schmo off the street or some empty front business, then I'd probably not trust the content.
OK cool, but the commonly used Photoshop's content aware fill, which they supposedly target, isn't even using any ML. They don't know what they're doing or what?
https://en.wikipedia.org/wiki/PatchMatch
While all this stuff around deep fakes and its detection is interesting I'm still left wondering why we don't use cryptography to prove if an image is authentic or not.
Because the world is impure and the link between the real world and the mathematical domain will never be ironclad.
For instance, say that Joe makes a deepfake and then signs it with his key. Sure, it's beyond doubt (assuming keys weren't leaked, etc) that Joe either took or created the picture, but that doesn't in itself tell you whether Joe made a deepfake or not.
It's the same as in supposed blockchain logistics operations. If Fred the farmer says he harvested x bags worth of grain but some were stolen before he could ship them, there's no way to mathematically verify whether the theft actually took place or the harvest just came up short.
In both cases you're going to need some kind of monitoring, and that's the purpose deepfake detection algorithms would serve.
I had the same realization several months ago. In the arms race between deepfake detection and generation it's widely understood that generation will win. That's why I started working on Tovera[0].
Will signing come from the camera? At that point you run into DRM.
Besides, what will the camera sign? The produced JPG? The raw file? What about rescaling? Do we want ZKPs that a JPG was achieved by nothing more than re-scaling and tone-mapping another JPG or RAW file? Those ZKPs are going to be massively big and slow to verify.
My main concern is the faking of official statements by public figures where having those signed addresses the issue of deep fakes. Of course there has to be infrastructure in place for this and people must know that you can verify them.
If we get to a point in society where unsigned images, or images signed from a questionable source are looked at skeptically, then that alone is a step in the right direction. I don't know the final solution and I don't claim to know it, but cryptography seems like a step in the right direction.
