Even image compression cannot defeat our cloak.
We use progressive JPEG , reportedly used
by Facebook and Twitter, to compress the images
in our dataset. The image quality, as standard
by Independent JPEG Group , ranges from 5 to
95 (lower value = higher compression). As shown
in Figure 15, image compression decreases the
protection success rate, but more significantly
degrades normal classification accuracy.
p.s. Mobile friendly copy-paste: "Even image compression cannot defeat our cloak. We use progressive JPEG , reportedly used by Facebook and Twitter, to compress the images in our dataset. The image quality, as standard by Independent JPEG Group , ranges from 5 to 95 (lower value = higher compression). As shown in Figure 15, image compression decreases the protection success rate, but more significantly degrades normal classification accuracy."
If the procedure fails then the GAN will know this and it would change the output so that it would pass both outcomes.
I think it does things like subtly change the shape of your eyes and whatnot. That makes people consistently difficult to pick out of a large set of unmodified faces, but a hu man still finds the images recognizable because they're comparing it to a smaller set.
They didn't create the model by training it on the facial recognizers they tested against, and they tested with several different feature extractors.
> However,we find that none of these transformations defeat our cloaks. The protection success rate remains 100% even when data augmentation is applied to cloaked images5. Ap-plying Gaussian blurring degrades normal accuracy by up to18% (as kernel size increases) while cloak protection success rate remains>98% (see Figure13). Adding Gaussian noise to images merely disrupts normal classification accuracy –the cloak protection success rate remains above 100% as the standard deviation of the noise distribution increases(seeFigure14). Even image compression cannot defeat our cloak.We use progressive JPEG , reportedly used by Facebookand Twitter, to compress the images in our dataset. The im-age quality, as standard by Independent JPEG Group ,ranges from 5 to 95 (lower value = higher compression). As shown in Figure15, image compression decreases the pro-tection success rate, but more significantly degrades normal classification accuracy.
Short answer: No not really. Long answer: Look at the FAQs :)
If a human's neural network can correctly correlate the before/after examples, so can a computer's. They might have found an issue with some modern implementations of facial recognition, sure. But it's a false sense of security to claim "when someone tries to identify you using an unaltered image of you [...] they will fail."
color _me_ skeptical, but
this is like saying we have functioning AGI; that artificial NNs are the same as the ones we have in our skulls. This to me, is an effect of the over-anthropomorphization of machine learning. It's a bad intuition to have.
However, I do agree. This is just one step in an arms race, and one iteration from being worthless.
Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks?
A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.
First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m
Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm
The resulting cloaked image: https://imgur.com/a/OSHXdbO
I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms
Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3
Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx
A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS
The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category.
So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me!
Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0
And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit
So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network.
Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it.
All in all, very interesting and thanks for convincing me that I should actually test it out.
If they were promising that cloaking would work well on GIS, that'd be a different matter. I can imagine wanting your images to not show up on GIS (because people would use them to try and find the source image on your profile, or something) but it's a different set of constraints at that point.
For cloaking a big use case would be "I took a selfie with a friend and want to share it on my instagram" and your goal is for that instagram selfie to not automatically connect with, for example, a surveillance photo of you at a protest. GIS is obviously not relevant to that scenario.
I echo the sentiment of other posters regarding reverse-image search. The original image should not be available to match against. That would be operator failure.
"similar image" search generally uses "perceptual hashing" which is not related to facial recognition, and not really a privacy risk.
You're not testing against what this tool was designed to defeat :-/
Here's the original image: https://imgur.com/a/Td4rhoy
And the cloaked: https://imgur.com/a/cPCiCZo
These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO
And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B
This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked.
Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general.
P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.
However, if someone is willing to go to that level of effort, the target probably needs to aim for something a little more forceful than tricking Facebook's autotagger.
Someone, somewhere said "huh...", and placed another filter into the pipeline to handle these types of images.
While looking for something to sound smart wrt the tank-training myth, I found this interesting page: https://www.gwern.net/Tanks "The Neural Net Tank Urban Legend"
And interestingly, looking at the link for "superresolution needing learned downscalers" found this: https://arxiv.org/abs/1907.12904 "Learned Image Downscaling for Upscaling using Content Adaptive Resampler", code available at https://github.com/sunwj/CAR
So, IDK, seems like this Fawkes approach will be an interesting paper.
Neural Networks do not learn what humans learn. They can learn completely different and sometimes much smaller features.
I guess that face rec software will quickly adapt though. That said, we have invisible watermarks that are very resistant to compression or other filters.
I believe, Model trained on cloaked images would defeat its purpose and make this technique useless.
 Su, Jiawei, Danilo Vasconcellos Vargas, and Kouichi Sakurai. "One pixel attack for fooling deep neural networks." IEEE Transactions on Evolutionary Computation 23.5 (2019): 828-841.
 Guo, Chuan, et al. "Countering adversarial images using input transformations." arXiv preprint arXiv:1711.00117 (2017).
 Liu, Yanpei, et al. "Delving into transferable adversarial examples and black-box attacks." arXiv preprint arXiv:1611.02770 (2016).
(If I were these researchers I'd totally be reaching out to AWS/Azure/GCE for additional research funding... <smirk>)
(So now you just need to somehow get as many cloaked photos of yourself uploaded and tagged to FB as they've collected in the last decade or so...)
So, if I adopt this and upload only cloaked images on social media, and the people I normally interact with also do the same, then facial recognition will be able to detect me based on someone showing the system that I’m present in the photo (even though it identifies me as the distorted version)?
If the above understanding is true, then even law enforcement could cloak all the photos they have and try to match captures with their raw photo set and the cloaked photo set to narrow it down for a human?
What am I missing?
(But I'm not sure, and have downloaded the paper and the apps to read and experiment with...)
I think that a simpler and more robust strategy to achieve good privacy is avoid posting personal information online and social media altogether.
However, simply not going on the Internet does not solve the problem people care about. People's desired solution is to use the Internet in a personal way and be safe - not just to be safe.
Do different photos of the same person produce unique results where even a comparison between two cloaked will result in a mismatch? The article mentions that only the comparison between unaltered and cloaked images will result in a mismatch. If that is the case, what's stopping someone from using this algorithm to generate a cloaked image from the unaltered one and then using both in order to identify you?
I wonder how this holds up when someone takes a photo of that 'protected image'. I can imagine that if these miniscule pixel-scaled changes aren't visible to the naked eye, my crappy 6 megapixel camera will overlook it as well. If I then proceed to feed that image into my image recognition algorithm, is it still protected?
So if your crappy 6 megapixel camera cannot take a clear shot of the cloaked pixels - or effectively applying a blur filter - would also affect the AI detection.
There's quite a lot of comments here that stink of Dunning Kruger candidates, who read the headline and first paragraph, then just started typing their random "wisdom" assuming they're smarter and better informed that the team of PHD researchers who wrote the paper being discussed. (Am I just overly grumpy and judgemental today? Was HN always this bad?)
I personally would like to see tests done on facebook by uploading these images and checking if it can recognize it.
Try taking a photo of your face or someone you know with as near symmetric lighting etc as you can manage. Now cut the image vertically and mirror each half and compare visually.
Frightening isn't it?
Now add fancy patterns that can be hidden within an image that eyes miss but algos don't. AI does not see the way you and I do. It can't. AI can be twiddled constantly to get it into line with what we perceive and we could call that evolution. In 200M years it might be quite good.
I suspect that progress will be faster than that but those machines can't type on a keyboard balanced on its knee whilst drinking wine and admiring a landscape with a setting sun whilst worrying about how to shop tomorrow, now masks are compulsory. What's the SO up to? The TV is showing crap and a new Netflix series is available but I can't be arsed ...
The march of our robot overlords is unlikely soon.
Certainly another tool in the privacy toolkit if you absolutely must surrender your likeness to someone else’s computer, but worth bearing in mind that this does not provide (and doesn’t purport to provide) the kind of privacy that strong encryption (or better yet, absent data) can provide.
Edited to add: it’s still damn cool.
Technical solutions have never solved this sort of societal problem. Expecting a few individuals to fight against massive institutions with a little clever math is not going to work.
That rant aside I am curious if this technique will lead to more resilent facial recognition and image parsing techniques to find the shape. Obviously the fact humans can still recognize it is a hint there is some other algorithim possible.
One thing that I took notice of was how long the program ran on my computer. It took about 5 and a half minutes to obfuscate 4 images on an i9-9900K with the cpu was pegged at 100% the entire time. I can't imagine how long this would take on a low end laptop: especially if I needed to cloak a lot of images in bulk.
Another thing I noticed is that the discoloration that is applied to the final images can be easily mistaken for bruising. If I were to see someones post on social media and they looked like my results I'd be inclined to think that the poster is recovering from a bad fight or is a victim of abuse.
Other than those two little nit picks the tool is pretty cool! However I don't think I will be using it myself due to the second point.
That said, given that for most people the threat model is social or work rather than legal, something like this would be terrific to build into consumer insta-photo devices.
> These adversarial examples have been recognized since 2014 (here's one of the first papers on the topic).
Adversarial machine leaning has been around since 2006. It's only since 2014 that people decided to call them Adversarial Examples.
If you tweak the values a bit lower it doesn't look so bad, but of course I haven't tested it with an array of DL algorithms.
This is just one side of a GAN, on the next iteration, it will be defeated.
Bottom line is that if a human can recognize, then it is possible for a machine as well.
Also, given that the big networks can just keep throwing more resources at it (I.e. GPT-3), it’s just a matter of increasing the network size to improve feature redundancy.
Adversarial examples transfer between different models trained on different datasets with different architectures.
A new model from yesterday's data is essentially the same architecture, just with some fluctuations in decision boundaries.
Might it affect the success over time? Sure. But not tomorrow.
I can be as careful as I want to be with my own media. That doesn't stop my wife from uploading the family photo to Facebook or a public camera capturing my image.
Check out https://cvdazzle.com (archive link, because honestly I'm surprised the site is still up: http://archive.is/v39xI)
This looks creepy but effective:
If a fixed number of bytes can be interpreted visually by a human as a specific human on a reliable basis, there is zero reason a computer cannot accomplish the same. At worst, we are talking about some minor deficit in the current ML libraries that were tested, likely somewhere in pre-filtering of training data. As mentioned in other posts, a low-pass filter is exactly what you would use to side-step this sort of thing.
From a much more sinister perspective, this is potentially even more dangerous than not applying this cloaking process at all. Presumably, there is some way to detect that this process has been applied and that it has certain hallmarks. Assuming it is resilient enough to survive JPEG and other typical compression schemes, I would wonder if perhaps this is a tool to positively identify those who would otherwise want to hide from authorities.
Also, they addressed low pass filters and other image degradation techniques in the article, and almost every adversarial example paper addresses them, and works in spite of them. You're not the first person to think of that.
In fact, it's basically the entire problem.
My software works absolutely fine on these images - it correctly identifies all keypoints.
I would question if this works beyond the specific pieces of software they tested against.
In practice, building new training sets is much more expensive than slightly changing the cloaking algorithm.
It's very important that it works 'cos clearview is so creepy. It's not creepy because of its technology, it's creepy because the justifications of its existence that are given by its CEO are sooo weak. "we can do it because it's not worse than google" (ie. we entirely skip the moral argument), "we can do it because it's for law enforcement" (let's frame our stuff in a way that it's only positive) , "we can do it because we ensure that those who use our tool are strictly controlled" (yeah, we're above the states), "all images are public therefore I can do whatever I want", etc.
If this really works, why are they not publishing it in a prestigious computer vision conference?
This may stop some internet marketers, but don't expect it to be effective against large corps and governments.
If you're asking: what if the model trains on "uncloaked" images, we talk about that extensively in the paper and provide technical answers and experimental results. Take a look.
i love you thank you.
Ben (on behalf of the team)