One black hat thing I'm curious about though is whether or not this tag can be weaponized. If I upload a real event and tag it as AI, will it reduce user trust that the real event ever happened?
The AI tags are fundamentally useless. The premise is that it would prevent someone from misleading you by thinking that something happened when it didn't, but someone who wants to do that would just not tag it then.
Which is where the real abuse comes in: You post footage of a real event and they say it was AI, and ban you for it etc., because what actually happened is politically inconvenient.
And the only way to prevent that would be a reliable way to detect AI-generated content which, if it existed, would obviate any need to tag anything because then it could be automated.
I think you have a bit backwards. If you want to publish pixels on a screen there should be no assumption that they represent real events.
If you want to publish proof of an event, you should have some pixels on a screen along with some cryptographic signature from a device sensor that would necessitate atleast a big corporation like Nikon / Sony / etc. being "in on it" to fake.
Also since no one likes RAW footage it should probably just be you post your edited version which may have "AI" upscaling / de-noising / motion blur fixing etc, AND you can post a link to your cryptographically signed verifiable RAW footage.
Of course there's still ways around that like your footage could just be a camera being pointed at an 8k screen or something but at least you make some serious hurdles and have a reasonable argument to the video being a result of photons bouncing off real objects hitting your camera sensor.
> If you want to publish proof of an event, you should have some pixels on a screen along with some cryptographic signature from a device sensor that would necessitate atleast a big corporation like Nikon / Sony / etc. being "in on it" to fake.
At which point nobody could verify anything that happened with any existing camera, including all past events as of today and all future events captured with any existing camera.
Then someone will publish a way to extract the key from some new camera model, both allowing anyone to forge anything by extracting a key and using it to sign whatever they want, and calling into question everything actually taken with that camera model/manufacturer.
Meanwhile cheap cameras will continue to be made that don't even support RAW, and people will capture real events with them because they were in hand when the events unexpectedly happened. Which is the most important use case because footage taken by a staff photographer at a large media company with a professional camera can already be authenticated by a big corporation, specifically the large media company.
also the three letter agencies (not just from the US) will have access to private keys of at least some manufacturers, allowing them to authenticate fake events and sow chaos by strategically leaking keys for cameras that recorded something they really don't like.
For all the folks that bash the United States for "reasons" this one gave me a chuckle. Our handling of privacy and data and such is absolute ass, but at least we *can* hide our data from big government with little repercussion in most cases (translation: you aren't actively being investigated for a crime that a judge isn't aware of)
Of course that says nothing about the issues of corruption of judges in the court system, but that is a "relatively" new issues that DOES absolutely need to be addressed.
(Shoot one could argue that the way certain folks are behaving right now is in itself unconstitutional and those folks should be booted)
Countries all over the world (EVEN IN EUROPE WITH THE GDPR) are a lot less "gracious" with anonymous communication. The UK actually has been trying to outlaw private encryption, for a while now, as an example, but there are worse examples from certain other countries. You can find them by examining their political system, most (all? I did quit a bit of research, but also was not interested in spending a ton of time on this topic) are "conservative leaning"
Note that I'm not talking just about existing policy, but countries that are continually trying to enact new policy.
Just like the US has "guarantees" on free speech, the right to vote, etc. The world needs guaranteed access to freedom of speech, religion, right to vote, healthcare, food, water, shelter, electricity, and medical care. I don't know of a single country in the world, including the US, that does anywhere close to a good of job with that.
I'm actually hoping that Ukraine is given both the motive and opportunity to push the boundaries in that regard. If you've been following some of the policy stuff, it is a step in the right direction. I 100% know they won't even come close to getting the job done, but they are definitely moving in the right direction. I definitely do not support this war, but with all of the death and destruction, at least there is a tiny little pinprick of light...
...Even if a single country in the world got everything right, we still need to find a way to unite everyone.
Our time in this universe is limited and our time on earth more-so. We should have been working together 60 years ago for a viable off-planet colony and related stuff. If the world ended tomorrow, humanity would cease to exist. You need over 100,000 people to sustain the human race in the event a catastrophic event wipes almost everyone out. Even if we had 1,000 people in space, our species would be doomed.
I am really super surprised that basic survival needs are NOT on the table when we are all arguing about religion, abortion, guns, etc. Like really?
> We should have been working together 60 years ago for a viable off-planet colony and related stuff. If the world ended tomorrow, humanity would cease to exist. You need over 100,000 people to sustain the human race in the event a catastrophic event wipes almost everyone out.
We are hundreds of years away from the kind of technology you would need for a viable fully self-sustainable off-world colony that houses 100k or more humans. We couldn't even build something close to one in Antarctica.
This kind of colony would need to span half of Mars to actually have access to all the resources it needs to build all of the high-tech gear they would require to just not die of asphixiation. And they would need top-tier universities to actually have people capable of designing and building those high-tech systems, and media companies, and gigantic farms to make not just food but bioplastics and on and on.
Starting 60 years earlier on a project that would take a millennium is ultimately irrelevant.
Not to mention, nothing we could possibly do on Earth would make it even a tenth as hard to live here than on Mars. Nuclear wars, the worse bio-engineered weapons, super volcanoes - it's much, much easier to create tech that would allow us to survive and thrive after all of these than it is to create tech for humans to survive on a frozen irradiated dusty planet with next to no atmosphere. And Mars is still the most hospitable other celestial body in the solar system.
> Nuclear wars, the worse bio-engineered weapons, super volcanoes - it's much, much easier to create tech that would allow us to survive and thrive after all of these than it is to create tech for humans to survive on a frozen irradiated dusty planet with next to no atmosphere.
This is the best argument I've heard for why we should do it. Once you can survive on Mars you've created the technology to survive whatever happens on Earth.
> I am really super surprised that basic survival needs are NOT on the table when we are all arguing about religion, abortion, guns, etc. Like really?
Most people in the world struggle to feed themselves and their families. This is the basic survival need. Do you think they fucking care what happens to humantiy in 100k years? Stop drinking that transhumanism kool-aid, give your windows a good cleaning and look at what's happening in the real world, every day.
I think doing this right goes the other direction. What we're going to end up with is a focus on provenance.
We already understand that with text. We know that to verify words, we have to trace it back to the source, and then we evaluate the credibility of the source.
There have been periods where recording technology ran ahead of faking technology, so we tended to just trust photos, audio, and video (even though they could always be used to paint misleading pictures). But that era is over. New technological tricks may push back the tide a little here and there, but mostly we're going to end up relying on, "Who says this is real, and why should we believe them?"
> If you want to publish proof of an event, you should have some pixels on a screen along with some cryptographic signature from a device sensor that would necessitate atleast a big corporation like Nikon / Sony / etc. being "in on it" to fake.
That idea doesn't work, at all.
Even assuming a perfect technical implementation, all you'd have to do to defeat it is launder your fake image through a camera's image sensor. And there's even a term for doing that: telecine.
With the right jig, a HiDPI display, and typical photo editing (no one shows you raw, full-res images), I don't think such a signature forgery would detectable by a layman or maybe even an expert.
> I worked in device attestation at Android. It’s not robust enough to put our understanding of reality in.
I don't follow. Isn't software backward compatibility a big reason why Android device attestation is so hard? For cameras, why can't the camera sensor output a digital signature of the sensor data along with the actual sensor data?
I am not sure how verifying that a photo was unaltered after capture from a camera if very useful though. You could just take a photo of a high-resolution display when an edited photo on it
It's true that 1990s pirated videos where someone snuck a handheld camera into the cinema were often very low quality.
But did you know large portions of The Mandalorian were produced with the actors acting in front of an enormous, high-resolution LED screen [1] instead of building a set, or using greenscreen?
It turns out pointing a camera at a screen can actually be pretty realistic, if you know what you're doing.
And I suspect the pr agencies interested in flooding the internet with images of Politician A kicking a puppy and Politician B rescuing flood victims do, in fact, know what they're doing.
That's a freaking massive LED wall... with professional cinematography on top. If you believed my comment was intended to imply that I believed that's somehow impossible, well... you and I have a very different understanding of what it means to "just take a picture of a high-resolution display"...
There's been a slow march to requiring hardware-backed security. I believe all new devices from the last couple of years need a TEE or a dedicated security chip.
At least with Android there are too many OEMs and they screw up too often. Bad actors will specifically seek out these devices, even if they're not very technically skilled. The skilled bad actors will 0-day the devices with the weakest security. For political reasons, even if a batch of a million devices are compromised it's hard to quickly ban them because that means those phones can no longer watch Netflix etc.
But you don't have to ban them for this use case? You just need something opportunistic, not ironclad. An entity like Google could publish those devices' certificates as "we can't verify the integrity of these devices' cameras", and let the public deal with that information (or not) as they wish. Customers who care about proving integrity (e.g., the media) will seek the verifiable devices. Those who don't, won't. I can't tell if I'm missing something here, but this seems much more straightforward than the software attestation problem Android has been dealing with so far.
Cryptography also has answers for some of this sort of thing. For example, you could use STARKs (Succinct Transparent Arguments of Knowledge) to create a proof that there exists a raw image I, and a signature S_I of I corresponding to the public key K (public input), and that H_O (public input) is a hash of an image O, and that O is the output of providing a specified transformation (cropping, JPEG compression) to I.
Then you give me O, I already know K (you tell me which manufacturer key to use, and I decide if I trust it), and the STARK proof. I validate the proof (including the public inputs K and H_O, which I recalculate from O myself), and if it validates I know that you have access to a signed image I that O is derived from in a well-defined way. You never have to disclose I to me. And with the advent of zkVMs, it isn't even necessarily that hard to do as long as you can tolerate the overhead of running the compression / cropping algorithm on a zkVM instead of real hardware, and don't mind the proof size (which is probably in the tens of megabytes at least).
Not if you do it, only if the chip also gives you a signed JPEG. Cropping and other simple transformations aren't an issue, though, since you could just specify them in unsigned metadata, and people would be able to inspect what they're doing. Either way, just having a signed image from the sensor ought to be adequate for any case where the authenticity is more important than anesthetics. You share both the processed version and the original, as proof that there's no misleading alteration.
> You share both the processed version and the original, as proof that there's no misleading alteration
so you cannot share the original if you intend to black out something from the original that you don't want revealed (e.g., a face or name or something).
The way you specced out how a signed jpeg works means the raw data _must_ remain visible. There's gonna be unintended consequences from such a system.
And it aint even that trustworthy - the signing key could potentially be stolen or coerced out, and fakes made. It's not a rock-solid proof - my benchmark for proof needs to be on par with blockchains'.
> The way you specced out how a signed jpeg works means the raw data _must_ remain visible. There's gonna be unintended consequences from such a system.
You can obviously extend this if you want to add bells and whistles like cropping or whatever. Like signing every NxN sub-block separately, or more fancy stuff if you really care. It should be obvious I'm not going to design in every feature you could possibly dream of in an HN comment...
And regardless, like I said: this whole thing is intended to be opportunistic. You use it when you can. When you can't, well, you explain why, or you don't. Ultimately it's always up to the beholder to decide whether to believe you, with or without proof.
> And it aint even that trustworthy - the signing key could potentially be stolen or coerced out, and fakes made.
I already addressed this: once you determine a particular camera model's signature ain't trustworthy, you publish it for the rest of the world to know.
> It's not a rock-solid proof - my benchmark for proof needs to be on par with blockchains'.
It's rock-solid enough for enough people. I can't guarantee I'll personally satisfy you, but you're going to be sorely disappointed when you realize what benchmarks courts currently use for assessing evidence tampering...
It also occurs to me that the camera chips -- or even separately-sold chips -- could be augmented to perform transformations (like black-out) on already-signed images. You could even make this work with arbitrary transformations - just sign the new image along with a description (e.g., bytecode) of the sequence of transformations applied to it so far. This would let you post-process authentic images while maintaining authenticity.
AI tags are to cover issues in the other direction: you publish an event as real, but they can prove it wasn't. If you didn't put the tag on it, malice can be inferred from your post (and further legal proceeding/moderation can happen)
It's the same as paid reviews: tags and disclaimers exist to make it easier to handle cases where you intentionally didn't put them.
It's not perfect and can be abused in other ways, but at least it's something.
> The premise is that it would prevent someone from misleading you by thinking that something happened when it didn't, but someone who wants to do that would just not tag it then.
And when they do that, the video is now against Google's policy and can be removed. That's the point of this policy.
Not convinced by this. Camera sensors have measurable individual noise, if you record RAW that won't be fakeable without prior access to the device. You'd have a straightforward case for defamation if your real footage were falsely labeled, and it would be easy to demonstrate in court.
> Camera sensors have measurable individual noise, if you record RAW that won't be fakeable without prior access to the device.
Which doesn't help you unless non-AI images are all required to be RAW. Moreover, someone who is trying to fabricate something could obviously obtain access to a real camera to emulate.
> You'd have a straightforward case for defamation if your real footage were falsely labeled, and it would be easy to demonstrate in court.
Defamation typically requires you to prove that the person making the claim knew it was false. They'll, of course, claim that they thought it was actually fake. Also, most people don't have the resources to sue YouTube for their screw ups.
Moreover, someone who is trying to fabricate something could obviously obtain access to a real camera to emulate.
Yes, but not to your camera. Sorry for not phrasing it more clearly: individual cameras have measurable noise signatures distinct from otherwise identical models.
On the lawsuit side, you just need to aver that you are the author of the original footage and are willing to prove it. As long as you are in possession of both the device and the footage, you have two pieces of solid evidence vs. someone elses feels/half-assed AI detection algorithm. There will be no shortage of tech-savvy media lawyers willing to take this case on contingency.
But who is the "you" in this case? There can be footage of you that wasn't taken with your camera. The person falsifying it would just claim they used their own camera. Which they would have access to ahead of time in order to incorporate its fingerprint into the video before publishing it.
Most consumer cameras require access menus to enable raw because dealing with RAW is a truly terrible user experience. The vast majority of image/video sensors out there don't even support raw recordings, out of the box.
Anyone with a mid-to-upper range phone or better-than-entry level DSLR/bridge camera has access to this, and anyone who uses that camera to make a living (eg shooting footage of protests) understands how to use RAW. I have friends who are complete technophobes but have figured this out because they want to be able to sell their footage from time to time.
The do have some use. Take for example the AI images of the pope wearing luxury brands that someone made about last year. They clearly wanted to make it as a joke, not to purposefully misinform people, and as long as everybody is in on the joke then I see no issue with that. But some people who weren't aware of current AIgen capabilities took it as real and an AI tag would have avoided the discussion of "has AI art gone too far" while still allowing that person to make their joke.
I mean they’re building the labeled dataset right now by having creators label it for them.
I would suspect this helps make moderation models better at estimating confidence levels of ai generated content that isn’t labeled as such (ie for deception).
Surprised we aren’t seeing more of this in labeling datasets for this new world (outside of captchas)
And this isn't new. A fad in films in the 90's was hyper-realistic masks on the one side, and make-up and prosthetics artists on the other, making people look like other people.
Faking things is not new, and you've always been right to mistrust what you see on the internet. "AI" technology has made it easy, convenient, accessible and affordable to more people though, beforehand you needed image/video editing skills and software, a good voice mod, be a good (voice) actor, etc.
> you've always been right to mistrust what you see on the internet.
But these tools make deception easier and cheaper, meaning it will become much more common. Also, it's not just "on the internet". The trust problem this brings up applies to everything.
This deeply worries me. A post-truth society loses it's ability to participate in democracy, becomes a low-trust society, the population falls into learned helplessness and apathy ("who can even know what's true any more?")
Look at Russian society for a sneak preview if we don't get this right.
It just goes back to trusting the source. If 5 media orgs post different recordings of the same political speech, you can be reasonably sure it actually happened, or at least several orders of magnitude more sure than if it's one blurry video from a no name account.
This bodes well for autocracies and would-be autocrats. It's the logical extreme of what they've been trying to do on social media over the last decade or so.
I was immediately thinking that the #AI labels are going to give people a false sense of trust, so that when someone posts a good-enough fake without the #AI label, it can do damage if it goes viral before it gets taken down for the mislabeling. (Kudos for the effort, though, YouTube.)
Behind the scenes, I'm 99% confident that Google has deployed AI detection tools and will monitor for it.
That said, unless all the AI generators agree on a way to add an unalterable marker that something is generated, at one point it may become undetectable. May.
I'm not aware of any AI detection tools that are actually effective enough to be interesting. Perhaps Google has some super-secret method that works, but I rather doubt it. If they did, I think they'd be trumpeting it from the hilltops.
We have to expect people to think for themselves. People are flawed and will be deceived but trying to centralize critical thinking will have far more disastrous results. Its always been that way.
Im not saying Youtube shouldn’t have AI labels. Im saying we shouldn’t assume they’re reliable.
>but trying to centralize critical thinking will have far more disastrous results
No. Having sources of trust is the basis of managing complexity. When you turned the tap water on and bought a piece of meat at the butcher you didn't yourself verify whether its healthy right? You trust the medicine you buy contains exactly what is says on the label and didn't take a chemistry class. That's centralized trust. You rely on it ten thousand times a day implicitly.
There need to be measures to make sure media content is trustworthy, because the smartest person on the earth doesn't have enough resources to critically judge 1% of what they're exposed to every day. It is simply a question of information processing.
It's a mathematical necessity. Information that is collectively processed constantly goes up, individiual bandwith does not, therefore you need more division of labor, efficieny and higher forms of social organisation.
> Having sources of trust is the basis of managing complexity.
This is a false equivalence that I’ve already addressed.
> When you turned the tap water on and bought a piece of meat at the butcher you didn't yourself verify whether its healthy right?
To a degree, yeah, you do check. Especially when you get it from somewhere with prior problems. And if you see something off you check further and adjust accordingly.
Why resort to anology? Should we blindly trust YouTube to judge whats true or not? I stated that labeling videos is fine but what’s not fine is blindly trusting it.
Additionally, comparing to meat dispenses with all the controversy because food safety is a comparatively objective standard.
Compare, “is this steak safe to eat or not?” To “is this speech safe to hear or not?”
I'm probably paraphrasing Schneier (and getting it wrong), but getting water from the tap and having it polluted or poisonous, has legal and criminal consequences. Similarly getting meat from a butcher and having it tainted.
Right now, getting videos which are completely AI/deepfaked to misrepresent, are not subject to the same consequences, simply because either #1 people can't be bothered, #2 are too busy spreading it via social media, or #3 have no idea how to sue the party on the other side.
And therein lies the danger, as with social media, of the lack of consequences (and hence the popularity of swatting, pretexting etc)
I suspect we're headed into a world of attestation via cryptographically signed videos. If you're the sole witness, then you can reduce the trust in the event, however, if it's a major event, then we can fall back on existing news-gathering machinery to validate and counter your false tagging (e.g. if a BBC camera captured the event, or there is some other corroboration & fact checking).
How does the signature help? It only proves that the video hasn't been altered since [timestamp]. It doesn't prove that it wasn't AI-generated or manipulated.
Signatures are also able to (mostly) signal that a specific device (and/or application on that device) captured the video. It would be possible to check if a video was encoded by a specific instance of an iOS Camera app or AfterEffects on PC.
Everything else - corroboration, interviews, fact checking will remain as they are today and can't be replaced by technology. So I imagine a journalist would reach out to person who recorded thr video, ask them to show their device's fingerprint and ask about their experience when (event) occured, and then corroborate all that information from other sources.
When the news org publishes the video, they may sign it with their own key and/or vouch for the original one so viewers of clips on social media will know that Fox News (TM) is putting their name and reputation behind the video, and it hasn't been altered from the version Fox News chose to share, even though the "ModernMilitiaMan97" account that reshared it seems dubious.
Currently, there's no way to detect alterations or fabrications of both the "citizen-journalist" footage and post-broadcast footage.
If I have a CCTV camera that is in a known location and a TPM that signs its footage, I could probably convince a jury that it’s legit in the face of a deepfake defense.
That’s the bar- it’s not going to be infallible but if you don’t find evidence of tampering with the hardware then it’s probably going to be fine.
This might be worse than nothing. It's exactly the same tech as DRM, which is good enough to stop the average person, but where tons of people have private exploits stashed away to crack it. So the judge and general public trust the system to be basically foolproof, while criminals can forge fake signatures using keys they extracted from the hardware.
This isn’t strictly some blackhat thing, people will attempt to hand wave inconvenient evidence against them as AI generated and build reasonable doubt.
