Whatever you think about this, it's here to stay - the code is open source and cached on a million hard drives already.
If we treat the normal rate of progression, we'll see a year 2023 version for 5 bucks off the App Store, your phone will live-convert the video you're recording at your drinking party with your friends into an all celebrity event. Probably with voice swap. Your drunken roommate's scratchy voice will be ML'd into Frank Sinatra, autotuned, face swapped, and uploaded for a few laughs to facebook.
This! One of the most ridiculous notions in articles about this when it first started coming out was "We should ban this technology!" and reddit started deleting threads that contained links to download the scripts and apps. Do these people still really not understand that's not how this works? Has the Streisand effect not been proven enough yet for the general populous?
I don't think banning is the solution, but with stiff enough financial or criminal consequences for publishing a deepfake without the person's consent or without using a "DeepFake" watermark it would certainly influence the way it's currently being used. Right now, no one's coming after the creators, but one big libel suit that gets enforced or the government passing a law would change this space significantly.
The only thing that would accomplish would be to make people less aware of the problem so when the actual bad guys do actual bad things, fewer people would spot it.
It will probably just get produced in China or Iceland or someplace. Putin must be already dedicating super computing time. Can't wait for the next election cycle.
If it's generated based on copyright-protected sources, it may be a violation of the copyright on the protected sources. Might also in certain circumstances violate right of publicity and/or defamstion laws to publish it, independent of whether the creation was a violation of copyright.
It's funny you say that because when you see deletions of it it's typically a copyright claim. I'm not sure if they're claiming ownership of the person's face or using the face that they're typically stealing another video to put this face on to bring it down but ultimately that seems to be one of the most prominent take downs.
By that logic why should we ban any technology? Everyone here is unanimously against child porn even though we understand that it is already out there and that there is no way to put a complete stop to it. How are deepfakes that different?
The truth is this technology is dangerous in the wrong hands. I don't know if that means there needs to be laws created regarding it, but I know that the difficulty in stopping it shouldn't be used as an excuse to not even debate that possibility.
So we ban child porn to try to curb the production of more of it. Its production is what is harmful, and we do everything we can to reduce its production, which involves trying to reduce the demand by making it illegal and arresting people who obtain it. It has no legitimate use, so there is no collateral damage when we ban it.
Banning this technology would be more like banning cameras, since they can also be used to create child porn. However, it would be even LESS effective than banning cameras, because we are talking about a software algorithm.
Also, you say this technology is 'dangerous in the wrong hands.' Why is that the case?
I think the only reason is because it has the possibility of decieving people into thinking it is real. The real way to reduce this risk is to either spread the idea that all video is suspect, and you can't assume who you see on video is who it appears to be, or to perfect technology that can instantly detect when this system has been used.
If you are able to easily distinguish between true and false video, isn't the entire danger mitigated?
>So we ban child porn to try to curb the production of more of it. Its production is what is harmful,
We also ban computer generated child porn even though there is no victim involved in the production of that. If you use this deepfake technology and stick an 18 year old's face on a porn video it is legal but if you do the same thing with a 17 year old it isn't. The production of those two videos is the same. The only difference is the potential victim if the video became public.
>The real way to reduce this risk is to either spread the idea that all video is suspect, and you can't assume who you see on video is who it appears to be, or to perfect technology that can instantly detect when this system has been used.
That sounds great in theory, but people don't work like that. We are built to believe what we see. We all know that almost every magazine cover has been mercilessly Photoshopped and yet the images on those covers have still been shown to have a strong influence on people's own body image.
> If you use this deepfake technology and stick an 18 year old's face on a porn video it is legal but if you do the same thing with a 17 year old it isn't. The production of those two videos is the same. The only difference is the potential victim
For reference, artificially created child porn (e.g. in mangas, or in other ways created) is in a legal grey zone in many areas, including Japan and Germany. It is absolutely illegal in other, e.g. Sweden.
> For reference, artificially created child porn (e.g. in mangas, or in other ways created) is in a legal grey zone in many areas,
No. The grayness only applies to non-photorealistic works.
Once you start making photorealistic deepfakes or diptychs, where the end result appears to be photographic, the color becomes much more black and white the world over.
Today it's being used to swap faces; arguably you can do that with PS, but this tech goes way beyond a simple still-image face swap.
The swap is a "reconstruction" of the face at multiple different angles, taking on the characteristics of the original facial movements, tilt of the head, etc. In a video - way beyond what would be easily possible using PS or other "manual technology".
But today - it's just a face. However, nothing says that an entire person's body couldn't be swapped in the future. Nor that the system couldn't "dream" new scenarios for the background, etc.
Or that it could be done real-time, with full-on voicing in the other person's voice, etc - looking, doing, and saying things they never said - perhaps even in a place they've never been.
In the future, you could easily take someone else's entire person, and have them do jumping jacks in your living room while singing Jingle Bells - and it would likely look like an original video.
The only reason it's not being done now is the amount of processing power that would be needed to do it; gobs more than what most people can afford (along with the time and power needs - just note that for some of those facial swaps it takes hours of processing on a decent system for just a few seconds of video).
I'm not saying this tech is evil or should be banned, however; I think the tech is very neat and can open up some exciting possibilities for future products and uses. To that end, it's exactly like Adobe Photoshop, which is why it should be explored and played with, rather than banned.
Society has to come to an understanding that mere existance of a video that hasn't been cryptographically authenticated by special camera equipment is no longer evidence of anything.. Like it is with digital photography.
>5% of all people even know that the concept exists.
I believe more than 5% are familar with the concept photorealistic computer graphics. Machine learning based face transfer is just a newer iteration that significatnly reduced cost.
Agreed, although I think what you're describing is how it'll be sold, yet most people will surreptitiously use it to indulge in fantasies with "normal" people which will be even more damaging.
Got a picture of that co-worker you like? Congrats, now you have him/her in some porn. Interesting times for sure and I think we'll see a lot of flak and fallout in people's personal lives from their creations, far more than we see from "typical" porn use now.
I'm actually looking forward to living in a world in which pornography is so tailored, specific, and effective, that it becomes mundane. Maybe we'll have some other things to talk about after the world gets it out of its system.
Or, we end up creating a new digital addiction that partitions a class of society into preferentially existing digitally. Maybe this can just be a precursor to a VR caste that we never see in reality.
People will have karma scores generated by the number of people using their characteristics for their virtual porn. The web will be autocrawled looking for updates to the score.
I think people who are "caught" today looking at generic porn would be seen in a far more negative light if they instead had porn with the babysitter's/co-worker's/boss's face on it. I think that would be relationship-ending rather than merely an embarrassment or a career blip.
You would need more than one picture for your training set.
A minor detail, but the reasons these early products are compelling is that they are based of a large corpus of HD images taken from different perspective and angles and include various expressions and even stills while the subject is talking or laughing or expressing different emotions that a single picture of your co-worker won't.
I know we can create vaguely convincing synthesised voices, but I've not run across a program which can take my recorded voice, and convert it to a celebrity, or even shift apparent pitch or gender convincingly.
I was about to say that those clips still registered as a computer quite easily to me, until I got to the comparison with a human voice.
I think I've just gotten so used to that voice as the "google" voice that I automatically associate it with computers. It would be strange to meet the human that was providing the human voice in those samples.
"Deep Voice 2 can learn from hundreds of voices and imitate them perfectly. Unlike traditional systems, which need dozens of hours of audio from a single speaker, Deep Voice 2 can learn from hundreds of unique voices from less than half an hour of data per speaker, while achieving high audio quality." - http://research.baidu.com/deep-voice-2-multi-speaker-neural-...
Now imagine that with Tacotron quality, and you'll get that "strange" effect with anyone, meeting their vocal clone.
This is still text-to-speech, so it's not live-copying your intonation, but you could easily imagine a seq2seq network designed to do so.
I had a very strange experience like that recently when listening to Radio 3, the BBC's mostly-classical channel. They had an opera programme with guest presenters from the Met Opera in New York. The usual BBC presenters of course have British accents, and one of these American presenters had a particular accent that my brain latched onto as matching the sound of synthetic speech. I just could not suspend disbelief and convince myself that this speech - which rationally of course I knew was human - was that of a real person rather than some sort of AI assistant. It was a very strange feeling.
I did have a fever at the time, which might not have helped.
It's intent and emotion that I'm really interested in - which to the best of my knowledge, computer generation still isn't good at. (This is for VR games, so high quality voice acting is a priority)
Hence, if I could find a program which could reliably turn one actor's voice into another, I could use their acting ability, but with more characters and less requirement for them to "put on" voices. That's powerful because really good actors are thin on the ground, and also because trying to hold a different voice or accent can limit the quality of the main performance.
To add a data point, a friend and I made one of these over the weekend (with Trump and Michael Scott https://youtu.be/0Rexuh-VY6E)
It was shockingly easy to do. We're technical, but neither of us know a lick about machine learning. It took a couple hours to collect training data (we turned speeches/interviews from each of them into thousands of photos), and 20 hours to train the model.
In the future you could automate the data collection for a person even more, to the point where they just need to film a selfie of themselves for a couple of minutes in a couple of different lightnings, and boom, after you train their decoder, you could put them on any celebrity.
IANAL, but Crispin Glover sued the creators of Back to the Future II for using a mold his face as George McFly without permission [1]. While he starred in the first film, the second film used a mold of his face since he ended up not joining the project.
I’m not a lawyer, and don’t know what jurisdiction you are talking about, which celebrity you’re talking about (alive/dead/long dead, where do/did they live), etc, but I think the model copyright will be yours, but you will be restricted as to where you can use it because others have the right to make money with its looks.
Law works similar with copies of physical objects. For example, you are free to make a 100% copy of an iPhone, the Mona Lisa or a pop song and enjoy them at home, but you can’t display them in public or sell copies, or even give them away (did I say I’m not a lawyer?)
Discord banned multiple servers that r/deepfakes created and their reasoning was that it's being used for creating involuntary porn and so is against their ToS. I'm curious to hear your thoughts on the legal status of deepfakes as well as the moral and ethical questions behind the usage of the tech.
A thought experiment : I run a chat room site and will be adding support for user created chat rooms (something like discord for cryptocurrency communities with real time price widgets etc). Let's say I went ahead and got them to use my chat room, what would the consequences be? And what do you think about the moral and ethical questions behind hosting such a community? Am I now enabling the creation of deepfakes? By choosing to avoid hosting such a controversial community, am I imposing my moral and ethical reasoning on others (the free speech argument)? How do think about the deepfakes project, which is mostly being used for creating celebrity porn, but the tech itself is interesting and has applications(targeted ads or in stunt doubles etc)?
To be clear, I'm not trying to host a chat room for them, this is just a thought experiment. I don't like what it's being used for and don't want to support it in any way.
Using other people's visage without their permission is gross. It's immoral and unethical.
The idea that you can have morals and ethics without sometimes acting on them is also sort of incoherent. If you claim to believe it is immoral to do something and then do business in spite of that, it directly puts the lie to your claim. This is separate from whether you believe that there should be legal consequences or other government action attached to certain acts.
People act against their morals constantly and all the time. Morals often conflict with each other and a person may have moral ideals that they recognize as impractical to actually enact due to friction either with their other morals or reality itself.
> The idea that you can have morals and ethics without sometimes acting on them is also sort of incoherent.
This depends on your ethical system and your reasons for believing it immoral. For example, if the circumstances change enough to move something from immoral to merely neutral, some ethical systems will then allow you to do it. A utilitarian example: you might generally find something immoral (because it causes harm) but in the presence of somebody who gets great pleasure from it, you find it allowable (actually, it becomes a moral duty to the utilitarian, but you get the gist).
> If you claim to believe it is immoral to do something and then do business in spite of that, it directly puts the lie to your claim.
Not necessarily, doing business with someone means purchasing their services or products, it doesn't mean I agree with everything they do (with that money). Business is not charity.
Otherwise I probably wouldn't be able to buy half my black metal collection :)
But the issue here isn't whether training on copyrighted face images is legal. You can totally take copyright-free photos of anybody as long as they appear in public.
The more interesting question I think is whether you own the copyright to the 3D capture of your face. For example disney won't let you get away with using models of their characters. Don't I have the right to not let anyone use the "model" that I spent my literal life growing?
It's even more complicated than your Disney character analogy because if you can own the copyright of your likeness you can sell it. Imagine a young actress signing over her likeness to a predatory agency early in her career before making it big.
It's probably not relevant to bring in copyright here.
The restrictions for using your likeness or publicizing deepfake videos with your face on them come from other laws, not copyright. For example, "disney won't let you get away with using models of their characters", but (depending on how you use them) it's just as likely to use trademark restrictions even in situations where copyright law would let them use them. In a similar manner, the fact that you can take copyright-free photos of anybody as long as they appear in public doesn't necessarily mean (for example) that you're allowed to publish those photos like you can your own; or that this has any impact whatsoever on your rights to use copyrighted face images - you could have taken the photo, but you did not, so standard restrictions apply.
"If some law prohibits others to do that it does not really equal to "the copyright to the 3D capture of your face" even if some restrictions are similar, it's an apples vs oranges thing, it shouldn't be called that - terminology matters, because it implies different rules.
For this discussion, the main question is about restrictions for publicizing fake videos of other people without their consent. There are some restrictions, they're not copyright-based and also are quite different for every jurisdiction, as they aren't harmonized with global treaties like copyright is. Also, these restrictions are different for public people (e.g. the examples with Trump or famous actresses), and the restrictions for distributing videos with your face would be different.
By default you don't own your likeness, but you can file the paperwork to own it. Michael Jordan famously wasn't in any of the major basketball games during his prime because he owned his own likeness, so there would have to have been separate licensing deals apart from the NBA to include him.
It also shows why Discord isn't such a good choice for most people. They're obviously susceptible to knee-jerk moves directed by their public relations department.
Second, it's not actually a Discord server no matter how much they market it that way. Just like it's not a Facebook server. They just try to muddy the waters vs. real voice chat servers you can run yourself like mumble or teamspeak.
People should move away from centralized services where you are the product like Discord and use real voice chat servers. Now there's no ethical issue at all. Cut the gordian knot.
Discord is popular due to easy adoption into it. Mumble takes a bit more effort and I've wasted an hour with a friend trying to make it all work voice and roles. Discord just works. If Mumble could be more user friendly or similarly friendly it would go a long way.
"It also shows why Discord isn't such a good choice for most people. They're obviously susceptible to knee-jerk moves directed by their public relations department"
lol why because they banned people for creating fake revenge porn?
Legal status, i don't think is the issue. It's not illegal to produce likenesses, as long as they aren't published thus allowing people to believe the rendered facsimile is the real person.
I think publishing the impersonation is unethical since it allows people to believe it's the actual person, so it's fraudulent from that perspective.
By the time the next US election comes around it's very likely this technology, combined with Lyrebird etc. will allow anyone to make [politician] convincingly say or do anything they like.
I can't make my mind up if this will make the current fake news / filter bubble situation worse, or if it will make it so bad that people learn to distrust any kind of digital media entirely.
I'm still not sure, given that the use of these tools for political gain seems inevitable at this point how we will actually verify a video is real in the near future.
I find the more concerning thing is not that people will believe that a fake is real, it's more that they won't believe something real is real anymore.
Already nowadays the whole thing about fake news and alternative facts has segments of the population flat out not believing the truth. This may very well be greatly exacerbated in the future where, regardless of the amount of evidence you bring, people will just disregard it as fake.
If it can be faked, is it really evidence? I mean, you're worrying about people not believing assertions at that point. This is already something that can be faked just not by the masses yet and not easily. You should always look for multiple sources of anything and not just believe one source that shows you any evidence of something being true.
There was this essay posted on HN some time back, where the main thesis was that teaching people to be critical about media strengthened the positions of antivaxers, flatearthers and so on, because they did not rely on traditional media so they were then sort of considered more trustworthy. And now deepfakes etc throws just more fuel in the fire that burns trust, people end up more and more believing things that happen to make them feel good.
Believing 'the truth' is a huge can of worms here.
Verifying the legitimacy and accuracy of 'news' claims and representations has always been very difficult for the lay person. We are now well into an era when 'the truth' is even harder to find.
'Believe nothing of what you hear and only half of what you see' (Edgar Allen Poe) has now become 'believe nothing of what you see' too, at least until you have verified to your level of satisfaction...
We need to start thinking about what criminal justice will look like when digital recordings are all considered hearsay. That's bound to happen.
And perhaps it should... if average people have the technology now, how long has it been in the hands of police departments, newspapers, militaries, and intelligence agencies?
We tend to focus on fakes with high-profile people like politicians and celebrities, but it can clearly be done with everyday people as well. The author was able to produce a fake video segment of a well-lit, animated conversation involving his wife. She's directly in front of the camera, speaking and making hand motions for several seconds. How much easier would it be to insert a face into grainy security camera footage, or a shaky clip of a large crowd at a political rally?
Not much will change. People have been able to convincingly fake still photos almost since cameras were invented, yet courts accept some photos as evidence. What matters is the provenance, chain of evidence, and testimony of witnesses about the validity of the photo. Videos will work the same way.
The presence of one source can be easily dismissed, but if you have 500 independent sources that produce video of the same event from their own angles and timespans in an interconsistent way, it's far more likely that the recorded event happened as opposed to all of these different sources lying.
With a few sources you can build an accurate single model of an event. You then edit that event as you see fit and then generate as many "independent" sources from different viewing locations as you want. Then upload them to different social media sites at different times.
I am sure you can do that. The hard part would be building consistent histories to back up those uploads. An upload from my facebook account, verified by other incidental data like GPS tracking on my personal phone, would more merit treatment as an independent corroboration, whereas a random reddit upload from a throwaway account would not.
And the real verification would come from ultra-high-res captures from drone cams, self-driving car cams, traffic cams, etc that are constantly running and recording everything, so that you can't realistically fake soemthing without simultaneously compromising uber, amazon, walmart, etc.
> but if you have 500 independent sources that produce video of the same event
This only works in large events. What if you film a politician lying at a small event, but his image team makes 20 fake videos showing the event in a different light? I guess you're the liar then.
Firstly, people already know politicians lie. Nobody is going to care about your video in the first place.
Secondly, your analogy is wrong. The new hypothetical you present is equivalent to the old model where you are all giving verbal accounts and his staffers lied. If you are going to compare to an old-tech scenario where you one-up them with a more detailed recording than they have, you need to do the same with new technology to keep the comparison accurate.
Thirdly, why aren't you the liar?
Fourthly, you can always go one level up. Show the footage of them constructing doctored footage.
Really? How much harder is it to run the OP's script with 500 input videos as opposed to only one? Especially if they're very similar? How many people have the expertise needed to identify a faked video, and how much training would they need?
At that point there is no such thing as identifying faked videos. Instead you'd rely on the abundance of independent videos and see how consistent they are. Generating a lot of them doesn't do you any good unless you can insert them into the black boxes of the 3 planes and 40 cars in the area, the databases of all the drone operators, the internal storage of all the nearby smartphones, etc.
The difference is with testimony, the witness can be as moral and honest as possible but still miss report what they heard. With faking audio, it is either correct or someone purposely tried to fake evidence. That means that if the source of a recording could be verified, we know that it is very close to true. With witnesses, there is always doubt.
You're right to be skeptical of witnesses, but recordings are also not either/or. Even if you have some sort of digital signature scheme such that a camera can somehow sign its footage and the time and location it was recorded (a scheme that currently is fairly uncommon), how could it encode the circumstance in which the footage was recorded? Who is responsible for maintaining the signature scheme, and why can we trust them? In the absence of any aspect of that scheme, why shouldn't we closely scrutinize video and audio evidence as much as we scrutinize witness testimony?
The thing is that we have to believe someone when trying to understand what happened. A person, no matter how honest, can make mistakes. With a recording all that is necessary is to make sure that it is honest.
> And perhaps it should... if average people have the technology now, how long has it been in the hands of police departments, newspapers, militaries, and intelligence agencies?
To be honest with you, when I was younger I bought into this.
Nowadays....it kinda seems like with the open source community, consumers are getting stuff first. Not always, of course, but often enough.
And police departments are one category that I can be quite certain are lagging behind, they are definitely not getting stuff before consumers have it.
Yes. As a grad student in ML, on a few occasions I worked with sponsors under the US military trying to help them use some code from projects I'd open sourced, and it was always a struggle.
Frankly, the suggestion that government agencies -- let alone local police departments -- have sophistication in AI/ML years ahead of publicly known work from academia, industry, or even side projects on reddit, will verge on comical to anyone who's actually worked with these folks.
You're not talking about a monolith. Sure, if you've met 20 cops, it's unlikely that you've met any cops who are technological geniuses. Most LEOs haven't had a reason compelling enough to figure out how to fake a video. Can we be sure that no LEOs have? If we can be sure of that, can we also be sure that no intelligence agencies have? That seems unlikely.
The longer time goes on, the more you realize that what companies can get to the public in a year for $1000, takes 10 years and a multi-billion dollar contract for any government (other then priority bureaus like NSA) to obtain. Like good luck getting anything through government bureaucracy and shitty subcontractors.
> if average people have the technology now, how long has it been in the hands of police departments, newspapers, militaries, and intelligence agencies?
Most of those people are average. Intelligence agencies can be assumed to have more advanced technologies in a number of areas but the gap between what they have and what's out in the more public world has narrowed.
We've already had image tampering technologies for a long time but there's not been a significant problem with fakes in the press or as evidence.
It is an important issue anyway and figuring out ways to being able to differentiate the genuine from the fake, with cryptographic signing in cameras for example, are worthwhile.
Ultimately, though, don't you have the same problem with things like contracts? Especially with digital signatures, but there is not actually a reliable way to determine whether a signature is authentic. And countless people are going to jail because of complete junk science like bite mark analysis. This seems like the least of our problem.s
It's time for cryptographic signatures in cameras (phones and otherwise) so that you can prove something is unadulterated.
Apple signing key -> your phone's key -> signature over hash of the media (edit: for video, embed signatures every so often to you can verify a shorter clip out of the whole). Embed Apple's signature and your pubkey in the metadata of the image, and TwitYouFace will verify on upload that it's authentic.
Of course, you can't fix stupid unless you talk about a complete overhaul of public education... And even then, people are people.
The phone has enough metadata to make that easy to expose, at least in the simple case. There's GPS location, time, and even your Face ID verification (on iPhone X) to prove that it was really you holding the camera.
If you make your fake video after the event, the location and time will be obviously off.
That's fine, if you publish that video it will be signed by ggg9990 and you will have to decide if you want to be associated with it. Watchers will then have to decide if ggg9990 is a trustworthy source.
It boils down to trust, not whether the original video was real or not.
To put it another way, reputable news organizations already go to great lengths to ensure that their stories are well sourced. At the end of the day, they are putting their reputation on the line.
Try it. Seriously, try and record a realistically looking recording off the screen. Movie pirates have been trying to perfect it for decades, and even their best attempts look obvious.
Unless you build DRM right into camera sensor (which is, admittedly, not that hard), you can use an FPGA chip to receive your source video in HDMI format, convert it to CSI interface and send it to the phone SoC over the camera flex cable.
What if all legally acceptable cameras have to use multiple sensors?
In other words, 2 main stereoscopic sensors, each recording their cryptostream to their storage. In addition to that, you have 2 UV sensors doing the same thing, and 2 IR sensors as well?
The idea being that it's much, much harder to fake something if you have to take IR and UV into account.
Isn't this also where something like a blockchain is useful? Where data is dumped into it realtime by this mythical secure camera, with all its associated metadata.
Then you can ask: "does the time the data was added agree with the time the camera says the data was recorded?" "Does the blockchain agree that the data hasn't been modified after it arrived?"
Add to that the possibility that there are multiple cameras all recording this incredibly detailed video from different angles, and the evil counterfeiters will (hopefully) have a dickens of a time faking all this.
Although with all this wonderful processing, networking and storage power to enable this secure camera, you can probably create a faking system too.
So what do you propose? At some point a system of trust must be established or are you going to travel around the world, attend every event, see everything with your own very eyes, run the analysis, crunch the data etc... to be convinced of the information?
It's possible to simply believe less. Not only will you be less likely to get screwed over, you'll also be happier, when you no longer have a big handle on your nose that the bastards can use to lead you around.
Hasn't ios DRM continued to get more effective over time?
If there are no known exploits, a signed video, marked as having a network based timestamp, is at a minimum incrementally more reliable than a video without such metadata.
This is at the same time a great solution, and a useless one. Sure, you could verify that a video came from device X by carefully comparing the signature of the video was signed by the public key of the phone, but who on earth is going to do that for a video their aunt posted on Facebook?
That’s a big challenge: for big news stories it’d be useful if it meant that TruePatriot420 has to explain how their phone was used to record a candidate’s private conversation in a different state but I think many of the most damaging things for many victims won’t hit that level of scrutiny. If some horrible person tries to smear their ex, most of the damage will come simply from circulation.
What might be worth trying would be some sort of verified badge if this became common enough that players could meaningfully highlight its absence, similar to the way browsers are now showing non-HTTPS sites as insecure. If Facebook badged the video with a big “this iPhone video doesn’t have the usual signature” message that might slow people down.
For the deepfake porn or fake news problem I also wonder whether there’s a useful fuzzy ID service where a player could display a “this is substantially similar to <source video> and may be a copy” warning which wouldn’t be so inaccurate that nobody would heed it. That could be really important if we started seeing political fakes where someone is shown accurately but their words are faked.
Facebook / Twitter / etc verify the crypto signatures on media ingress. Then they can give proper attribution and show a checkmark or something that attests the media hasn't been altered.
So in your feed, there will be some media with checkmarks and some without and you can choose which to believe more.
Alternatively, a competitor could come along and reject all media that doesn't pass cryptographic verification.
It would absolutely have to become more consumer and lay-user friendly.
Of course, depending on how that's implemented it could just be another vulnerability in easy-to-use proofing.
But if there was some way to add an abstraction on top of the raw keys, keeping them still accessible if the user is keen, then that might be a step in the right direction.
Your fears are greatly overblown. Haven’t we been able to “Photoshop“ people convincingly for a very long time? People are less sceptible to this than you think.
Even when scandals do break, it’s not a picture or video that’s proof, it’s a picture or video that produces witnesses.
or there was that time a german variety show 'false flagged' the germans by doctoring a video of yanis varoufakis to have him 'stick the middle finger to germany' during a talk he gave (https://www.youtube.com/watch?v=Vx-1LQu6mAE). There were witnesses; but they didn't stop the confusion and outrage that ensued due to the effects of viral media. Would this pass a sophisticated technical analysis of the video? No - is there time and public understanding for the results of said sophisticated technical analysis to prevent the damage before it is done? no.
The notion that witnesses or forensic analysis can combat the speed of information, especially across borders, is a bit quaint.
It's of course technically possible but it's also much more difficult. You basically have to photoshop a long series of frames and make sure that it looks natural and consistent in sequence. You have to take into account changes in lighting, focus, motion blur, things passing in front and behind the object etc...
In your example they modified a very clean source video, image is perfectly stable, very little motion and they edited a very small portion of it for a very short duration. TFA is about replacing one person by an other while they're talking and moving on a potentially lengthy video clip. Doing this convincingly by hand would be hard work.
I think a good example of the difficulty is with the infamous "moustache removal" for reshoots of Superman in the Justice League. It cost a fortune and the results were not exactly seamless. Also the re-created Princess Leia in that Star Wars movie, not bad but definitely still in the uncanny valley.
Its something entirely different to fake a face instead of a finger in a video. Then again, the deep fakes I have seen so far arent really that convincing either. But one can only wonder how much you can improve such a thing. With politicians of whom a wide array of video material exists and were you have the option to create an incident, instead of altering existing ones, the possibilities are really interesting.
I posted the Böhmermann incident 2 minutes prior and see it more as an example for the state of the art in video manipulation. I think its hard to see the response back then as an example of future interaction with the topic of video fakes. Which when it comes to altering faces in videos is still in the future.
I edited the post you replied to with "in a video." in the first sentence to clearify
It's ironic that on a discussion about copying images of parts of someone's body over to images of another person's platz would ask "did we really post that link at the same time, or did you edit your other response to match?" i.e. copying over parts of one person's comment to another person's.
(And the GAN is the neural network model which does these transformations)
We've always had AfterEffects and CGI. Some viral videos have been proven fake , but for the most part a lot of those faked videos were low res to make it more forgiving to fake them.
Deepfakes has made it so the average user, with no knowledge of Aftereffects / composition, to basically run a python script so long as they have high res images + several angles / expressions of the target user. That same folder set of images can be replicated in multiple scenes with just the cost being GPU rendering time
Deepfake videos can also have a much higher resolution than "viral videos" with aftereffects to fake effects.
Deepfaking also leads to a lot of legality issues as well.
TL;DR deepfakes does make faking videos much cheaper, higher resolution, and easier now
My point was that they don’t lead to legality issues. This has been the case since the invention of the digital camera. No matter how convincing digital images (now video) has seemed, they aren’t “proof” in the eyes of the court because they can be faked. There’s nothing new here.
> or if it will make it so bad that people learn to distrust any kind of digital media entirely.
There are already fake news sites out there that are "so bad" you'd think nobody would believe them, and yet...
Still, it's important we inform people this is possible, so that at least the more reasonable among them can view future videos through a skeptical lens. I'm sure by now many HN readers are not surprised by this technology, but many outside of HN would be blown away.
Herd immunity? Perhaps naive people believe those spam sites because smarter people believe the better spam sites that share the same narrative...
However, distrusting digital media isn't the solution either - there are plenty of trash magazines publishing conspiracy theories and other absurdities. Plus, digital media has revealed huge segments of America are becoming culturally isolated and undemocratic. A return to print media might shift funds back to deep professional investigative journalism that covers those issues... or just give owners better returns.
Putting aside the US or first-world elections/media..
It's worrying to think of how this technology could be used in parts of the world where there is much less widespread understanding of this level of digital manipulation.
I don't think this will be a big issue. The same could already be done with still images years ago, if not decades. And with enough time and resources, the same can be applied reasonably well to videos, manually frame by frame or using CGI.
It takes a lot more to make it undetectable as fake though, and I guess the same applies to deepfakes. Once people get used to the fact such face swaps can be done, they'll be more sceptical and demand realness verification in case of published sensitive material.
I'm not so much concerned about people making politicians and other personalities say something they haven't, rather I worry that these personalities will now have plausible deniability for any footage of them doing or saying something embarrassing. It's always harder to prove a negative and in this case you'd have to prove that it can't be a fake. Now imagine figuring out which video is real amid a flurry of fakes.
How will we actually verify a video is real in the near future? I’ve been thinking about this a lot lately, for many of the same reasons.
I think one answer can be found in device-level signing of images and A/V feeds. A recorder can integrate its cryptographic signing and keystore on the same chip as its CCD/audio controllers, like Apple’s T2 chip does, so that it has the ability to sign direct feeds from the hardware as being such. This might be more complicated for A/V than still images, since a large enough recording would have to be written to disk. Perhaps some rolling scheme could be devised.
For an added measure of verification, the SoC could upload them (optionally encrypted) to a distributed public store like a blockchain or IPFS, proving that they were created at/around a certain time.
This seems like it would be useful in concert with other tools for news agencies (who should be leading the way in providing as much documented research and evidence for things as they can, as a matter of course, while protecting their sources), law enforcement (who is losing the public’s trust when it comes to the chain of custody and testimony), etc.
We jumped very far ahead both in how much importance we place on digital documents and in how easily we can forge them, so we need to catch up fast on finding robust mechanisms for verification.
People should never have been trusting media in the first place, digital or otherwise. And for the most part, they don't have to. Humans believe what they want to believe or are told to believe. Evidence has little to do with it.
It will make things way worse, since politicians from now on can and will plausibly deny any video evidence about them. Candidates will literally rewrite history instead of merely suffering from amnesia and any real evidence will end up in battles between one TV show 'expert' against another.
On a bright side, young aspiring actors can make porn movies to get some cash early in their career without necessarily having to fear it will ruin their career many years later - those movies will exist anyway.
Just like there is a social system in place to sift fact from rumors (via watercooler chatter, snopes, networks of trust), at least to a degree where society doesn't collapse into chaos, in the same way will we be able dampen any damage that this new technology might cause. I think this whole "fake videos of politicians" will become a way smaller issue than what doomsayers predict it will be.
It's likely that we'll soon enter a new era of technological cat-and-mouse as tools are developed to automatically detect artifacts caused by digital processing and counter-techniques are developed to minimize the appearance of those artifacts.
Something that comes up a lot on the deepfakes sub is the notion that this should make people more skeptical. They use the phrase "If anything can be real, nothing is real" and I think it makes a lot of sense.
That's not necessarily a good thing. Sometimes it's necessary for people to believe something factual. Blanket skepticism can be very destructive in these cases.
Kerry was getting photoshopped into propaganda photos in 2004. The new thing isn't fake documents. The new thing is a highly scalable social media infrastructure to share false documents widely.
What is "Trump watersports"? I haven't heard of it and that is a searchterm I am not super interested in getting google NSFW results from. Is it what I think it is, I haven't heard about this at all?
The rumour that Russian intelligence have video of Trump getting pissed on by prostitutes, the #goldenshower hashstag is used I believe. No idea of its veracity.
There is a conspiracy theory about trump urinating on a prostitute on his visit to Russia some years ago. The theory states that Putin has a tape of it.
In the realm of innocent fun with this stuff, just how much time would it take to insert myself into the entirety of a movie? I'm just imagining how fun it would be to invite friends around to watch something we're all excited about, only to convincingly insert my face into the main character's.
Or, once advert industry learns about it, how about replacing all the things on the table in Audrey Hepburn movie with BRAND NEW LEMONMELON FLAVOURED OREOS! :)
Ideally on the fly, with each box sold to the highest bidder, adsense-style.
I've already encountered a minor version of this with the film "Demolition Man": when I first saw it on UK TV, it mentioned Pizza Hut at one point, but more recently it was on again and that line was overdubbed really badly to say "Taco Bell".
It's a weird thing to watch an film turn out differently to how you rememebered it when you're not told it's a different cut. Feels like low-grade gaslighting.
(Another example is the translation of "天下" - Tianxia - in the film Hero, which is both critical to the film and translated differently in different versions)
I have also thought about this, and whether I could live with myself after making a huge pile of money to relentlessly annoy every other human on Earth.
I conceived it as video bloggers setting up a dynamic replacement volume in their scenes, and compositing a 3-D scene for the sponsored product into it as the stream goes out. One viewer might see an open Coke bottle. Another might see a half-eaten package of Oreos. Another might see framed photographs of adoptable rescue pets. All the vlogger does is avoid entering the ad-volume and the software does the rest.
Inserting into old movies would require a bit more sophistication, I think. You have to discern the geometry of the scene, model it, then discover the insertable volumes, match the camera movement in software, and finally use a shader to match the object textures to the film grain.
> Inserting into old movies would require a bit more sophistication, I think. You have to discern the geometry of the scene, model it, then discover the insertable volumes, match the camera movement in software, and finally use a shader to match the object textures to the film grain.
Only if you're changing volume. We can start with swapping flat surfaces (say, cereal boxes).
A cereal box face is still two triangles (or a quad). It's simple geometry, but still geometry. If you can do two triangles, you could do 12. If you kept it simple, you could get away with image planes and occlusion masks, I guess.
I'm guessing this would dovetail nicely with up-converting 2-D movies to work with VR rigs. If you can reconstruct a scene well enough to fake some depth, it wouldn't be difficult to insert extra models into it.
The dystopic endgame would be to automatically generate the models from the 2-D video and randomly fill some of the volumes that don't interfere with the existing scene with advertising models.
There was a YC company in my S11 batch that had this, 6 years ago. Not sure why they didn't pursue it, but I think in part it was too hard to get brands on board.
Many, many hours - depending on what you have for a system.
These small clips can take several hours to render; my (unwarranted) assumption is that the machine being used likely has the equivalent (minimum) of a 1060 or 1070 (at least, that's what I'd use).
Now - if you have the resources to own or build a multi-GPU machine with scads of RAM and the very best CPU - it will still take hours, but you might be able to get it down to "less than a day's worth".
That's my best guess based on what I have seen so far and my own limited personal experience with deep learning and ML tech (nothing involving face swapping or such - more mundane things thru MOOCs I've taken in the past, using my 750ti SC as the GPU with tensorflow).
I am reminded of Ghost in the Shell : SAC where they mention that video and photographic evidence became inadmissible in court due to how easy it was to produce fakes.
I think we are perhaps 1-2 years away from this becoming a practical consideration for the judicial system.
I think the reality is that there's a burgeoning opportunity for cryptographically secure chain-of-evidence cameras. Sure, no protection is perfect, but if a manufacturer can reliably state "It will require [X] level of resources and effort to falsify data in our camera", then a lawyer can present footage and ask the jury to draw their own conclusion about whether a criminal's claims of "fake footage!" are reasonable.
Canon used to sell a cryptographic addon that could 'prove' a file was generated by a specific camera. I couldn't find it when i looked just now, i think the implementation was broken.
So everybody is going to go out and buy new phones and cameras?
Also, using Meltdown and Specter as examples, can these companies actually build cryptographically secure devices. It's easy for a manufacture to say "Our device is secure", and it's easy for the judge to say "You go to jail for life", but it creates a big mess in the legal system when we realize 3 years later that someone extracted the private key from a device, framed somebody, and that person had been put in the electric chair between now and then.
> So everybody is going to go out and buy new phones and cameras?
Probably not. That sounds a little ridiculous. But probably so in the case of security cameras or bodycams or dashcams.
> but it creates a big mess in the legal system when we realize 3 years later that someone extracted the private key from a device, framed somebody,
Yeah, I covered that. Did you read my second sentence, or only the first one?
Evidence can already be faked. Eye-witness testimony has been faked for millenia. Photographic evidence started being faked shortly after the invention of the camera. What you do is, make sure the jury knows how hard it would be to fake, and let them make an informed decision.
Does that mean it's technically possible to frame someone? Yes! Just like it is today. Sucks to live in an imperfect world.
> and that person had been put in the electric chair between now and then
That's a valid argument against the death penalty. But that's a different subject entirely.
I liked that scene in GitSSAC because it is animated, so any proficient artist could create a realistic fake. How do you show an animated fake when everything is animated?
Is it just me or are these "deep fakes" just not even very convincing?
I felt the same way about all the Nicholas Cage ones that were viral recently. It's so obviously a very shallow 'effect' and can't compensate for very basic differences in bone structure, hair, etc.
They'll get more convincing. Then we'll actually get better at recognising subtle differences. Then they'll get even more convincing. Someone will develop algorithms to identify those fakes. Then they'll be further developed and improved to the point that they're impossible to distinguish at every level.
> Someone will develop algorithms to identify those fakes.
One twist: don't adversarial generative networks already work by including a "fake detector" and beating it? This cat-mouse game might last, like, all of a week.
Most of the submitted deepfakeson reddit are the first couple experiments of inexperienced amateurs trying out the technology. The worse examples are not representative of how the technology works, they usually indicate a flaw in how that particular person build their data; and the best examples are almost undistinguishable.
The technology works now, it's just at the stage where it's simple enough for a random person to build something but still tricky enough so that not everyone can build something good; it works, but not necessarily right out of the box.
This is just the beginning though. The iPhone 1. Imagine the iPhone X version of this. It's open source now; it was made by one person. Now imagine it after 10,000 people have worked on that code. This will become so streamlined and efficient you won't be able to tell at all. Some of the good ones are already extremely convincing, you just have to give it appropriate training data, choose a good model to swap with, etc.
Something to keep in mind is that part of the problem with these fakes is the shallowness of the training data. If rather than, say, 30k images being used - 3 million were instead - that would cause a great change in the quality.
But - that would also take a ton more processing; instead of a clip taking several hours to create, you might be looking at it taking a few days. That, or you'd have to throw a ton of money into better hardware to handle the extra load.
I've seen a number of these deepfakes, both SFW and NSFW, a lot of the unconvincing ones are done by people that haven't specified a long enough training time (12+ hours normally IIRC) or produced enough res image data sets / were testing the data for first time.
You have to find a target actor/actress that looks similar to what you are deepfaking (e.g. family member)
It makes the use of stunt doubles easier as well since the actor's face can be mapped to the body double. And it isn't like this point in time wasn't predicted by just about everyone in CS. Given enough computer power and the right algorithm pretty much any data is fungible, video or otherwise.
But we've just entered the most dangerous period when most people still believe what they see in a video is what was originally videoed. It is when people will use this technology for deception and get away with it like they did with Photoshop before 'shopped!' came into the more general populations consciousness.
A cool, possible solution would be to have every participant in the video to digitally sign the final file to verify that they're actually in the video, consensually. That way, if anyone finds a deep fake video, it would be easy to dispute it since it doesn't have the person's signature.
Of course, this would be exceedingly difficult in practice, since you'd need the signing to be opt-out instead of opt-in for it to be truly effective.
So, what's the difference between this and photoshoping someone's face to create fakes? Both are the same thing fundamentally, right? I understand the technological significance of it, but not the philosophical/legal etc differences of it.
It’s a good question. Is the answer that to a lot of people video is their reality, whereas static images is some crusty technology from the decades before they were born?
For those of us around when the optic nerve implants are available, it's going to be a wild time.
You'll be able to experience your own subjective reality, completely differently from anyone else.
Want to replace the grocery store with a Minecraft dungeon? Sure! Want to see dragons soaring in the sky above? Sure! Want to pretend your self-driving motorcycle is a speeder bike from Star Wars? Sure!
In these discussions, its easy to think about the person harmed by a (known to be) fake video of them having sex.
I think it's odd that we purposely ignore the joy created for the other side of (thousands and millions people). In a utilitarianism view we'd weight these sides in a 1000:1 ratio because of this.
I don't think a video that I know and everyone else knows is fake is very harmful. And it's clear that many people get joy out of it. I think it should stay (though it's here to stay regardless of the law).
I'm not quite sure you can just hand wave away the harmfulness. It's very possible that a lot of people would feel highly violated by it. It basically takes away your control over your modesty and to an extent even the autonomy of your body. Yes, the videos are fake, but that doesn't change that the entire premise is for people to imagine it's real.
I'm much less decided about the legality side of this, but I think the from the moral side, this is pretty hard to defend. Obviously this is taking things a bit to the extreme, but where exactly do you draw the line from a utilitarian perspective? Is a gang rape morally permissible if the perpetrators enjoy it enough to outweigh the victim's suffering? Why should people ever be (morally) allowed to derive their pleasure from another's suffering?
I can’t really see much difference between the original and the faked version. It would be better if it swapped the hair too. (Yes, I do have problems recognizing faces in real life.)
I keep wondering about how this approach can be composed, vs just doing a single replacement. Example: replace person a head in this video with person b (current examples), make composed person more tan, swap hair on composed person with this other person c hair, add muscles, change shirt with this other shirt, etc, etc. It really seems like the same type of operation reapplied repeatedly. Could be powerful character editor type functionality.
When I was a kid in the mid 80s, I got a racing game (Revs) that let you enter names in for the CPU racers. Looking back it seems ridiculous but I remember it being such a thrill to pretend I was racing against "people" I knew. Even that was exciting to me, so what's happening now is absolutely mindboggling(!)
Before movies, why not insert friends into your video game?
Or how about creating a company that sells profiles each of which would be a collection of high quality photos/videos. In turn if you have a video game or movie you can get a random actor with the body you want and swap the face with the face you want?
Or even better create a service where individuals can upload their own photos which they can get some of the profit if their profile is used.
Another Idea make Bad Actors more convincing with "stock reactions". Track the face of your actor copy Anthony Hopkins reaction face over it and than use deepfake to get the original face back.
Within a few years there will be apps that take movies of two different people and allow you to just tell it the parts to swap. People will be able to easily make their own videos like this easily. Banning the technology is a temporary and questionable fix at best.
When I see this I think of the future of human robotics interfaces. A robot with a screen for a head, will be able to express anything we do, and tailor its responses to the environment. That, in turn, will probably make them much more acceptable in society.
I think it means you'll be able to replace all actors in all films with Arnold if you like.
Films may come with default actors and the viewer will be able to choose alternate body types or faces.
Would be interesting to see how that affects Hollywood celebrity. Will the celebrities do the movies still or just use stand-ins and lens their visage to the final cut?
At what point will the star of live action movies be a computer generated skin?
If we treat the normal rate of progression, we'll see a year 2023 version for 5 bucks off the App Store, your phone will live-convert the video you're recording at your drinking party with your friends into an all celebrity event. Probably with voice swap. Your drunken roommate's scratchy voice will be ML'd into Frank Sinatra, autotuned, face swapped, and uploaded for a few laughs to facebook.
It's here to stay whether you like it or not.