
Facebook Discovers Fake AI-Generated Profiles - dpflan
https://www.nytimes.com/2019/12/20/business/facebook-ai-generated-profiles.html
======
jedberg
It's ironic that as we move into a world where anyone can be a "journalist"
and anyone can have a voice, we need professional journalists more than ever.

Now a journalist will not only need training in writing well, conveying a
story with facts, and investigating, but they will also need to become experts
in identifying fake source material.

In a world where anyone can make a video or photo of anyone doing anything,
we'll need to rely on reputable journalists to vet that material for us.

Saw a video on YouTube of the President saying something bad? No idea if it's
real anymore.

Sure, a lot of this technology has existed for a while, but it was usually in
the fringes (see The National Enquirer). Now it's going mainstream.

And the worst part is, there are a whole group of people who will not trust a
professional journalist (sometimes with good reason) but will trust anyone
online who happens to provide evidence that boosters their existing opinions.

~~~
nemild
One of the scariest quotes for me was Facebook's Little Red Book in 2012 (a
book handed out to every employee) that had this quote:

> "When everyone has a printing press [i.e., Facebook], the ones with the best
> ideas are the ones people listen to."

In retrospect, it gives a ton of insight into the flawed beliefs within the
company.

[https://twitter.com/nemild/status/1006533287378968576](https://twitter.com/nemild/status/1006533287378968576)

[https://www.businessinsider.com/inside-facebooks-little-
red-...](https://www.businessinsider.com/inside-facebooks-little-red-
book-2015-5)

~~~
ekianjo
> "When everyone has a printing press [i.e., Facebook], the ones with the best
> ideas are the ones people listen to."

Should be:

"When everyone has a printing press, the most persuasive ones are the ones
people listen to"

~~~
roywiggins
That's a tautology!

~~~
TeMPOraL
A good tautology can be helpful in revealing errors in reasoning. Like here,
the sentence "ones with the best ideas are the ones people listen to" sounds
reasonable; the equivalent statement "best reasoning is the most persuasive"
is clearly wrong.

~~~
drdeca
“Stating tautologies is sometimes necessary in the presence of those who
believe self-contradictions” - paraphrased from weird sun twitter

------
mgiannopoulos
Reported by Snopes for months now, not “discovered” by Facebook
[https://www.snopes.com/news/2019/10/11/pro-trump-outlet-
link...](https://www.snopes.com/news/2019/10/11/pro-trump-outlet-linked-epoch-
times/)

~~~
tempodox
Would anybody expect the truth from FB? No way they wouldn't spin this to
their best possible advantage.

~~~
mgiannopoulos
I would expect the NYT to do some proper reporting though instead of
copy/pasting Facebook’s press releases.

~~~
joshspankit
What leads you to expect that?

------
rvz
And so it appears that the inevitable has 'finally happened'.

Mark Zuckerberg's website is now a place not to be trusted as a source of
news, finding friends and meeting others if there's a possibility of AI
generated profiles being proliferated on the website.

I've heard that meeting real humans outdoors makes it less likely to be
deceived. Their authenticity is vouched via word of mouth and experience by
other humans, rather than robots on the internet.

~~~
MuffinFlavored
Require manual profile validation for new Facebook accounts?

Probably would get downvoted if I said SSN + a credit check.

Phone number required + needs to be validated + can not be reused + can not be
Twilio.

Prove your address? Prove your ISP cable bill? I don't know. Everything I
guess, people here are going to tell me it's an invasion of privacy and they
don't trust Facebook to do it.

But they'll let CreditKarma have every bit of financial information related to
them possible.

People will apply for a credit card online (employer, address, social,
income)... make a universal login process for that.

Maybe like blue Twitter verified checkmarks next to all profile names.
"Verified human"

~~~
Barrin92
or simply charge a few bucks for account creation.

~~~
pushpop
You then run the risk of excluding developing economies. Unless you adjust the
payment to be region specific, in which case you’d just get foreign powers (be
that corporations or state sponsored) using their wealth to fund hordes of new
registrations from said developing economies; thus negating the benefit of a
sign up fee.

~~~
yc_2345
As someone who grew up in a developing economy, I find this view extremely
patronizing. Usually, what happens is that a local company fills that demand
gap.

Also, I think it is good for all parties concerned, because most developing
economies will have a big problem becoming developed economies if their people
are under constant surveillance from entities who may benefit from their
continuation as developing economies.

Case in point: the US dollar is a horrible currency if looked at in isolation
- after all, why the heck should the rest of the world pay American kids lots
of money to have a great time at college and then turn around and default on
their debt? Student loan forgiveness is almost a certainty over the next few
years. But people still flock to the US dollar because it is _far better_ than
the other places money could go. There may be a hint of conspiracy theory in
this, for sure, but it does not require a wild stretch of imagination to
expect that the country where Facebook/WhatsApp/Instagram is headquartered
actually has a real incentive to foment violence in other countries and make
their own economy and thus currency look better in comparison.

~~~
pushpop
> _As someone who grew up in a developing economy, I find this view extremely
> patronizing. Usually, what happens is that a local company fills that demand
> gap._

...which means if a western subscription services wants to compete with a
cheaper local equivalent, they need to charge a comparable local rate. That’s
not patronising, this literally how all businesses work when having to compete
with regional price differences.

As for the rest of your post: it sounds like you have a proverbial axe to
grind. I’d be the first to agree that there are some serious issues with some
of America’s domestic and foreign policies but none of your points are even
remotely relevant to the conversation (and some weren’t even remotely based on
reality)

~~~
yc_2345
>>...which means if a western subscription services wants to compete with a
cheaper local equivalent

Interesting comparison. Let us discuss that a little further.

[https://www.nytimes.com/2018/10/15/technology/myanmar-
facebo...](https://www.nytimes.com/2018/10/15/technology/myanmar-facebook-
genocide.html)

So Facebook provided free internet to Myanmar, and it was better than the
"cheaper local equivalent" because it was actually free. Soon all the ISPs in
Myanmar were out of business. Facebook was now synonymous with the internet in
Myanmar, and suddenly it became a lot easier to influence people into doing
pogroms. Now clearly a "western subscription service" successfully competed,
and unfortunately, the pogrom did not affect the "western subscription
service" in any way while millions of real humans became refugees.

Generally, people's answer to this is something like "yeah, it is too bad the
people were such idiots to be provoked into such violence" or in some way
blaming the naivete of the local population. But imagine if Facebook was
actually asked to pay compensation for every affected person, or to stop
business operations in Myanmar altogether. See, you can _actually_ do that
with a local company because they have some real skin in the game.

To complete my point, I don't actually give a damn if a "western subscription
service" can compete for my business unless I have a way to drag Zuckerberg
(figuratively speaking of course, a minion of Zuckerberg will do) to my
country for trial and then send him to the local prison if found guilty of
some local law. Think of it as "terms of providing service".

>>it sounds like you have a proverbial axe to grind

Indeed I do. It is called "stop patronizing the people of my country".

>>none of your points are even remotely relevant to the conversation

If you feel so, that's OK, although I would obviously like to see a more
specific refutation.

~~~
pushpop
Facebook is an edge case because that’s a problem in every country and frankly
that whole “free internet” mission was clearly a bullshit move from the outset
(even people in the west weren’t the slightest bit convinced by Zuckerburgs
motives). However that still doesn’t prove your claim that every western
company exists in developing counties specifically to further your economic
strife (which is the accusation you’re levelling against the west).

If you want to argue that American corporate culture is a toxic breed of
short-sighted greed or that developing economies are better served with local
businesses then at least try to do so without all the conspiracy theories and
personal attacks. It’s not exactly a hard topic to argue rationally after all.

> _Indeed I do. It is called "stop patronizing the people of my country"._

It’s not “patronising“ to say that companies need to offer regional rates to
compete in regional economies.

You might argue that its unfair to put local companies out of business or that
western corporations deserve no claim in developing nations; but that’s
categorically not what “patronising” means (and also a very different point to
the one I was making).

Don’t just take my word for it though, look the word up in any of the numerous
online dictionaries :)

>> _none of your points are even remotely relevant to the conversation._

> _If you feel so, that 's OK, although I would obviously like to see a more
> specific refutation._

I wasn’t trying to refute your claims (mainly because most of them were far
fetched - even by your own admission). I was saying they’re a complete tangent
to the topic of conversation.

Look, if you want to bleat on about the evils of western influence on
developing economies then fine. But at least acknowledge it’s a largely
unrelated rant to the conversation that precedes it.....and also don’t make
accusations about other HN posters when you don’t understand the terms you’re
accusing them of.

~~~
yc_2345
>>Don’t just take my word for it though, look the word up in any of the
numerous online dictionaries :)

>>You then run the risk of excluding developing economies

"Patronizing: apparently kind or helpful but betraying a feeling of
superiority; condescending"

I see. So, let us back up a little to your first statement. Why is the risk of
excluding developing economies a problem (for the developing economy), again?
I will be very happy to take back my assumption about what you implied there,
provided you can give me an explanation for why it is a "risk" and who
actually bears the risk.

>>Facebook is an edge case because that’s a problem in every country

It is not an edge case if it leads to making assumptions which are clearly bad
for our future. Because Facebook isn't getting any real punishment, people are
only going to say "Nothing happened to Facebook. Why should the other
companies worry about lying to governments/ lying to customers/ privacy
issues/ data collection / retargeting / lookalike audience" etc.

>>However that still doesn’t prove your claim that every western company
exists in developing counties specifically to further your economic strife
(which is the accusation you’re levelling against the west)

I am not sure why you feel that way. Specifically, what I am saying is that I
don't like people starting off with statements such as "you then run the risk
of excluding developing economies" as if that is automatically a bad thing for
the developing economies.

~~~
pushpop
> _I see. So, let us back up a little to your first statement. Why is the risk
> of excluding developing economies a problem (for the developing economy),
> again?_

This is the question you should have opened with :)

I never said it was a problem for the developing country. I was talking purely
about the economics of Facebook. If _they_ (Facebook) want to expend their
demographic then they would need be affordable in each economic environment.
Whether _you_ (the respective county) want them there or not is another matter
entirely.

Frankly I’d be happier if Facebook failed in my own country rather than
expanded their global presence into yours as well; but that wasn’t the topic
being discussed at that time.

> _It is not an edge case if it leads to making assumptions which are clearly
> bad for our future._

Technically it still would be when using the correct definition of edge case
;)

> _Because Facebook isn 't getting any real punishment, people are only going
> to say "Nothing happened to Facebook. Why should the other companies worry
> about lying to governments/ lying to customers/ privacy issues/ data
> collection / retargeting / lookalike audience" etc._

I agree that’s a problem but, once again, you’re arguing a completely
different topic.

> _I am not sure why you feel that way._

Frankly put: because I’m not biased. You feel like you’ve had an injustice so
you’re lashing out at everyone. I totally get that. I’ve been really
privileged to grow up in the U.K. but I’ve also spent a lot of time in
developing countries so I’ve seen both sides of the argument.

> _Specifically, what I am saying is that I don 't like people starting off
> with statements such as "you then run the risk of excluding developing
> economies" as if that is automatically a bad thing for the developing
> economies._

That literally wasn’t what was said though. The context was never about what
was good or bad for those counties; economically, socially, not by any other
metric. The context of my original post and the entire chain of conversation
that preceded it was a theoretic one about how Facebook might counter abuse on
its platform. The context was always about Facebook specifically. This is why
I have repeatedly said you’ve gone waaaaaay off topic.

If you want to change the conversation to the evils of western corporations
then please bare in mind that HN is ostensibly a tech and business forum so
people will often talk about things from the perspective of tech giants even
if they don’t actually support the operations of those corporations. Thus it
would benefit everyone if you didn’t jump to conclusions, making personal
accusations about the opinions of HN posters on entirely different subject
matters just because they happen to make an impartial point on a company you
have a personal vendetta against.

------
jyu
The cynic in me thinks Facebook let this slide to hide the anemic "growth" of
their userbase and activity. Bot accounts will still click on ads, like pages,
and fake interactions that generate revenue for Facebook.

~~~
CM30
Aka the usual problem these social media services have; bad behaviour from
bots and users likely looks great on the stats spreadsheet, even if it's at
the expense of the quality of the site/community.

------
nullbull
Why take them down? That's someone's free speech right? This is what we're
told about false information and political ads with outright lies in them,
aren't we? Why would this be any different?

Rather predictable that the "line" for taking something down suddenly appears
when you're lying to Facebook. Lie to other users and it's fine... as long as
you are a real commodity, ahem, I mean "person."

Cease to lie in a way that allows Facebook to make money (not much point
serving ads to bots, is there?), and all of the sudden you're taken down.
Funny that.

~~~
sbarre
Facebook is a private enterprise. Free speech doesn't apply. Compliance with
government/social pressure is what applies (for better or worse).

That's the thing that so many people don't get.. Facebook is not a public
space. You have no inherent rights there, and they can do whatever they want
with your acccount/content/etc (or at least whatever their TOS says)..

~~~
colejohnson66
No. The _First Amendment_ doesn’t apply. The concept of free speech can be
applied anywhere.

~~~
sbarre
That's extremely pedantic and doesn't really add to the discussion.

My point was that these private companies police their content based on the
impact it has on their bottom line and any laws they have to respect, and
that's it.

They don't do it to protect any kind of speech, groups or individuals, and no
one has any _rights_ (related to speech - yes AODA and other such legal rights
apply), in the civic sense, when using Facebook or other social media
platforms.

------
joeblubaugh
“Discovers”

People have been talking about this group for months and Facebook hemmed and
hawed and delayed and delayed

~~~
Barrin92
it's actually hard to overstate how significant the effort of the epoch times
is. they gathered _billions_ of views on facebook, more than any other news
organisation.

 _" In April, at the height of its ad spending, videos from the Epoch Media
Group, which includes The Epoch Times and digital video outlet New Tang
Dynasty, or NTD, combined for around 3 billion views on Facebook, YouTube and
Twitter, ranking 11th among all video creators across platforms and outranking
every other traditional news publisher, according to data from the social
media analytics company Tubular."_

[https://www.nbcnews.com/tech/tech-news/trump-qanon-
impending...](https://www.nbcnews.com/tech/tech-news/trump-qanon-impending-
judgment-day-behind-facebook-fueled-rise-epoch-n1044121)

~~~
mtnGoat
but were those views REAL?

------
neonate
[http://archive.md/abR53](http://archive.md/abR53)

------
avivo
FYI, I actually just published a guide on how you can help ensure that
("deepfake") tools that you make are less likely to end up misused like this:
[https://www.technologyreview.com/s/614912/ethical-
deepfake-t](https://www.technologyreview.com/s/614912/ethical-deepfake-t) ...

(Also at /newest; 3rd page now, and the other submission. Ideally those topics
at least would be merged?)

Summary of mitigations:

# Limiting users or use cases

\- _User Vetting_ : Carefully vet those who can use a tool (Example: Synthesia
only working with vetted enterprise clients.)

\- _Usage Constraints_ : Limit the synthesis possibilities. (Example: Humen is
providing only a limited set of movements for generated videos.)

# Discouraging malicious use of flexible and public tools

\- _Clear disclosure_ : Ensuring and requiring that manipulated outputs are
clearly labeled (including via metadata).

\- _Consent protection_ : Requiring the consent of those being impersonated.

\- _Detection friendliness_ : Supporting those working on detection and not
seeking to fool detection systems.

\- _Hidden watermarks_ : Embedding data through robust watermarks.

\- _Usage logs_ : Storing information about usage and media outputs.

\- _Use restrictions_ : Enforcing contracts that prohibit forms of violative
or malicious use.

Not all apply to every system of course. The article has more detail!

~~~
eivarv
Good article, but for the sake of argument: Don't you think any technical
"solutions" to this issue will also eliminate the possibility to use these
tools for ethical goals (let's say the roles were reversed or whatever) – in
addition to all the other gray-area, and inevitable arms-race of
detection/avoidance?

~~~
avivo
Definitely.

I mean I think e.g. you could have incredible art projects and experiences
created out e.g. of giant undetectable fake crowds. There are real tradeoffs
in terms of what we value or need in a society.

~~~
Lammy
I think technical limitations like this are the worst possible solution. The
genie can't be put back in the bottle with any particular technology, and the
lack of widespread access and awareness means the damage from any particular
deepfake that _does_ get released can be that much higher. There is zero
chance this tech won't be in the hands of every government-sponsored group on
the planet, not to mention all the other independent bad guys out there
online.

------
noxer
I just leave this here
[https://www.thispersondoesnotexist.com/](https://www.thispersondoesnotexist.com/)
Probably the source of these images.

~~~
dwd
This other article linked here, discusses it:

[https://news.ycombinator.com/item?id=21847583](https://news.ycombinator.com/item?id=21847583)

Interesting that the eyes on every TPDNE image have a specific position that
could (for now) be used to detect them.

------
narrator
I have said for a while now that the future is their AI vs our AI and nobody
can turn the AI off because then we'd lose.

------
tschellenbach
For many use cases it would be great to have a real/verified oauth service.
Dutch government does something like that with digid. Imagine how much fraud
could be prevented if there was something like that in the US.

~~~
ceejayoz
Estonia's long been ahead of the game here.

[https://en.wikipedia.org/wiki/Estonian_identity_card#Cryptog...](https://en.wikipedia.org/wiki/Estonian_identity_card#Cryptographic_use)

------
cloudking
Why doesn't Facebook have an identity verification process? They literally
allow anyone to sign up as any person. There is a show on MTV called Catfish
that is dedicated to people who create fake profiles to trick people into
relationships.

~~~
brisance
Because they're incentivized to show continuously growing subscriber numbers?

------
qznc
Recently I played around with the newest iPhone filters. The pictures had
weird shadows and distorted backgrounds.

Are distorted backgrounds really an indicator for fake profiles? Smartphones
do all kind of weird stuff to "real" images.

~~~
kortex
GANs tend to leave a very distinctive kind of artifact which is hard to
describe but I think anyone who goes through a lot of GAN imagery knows what I
am talking about. It's a particular kind of distortion which is a liquidy
psychedelic weirdness, that you don't really see in other media fakery.

If you are interested, media forensics is an entire field about this sort of
thing.

------
tempodox
This gives me flights of fancy. Imagine (something like) FB (re-)built as a
fake AI-generated company (with fake AI-generated users). It would feel like a
circle being closed, something reaching its ultimate consequence.

------
ddmma
People are faking their profile pictures for sometime using filters or AI
cameras, bamby eyes. Already a digital ego fairytale now deepfakers would open
new positions as ’ghost busters’

------
yellow_lead
You better believe black hat SEOs are using AI for spam like this.

------
aaron695
> Facebook Discovers Fake AI-Generated Profiles

I think the profiles are human generated. I see no evidence otherwise.

If they have a GAN why did they repeat the photo?

I'd think they got the photo from some open GAN project.

[edit] Researchers actual say they might be from thispersondoesnotexist.com
[3.14] But I guess that spoils a good HN story.

[3.14]
[https://graphika.com/uploads/Graphika%20Report%20-%20Operati...](https://graphika.com/uploads/Graphika%20Report%20-%20OperationFFS_Fake_Face_Storm.pdf)

------
SubiculumCode
It makes me reconsider the early RealID efforts in social media. Perhaps
perhaps.

------
akerro
Seriously is it what nytimes is now?

[https://gfycat.com/maturefarflunghagfish](https://gfycat.com/maturefarflunghagfish)

------
fit2rule
Cool. Now it needs to work on algorithms which detect whether a journalist is
working for the CIA.

That'd be progress.

------
GarrisonPrime
Ironic, since Facebook itself uses a sort of AI to create fake shadow
profiles.

~~~
EsssM7QVMehFPAs
Source?

------
willart4food
"Discovers"

------
tomaszs
Keep calm. It is false positive. It was only Facebook customer service bots

------
chrstphrhrt
"Discovers", yeah sure.

------
username4567
I look forward to this block of fake accounts registering to vote and voting
by mail. Will any accounts go rogue and defy their creators? How does a
candidate appeal to this influential group?

------
snowedin
There's been fake generated profiles on Facebook for over a decade. I know
this because I had some, I was targeted by some, and I had "security friends"
who generated some. Over the years I've seen several campaigns of groups of
fake profiles trying to collect access (by "friending into" friend circles).

At university my security club had a member who gave a presentation on
algorithms for detecting and generating fake profile posts for Facebook,
through which he created sock puppets and also found a ton of others belonging
to "who knows which group". That was around 10 years ago.

------
ansmithz42
A fascinating thought on this for me is the basic American belief that "bigger
is better". What I mean here is that the companies generating these fake ids
will generate lots of the fake ids, under the belief that the more they have
the more influence they have. At some point, the fake ids will reach a tipping
point and outnumber the real ids thus making Facebook itself irrelevant. Sort
of a "DOS" using fake ids, in this case a denial of relevance. There are some
interesting paths that this could take, but I don't see any incentive in
Facebook really doing much about this because their power actually lies in
also "Bigger is better". So, generation of fake ids could be a blessing by
overloading Facebook and making it irrelevant.

------
scarejunba
The most fascinating thing to watch is the democratization of propaganda. Now
you no longer need the printing press. You can speak with a million voices
without being a Getty or Rockefeller.

The NYT no longer has a monopoly on convincing lies.

These are interesting times. I think we're close to the emergence of
trustworthy celebrity journalists.

~~~
Gibbon1
Scuttlebutt is Randolph Hearst bragged about getting the US into the Spanish
American War.

------
hooande
This is just using GANs to generate fake profiles. It does scale better than
making up fake people by hand, but to what end? I'm not sure if there's a
direct relationship between the ease of creating fake accounts and their
cultural influence. And it seems like facebook doesn't have trouble detecting
them.

Convince me that this isn't just "add AI to the headline" clickbait

