Hacker News new | past | comments | ask | show | jobs | submit login
An Update on the Security Issue (fb.com)
111 points by minimaxir 5 months ago | hide | past | web | favorite | 85 comments

What I've taken away from the last few years of watching large companies deal with bad actors is that cyberwarfare is here. The public is yet unaware that we technologists have become front line soldiers battling against nation state actors and brigands, and not just for their child's birthday address. I've listened to plenty of OWASP/Black Hat talks about critical pieces of infrastructure ("air gaped" power grids for example) hacked by red teams. This is scary stuff. We all need to start treating this like the hidden war it is and start planning for the worst. Cyberware is no longer the warfare of the future it is the now.

I definitely agree. I'm very worried about the potential for a World War 3 scenario where not only has the warfare technology in combat, the weaponry and ammunition, the missiles and bombs and such becoming more precision and deadly components of war but beyond that the parties involved have the added digital technology, imaging technology, unmanned aircraft vehicle technology, and all the rest of the warfare-related tech that's cropped up over the last decade plus added to their arsenal.

In an all out war scenario we don't really know the extent this new technology will ultimately play. Is it going to be the type of scenario like the first two World Wars where rifling, tank, and air combat technology is going to be one of the deciding factors? Is it going to be something more akin to World War 2 where Alan Turing's work on deciphering of the Enigma Machines / Nazi's communication codes is going to play an integral role in preparation, troop movement decisions, and gaining the edge on the enemy?

I get the feeling, personally, that it's going to be a combination of everything above and more. Those individual military's capability of incorporating all of the tech at their fingertips and applying the knowledge / inferences gleaned while also making the best use of their cyber, hard, and soft military tactics is going to be a major deciding factor since the war will probably be staged on many fronts (physical and digital) between many allied nations versus many other allied nations.

The ability to communicate and execute will be as important as it always was (e.g. maximum importance, as usual) and I strongly feel that history will reveal the more technologically advanced and more wisely dependent nations of those technologies will write history as they become the winning nation(s) / powers.


Thank ya! I edited my comment as well to be more involved in the conversation, since my comment asking you about the spelling is a moot point.

In the US the general public, engineers who discover and are tasked with preventing these attacks, and to a large extent the companies that are attacked are faced with a perverse set of incentives at the moment. Notifying federal authorities is often the best/only recourse. This results in a long waiting period where they determine the geo political or national interest in these attacks and they may make recommendations on how to best navigate and mitigate the impact but there is 0 incentive to make the fact that an attack occurred public knowledge.

How would you propose we shine a light on what bad actors, companies and individuals who are attacked, and even our own federal law enforcement would rather keep secret? How do we give law enforcement the ability to trace these attacks to their proper source during an ongoing investigation, but keep them accountable?

With GDPR, you actually MUST inform government institutions within 48 hours when personal information is leaked.

The penalty are potentially very hefty fines. (up to 2% of annual revenue, IIRC)

It is oddly silent on what government institutions MUST do within 48 hours...

> For 15 million people, attackers accessed two sets of information – name and contact details (phone number, email, or both, depending on what people had on their profiles). For 14 million people, the attackers accessed the same two sets of information, as well as other details people had on their profiles. This included username, gender, locale/language, relationship status, religion, hometown, self-reported current city, birthdate, device types used to access Facebook, education, work, the last 10 places they checked into or were tagged in, website, people or Pages they follow, and the 15 most recent searches.

Another reason to not use your phone number as a unique identifier.

It's also a good idea to just not use Facebook, or at least scrub all profile details and any personal information including pictures.

On a more serious note, the unusual activity started Sept 14, took then 11 days before actually determining that it was an exploit, and took then 2 days after that to actually fix the issue. 2 days might not seem like a long time, but with automation, the hackers could have gone through a lot more accounts in that time frame.

Birthday is pretty bad. They treat these as if they're all the same when they're not.

> Within two days, we closed the vulnerability

That is very slow for a server-side vulnerability. Was the fix that complicated that it could not be safely deployed within minutes or hours? Or did the FB management not take the issue seriously?

I lead Security and Privacy Engineering at Facebook and wanted to chime in on this thread.

As soon as we found the vulnerability, our first priority was to determine how we could protect people most effectively. If we had closed the vulnerability immediately, the attackers could have escalated their attack to modify information or post as someone else by using the access tokens they had already acquired before we reset them. Instead, we determined all the potentially affected accounts and reset their access tokens in a coordinated way to prevent further misuse of the vulnerability.

We haven't reset 90M accounts all at once before. Identifying the attackers, defining our remediation, validating that it would work, and closing the vulnerability while simultaneously ensuring all accounts were secure is complex to do at scale.

>our first priority was to determine how we could protect people most effectively.

You could have taken the site down until it was fixed.

So protecting people was your second priority.


Everything should be in lock down mode while accessing vulnerability and working on a fix.

Yes exactly. A rational response to this is to delete your FB account.

I've been involved in incident response where that was actually the decision made, so this is a real choice (extremely costly, obviously).

When you respond like this, you discourage comments like the parent from being posted to HN.

It would be fine if your point was reasonable. But it's not. It's little more than an excuse to hate on Facebook.

He's just tearing down the PR speak.

Thanks for responding here with your thinking around the issue - I know there's always strong pressure to not engage or that any talking to the public will just backfire/make things worse, but I appreciate the insight into how you're thinking about these things and the complexity around the tradeoffs.

That sounds like a difficult thing to do in under a year. Why push the issue?

If you're for real I wonder whether your posting here obeys the company external communication policy :)

Look, I know you are going to get hammered here, but on the off chance you'll see this, can I make a recommendation? You all need to read up on war; because, let's face it, y'all are in one. Here are good starting texts for the cyberwar:

https://www.amazon.com/Art-War-Sun-Tzu/dp/1599869772 (the parts on the type of ground are very good and should be instructive to y'all)

https://www.artofmanliness.com/articles/ooda-loop/ (It seems you guys have been 'looped' pretty badly, this will start the process of fixing that)

https://www.amazon.com/War-Carl-von-Clausewitz/dp/1469947021 (the sections on 'fog of war' should help you all out, the text in general is informative)

Note, that last link is to a crappy "unofficial" copy of the book that looks like it was printed from Notepad; every page has a header of "On War.txt". Do not buy it, it's practically unreadable.

Thanks for the catch!

Under 2 days is good timing to code, test, and globally deploy a bugfix. The only way to close a hole more quickly is either shut down the system (which might be worse than the bug) or if the bug was extremely recently introduced and you can rollback to a recently tested good version of the system.

> > Within two days, we closed the vulnerability

"Within two days, we closed the vulnerability, stopped the attack, and secured people’s accounts by resetting the access tokens for people who were potentially exposed. As a precaution, we also turned off “View As.”

The last sentence could mean they disabled the feature and then spent two days fixing the vulnerability before re-enabling it. It's not clear though, maybe this is just wishful thinking on my part.

Doesn't the FB deploy process take a day in itself?

Edit: This hasn't been the case for years.

Or were they asked by FBI to keep the thing going’s so they could catch the attacker?

Lots of reasons to delay.

that's not how things work.

The amount of information leaked in this is so statistically significant that Facebook should be shutdown while they do an investigation. I currently get approx. 3 robocalls a day even though I'm in the Do Not Call Registry, and they are always from different numbers. They even know my name and address even though I don't publish it.

Having your information leaked like this has real consequences. After the issue with Cambridge Analytica, you are simply becoming a lab rat peasant for the oligarchy. Any autonomy that you may have had will soon fade away.

Oh god, maybe this is why I've started getting a ridiculous number of spam phone calls every day over the past month. Shame on me for giving them my phone number for account security/verification way back ... turns out it wasn't worth it.

I'm not on Facebook at all, and my spam call rate has shot up too. I think it's just an increase in spam calls, not a result of Facebook hackery. Phone numbers are dense enough that if you know area codes and exchanges (readily available information), you can just dial through the remaining 10000 numbers pretty easily and get enough hits to be worth it, plus you only have to do that once per phone # to have a good idea of whether that's a hit.

For me, it's mostly been about student loans, which I have not had for several years now. It's not even targeted. It's just spam. Though anecdotally I've heard other people say they tend to get different calls, none of which seem particularly well targeted, so perhaps it's merely very very poorly targeted.

> Phone numbers are dense enough ...

These guys know more than phone numbers.

They know names, spouses, children, employers and banking relationships.

Armed with this info they sound legit to even experienced and cynical infosec professionals

I'm not on Facebook at all, and my spam call rate has shot up too. I think it's just an increase in spam calls, not a result of Facebook hackery.

If anyone you know is on FB the chances are FB slurped up all your details from their address book.

Part of my point is that you don't need to "hack Facebook" to get basically every phone number. If you intend to indiscriminately spam everybody, you can just do that, you don't first have to "hack" anything. It's part of why robocalls are such a problem. The tech to do it is decades old; VOIP and later innovation certainly make it easier, but it's been possible and not that expensive for decades.

You only have to hack Facebook to improve your targeting or something, but as I said, I've seen at best no evidence, and really negative evidence, that the recent spate of calls is well-targeted.

Did you get a security notice from Facebook regarding this issue?

I got a security notice and I've gotten one new phone call every day from the same number. Blocking it the other day seemed to have worked.

Nope. Would it be an email?

Click https://www.facebook.com/help/securitynotice and scroll down

I see

>Is my Facebook account impacted by this security issue?

>Based on what we've learned so far, your Facebook account has not been impacted by this security incident. If we find more Facebook accounts were impacted, we will reset their access tokens and notify those accounts.

I also get about ten spam calls a day.

Not aware if there was an email, but I think there was a banner on the website that was shown to those potentially affected. If you didn't receive one then I presume you shouldn't have been affected.

Why? There's no precedent. Equifax got away with far worse, and literally nothing came of it. Fair's fair.

I mean morally, yea, I'm with you, but from a legal standpoint, I'm not.

I'm in the same boat. About 3 robocalls each day. I have resorted to keeping my phone on a custom 'do not disturb' profile so that the phone only vibrates if someone in my contact list calls me.

I've been using nomorobo (app for $2 a month) that uses a massive blacklist to block robocalls. It works really well and you can share whatever calls sneak through to them to add to their blacklist.

It's better than the do not disturb option because it doesn't also block text notifications.

Really this would all be massively simpler if iOS allowed your contacts to act as a whitelist and just push all other callers straight to voicemail.

Oh, I should have mentioned that texts from my contacts also vibrate my phone. All together: permitted calls, texts, and hangouts messages vibrate my phone, but otherwise the phone is silent. I don't use push notifications for anything else, not even email.

Really a good system for fighting robocalls and distraction.

Also using an app (Hiya)

It's been a godsend.

I looked at Hiya, but ended up picking nomorobo because Hiya uploads your contact information to the cloud (rather than comparing locally for neighborhood spoofing). Hiya was also 'free' and it was unclear how they're monetizing your data.

Can you provide a source? According to https://hiya.com/privacy, contacts "don't leave your phone".


"However, Hiya cannot send spam calls to voicemail without having your entire contact list on its servers. The app claims it needs contact access to avoid accidentally blocking people. As Nomorobo can do this without your contacts, this feels…sneaky."

Maybe this is incorrect or changed? It's possible they're just doing the comparison for neighborhood spoofing locally too.

> Also, can I point out the irony of ironies that the attackers used the "View As" feature, which is ostensibly a PRIVACY check tool, to gain access to all this private information?


This is crazy. It shows how the social graph is so brittle.

My girlfriend's account was one of the affected and since then she keeps getting these random messages from profiles who look like ex military. No idea how they even find her profile, because she actually disabled search on FB for her name/profile.

For facebook, this is just yet another security incident, but for millions of ordinary people whose contact details are exposed and many of who are women, a continuous threat.

> We saw an unusual spike of activity that began on September 14, 2018, and we started an investigation. On September 25, we determined this was actually an attack and identified the vulnerability.

I'm unclear why it took 11-12 days to uncover this?

Possible scenario:

Someone noticed a spike on Sep 14th which was a Friday. Person A probably sent an email to Person B asking for information. Person B finally replied to ask person C on Monday. Person C gets looped in Tuesday and assigns Person D to take a look at it. Person D is finishing up a task and starts looking on Friday. Everyone thinks this is fine as the issue is probably just a front-end bug that's causing looped requests. It's not considered a security issue yet.

They make a little progress on Friday, but are out of town in Tahoe for the weekend. Monday the 24th rolls around and they finally figure out that it might be an attack and report back up the chain, a process which took 24 hours. Now it's the 25th, which is 11 days later.

possible scenario 2:

1-Mark opens calculator

2-gets how much $$$$$$$$ he’ll lose if fb shutdowns for couple hours

3-lets hack continue


Joke aside, i’d estimate they have about in the ballpark of over 200 security engineers that have alerts for when smth like unusual activity happens

First, it's saying that the spike of activity started on 14th rather than that that's when it was detected or first acted upon. September 14th was a Friday. Due to the weekend, it's unlikely anyone would have noticed before the 17th unless the anomaly was actually triggering production critical warnings. The pratical timeline was one week.

Second, if the counters that were being triggered by the attack weren't obviously malicious, it's likely that for some amount of time it was treated as a lower priority figure out what's going on deal rather than a sign of something being critically wrong.

"Hey, we're getting 25% increase in access to the video uploader but the number of successful uploads is normal. Did we break the page somehow and people are having to reload the page?". And then somebody spends a day looking at what was pushed the previous Thursday before deciding that the widget is working correctly and escalating to someone else.

It's also highly likely that the product teams at Facebook (who would probably have started the investigation, since it was product-related counters going up rather than something obviously security-related) might not have sufficient access to logs to determine the exact sequence of events. Maybe they didn't have the raw IP addresses of the requests, so couldn't realize that all this increase in traffic was coming from a handful of systems. Maybe they could only see the requests to their own service, but not the full path of requests to other services that the attacker used to pivot from one vulnerability to another.

A week after detection to deciding it was a security issue seems like a pretty plausible timeline to me.

I happily deleted my Facebook yesterday. It's scheduled to be deleted permanently in 30 days. I can no longer criticize a company while using their product.

Anyone have any ideas on how someone outside the EU can get facebook to follow the right to be forgotten thing? I don't believe deleting my account will delete any of my data, although i've already deactivated and plan to delete permanently soon.

I think I read recently that deletion does now delete your data post GDPR.

Serious question... how does any smaller, less tech-focused company have a chance of keeping data secure if not even Facebook can do it?

You just do the best you can. Totally secure is a myth. As a smaller company, however, you have an advantage that your footprint is much smaller and easier to reason about. The larger you become, the more software you have to run, the more you have to manage, the more lucrative an attack on your systems will be for the attackers.

That's why in security we don't play the blame game when someone gets hacked. We know it could happen to any of us, might already have happened, and we're all doing the best we can with the resources we have.

> The larger you become, the more software you have to run, the more you have to manage, the more lucrative an attack on your systems will be for the attackers.

Also the more people you have who might be compromised. It's almost a certainty that there are people with access inside Facebook who are or have been corrupted. That can happen in a small company too, but the risk/reward is usually lower and there are many fewer people who can be targeted.

One person with a thumb drive can exfiltrate a highly damaging amount of data.

It's not even just corruption, there's plenty of phishing attempts targeted at employees and it only takes one to potentially gain access to a lot of data depending on how systems are set up.

To answer a different question of “why don’t we see more hacks of smaller orgs?”

- Facebook is an incredibly juicy valuable target, worth investing time in

- FB has a very complex application and massive codebase. Your Django web store is far more straightforward

- smaller orgs can get away with not reporting, and don’t get visibility when they are compromised

One underrated option is not to keep extensive personalized records on your entire user base in the first place.

Don’t store personal data for any longer than absolutely necessary (laws/fraud often require some records)

If you must, then encrypt it so that you can never access it (eg you never have a users private keys)

If you must be able to decrypt data (eg fraud/law what have you), encrypt the data to a key where the private key is separated from the rest of your system.

Finally: have billions of dollars and still lose it :-/

they don’t. your startup against 5,000 chinese. who’s gonna win

Well let's not jump to conclusions so quickly. Reading Facebook's report it states that their vulnerability was within a feature they developed. A smaller company is likely to have less complexity in their software. Less complexity == less chance of vulnerability typically.

You can't be perfect. And you likely have less talented engineers than Facebook can afford, but if your software doesn't provide these sort of bridges that Facebook developed then that's one less point of exposure you're testing/fixing/securing.

the u. s. government has hacked CA providers so even companies following best practices are vulnerable. how can any company possibly compete against nation state attack?

Got an account? Check if you are affected here: https://www.facebook.com/help/securitynotice

Technical question: Does anyone know if there are any common legitimate reasons to allow an authentication system to automatically generate access tokens for a client in response to something other than an actual user login?

The thought of it sounds horrifying to me so I'm wondering if I'm missing something and how/why they might have approved such a design.

Not common at all.

I was an eng at FB when the "View as" feature was being built. I wasn't involved in the project at all, and my memory is hazy (this was probably 7 years ago by now), but I remember people being like "wow this is crazy and probably a bad idea".

I don't actually know if View As generating access tokens was how it worked originally, or if that was a later refactor. Just, in general, spoofing privacy / access control is really scary, no matter how you do it.

However, at the time, Facebook was getting a lot of shit (rightfully) for not helping users understand the privacy model, and the View As feature was actually incredibly useful for helping users understand what was public. So my guess (pure speculation) is that yes, people acknowledged it was scary, but decided the value was worth the cost.

When you want to gather people's personal and private information and pretend it was a breach.

Conspiratorial but I like it.

When are we going to wake up and understand Western companies need state help to deal with state actors? This isn't script kiddie stuff, it's other countries with billions of dollars in backing, built to tear apart our infrastructure, steal IP and gain a competitive global advantage.

The idea that FB or any Western company can stand against tens of thousands of Chinese engineers with almost unlimited budget is absurd. As much as the government and government interference is maligned in tech, we need to start considering a serious public/private cybersecurity partnership.

I am of the opinion that our ruling class is simply too old to comprehend the importance of this issue.

This is ageism, but I can't think my way around it. The questions the Legislative body were asking the tech companies were so bad, and that's presumably after they'd been briefed by whatever experts worked for them.

Executive branch: "As far as the cyber, I agree to parts of what Secretary Clinton said."

FCC: "The thousands of dead people responding to our public request for comment is fake news. Also, it was a hack. But we weren't hacked."

Justice Samuel Alito on how cell phones can have huge storage capacity: "What if the person had on his person a compact disk?"

Justice Anthony Kennedy on what happens when you get a text, while sending one: "Does it say: 'Your call is important to us, and we will get back to you?'"

Western companies need state help

Imagine you are GCHQ or the NSA. You discover a 0-day. Do you a) make sure your own citizens and allies are patched ASAP or b) keep it quiet so you can use it yourself, perhaps on said citizens?

Is Facebook going to foot the bill?

On the 28th of September I got charged $900+ for Facebook ads that I never ran. I reported it as fraud to my credit card and let Facebook know. I would recommend everyone that have their PayPal integration to check their bank statement for unusual activity around that date.

I got part way through their stupid verification (I think I gave my phone number, but then they wanted photo ID). So I need to give them a scan of my passport to see if they failed to protect my other personal data...?

The post seems to suggest the number to be concerned about is 30 million users who had access tokens stolen. Seems to downplay the 400 million users who had their profiles leaked.

"They used an automated technique to move from account to account so they could steal the access tokens of those friends, and for friends of those friends, and so on, totaling about 400,000 people. In the process, however, this technique automatically loaded those accounts’ Facebook profiles, mirroring what these 400,000 people would have seen when looking at their own profiles."

Seems a lot more like this has leaked the privacy of 400 million users, but Facebook are trying to focus on the smaller 30 million (though granted, those are more seriously affected.)

400,000 not 400,000,000


The relation between Facebook, it's users and attackers is the same as between cattle ranchers, their sheep and sheep rustlers.

Most farmers I know have far more respect for their livestock than FB do for their "users".

Meh... sheep or FB users, they all get monetized in the end.

Don’t worry, Facebook. I was affect by this (even though my account has been disabled for 2 years) and realized I should leave finally. So I did. And nothing will bring me back to any service you have since you’ve only demonstrated CYA and uncaringness. If you really cared, you’d compensate me monetarily for my loss due to your negligence.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact