
Reddit Security Incident - pyreal
https://www.reddit.com/r/announcements/comments/93qnm5/we_had_a_security_incident_heres_what_you_need_to/
======
packetized
Interesting that the data accessed was very specifically only limited to:

* A complete copy of an old database backup containing very early Reddit user data -- from the site’s launch in 2005 through May 2007

* Logs containing the email digests we sent between June 3 and June 17, 2018

Also of note:

"Already having our primary access points for code and infrastructure behind
strong authentication requiring two factor authentication (2FA), we learned
that SMS-based authentication is not nearly as secure as we would hope, and
the main attack was via SMS intercept."

If this doesn't put the nail in the coffin of SMS-based 2FA, I'm not sure what
will.

~~~
BOBOTWINSTON
Can you or someone else explain/link to me the basics of why SMS-based 2FA is
so terrible? I've never really heard the sentiment before, but it appears to
be common knowledge.

~~~
kevin_b_er
Because the telephone companies are terrible about security and often highly
disorganized internally. They are beyond stupidly susceptible to Social
Engineering and any "passcodes" against giving away access do not stand in the
face of stupid customers and the need for customer service to satisfy them.

Your number can easily be stolen or redirected to get and sometimes send SMS
from/to your number. Your cell phone account is the linchpin for a very
extensive identity theft attack.

~~~
slg
In their defense, being able to successfully identify a customer is who they
say they are is a difficult problem that is only compounded when you might
only speak to a customer as infrequently as every few years. 2F devices and
codes can be lost. Passwords and pins can be forgotten. Answers to security
questions can change. Have you ever tried to access your own account with a
company like this without this data? There are few things more frustrating
than being locked out of your account because you can't recall what you said
your favorite movie was in 2012. Throw in the low odds of actually being
targeted in a social engineering attack and companies optimize for customer
satisfaction and convenience over security.

Blaming companies for responding to that incentives isn't going to accomplish
anything. The way to fix things is to change the incentives by either
increasing the punishment for falling for social engineering or create a
system that makes it easier to remotely identify people.

------
slg
The hacker(s) took a database backup from 2007. I have never worked anywhere
that has kept a backup that long. It is possible it is some sort of final
archive before a large migration, redesign, or something like that. However if
the intent is to keep it forever it should at least be encrypted. As far as
I'm aware, the only strong reason to not enable encryption on backups is to
allow a secondary backup or mirroring system to compare the changes between
backup files rather than reprocessing the entire thing as a single new file.
That reason disappears for an archived backup.

~~~
btgeekboy
Given the weird collection of stuff they got (including the ancient database
backup) I wouldn’t be surprised if this was the contents of an admin’s home
directory.

~~~
Symbiote
+1 insightful.

While looking at GDPR compliance, I came across a guide that said "backups are
kept for as long as it will take you to notice the missing data and restore
it. Exported data kept for longer than this is an archive".

That helped me realise I really shouldn't be keeping 5-year-old database
backups for some systems; a few months is plenty sufficient time for us to
notice any corruption. As part of that clear-out, I searched for and deleted
many old mysql-backup-2012-just-in-case.tar.gz from /root and similar places.

~~~
dserodio
Do you have the link to that guide by any chance? I haven't found a guide
that's practical enough for my needs yet.

~~~
Symbiote
I think it was this, and the source it links to.

[https://community.jisc.ac.uk/blogs/regulatory-
developments/a...](https://community.jisc.ac.uk/blogs/regulatory-
developments/article/gdpr-backups-archives-and-right-erasure)

------
zokier
While everyone is piling on how SMS 2FA is oh so bad, it is worth noting that
it is supposed to be the second factor here. So what happened to the first
factor is the obvious question. Someone was using weak/compromised password or
got social engineered would be my guesses, neither which are very good
options.

~~~
SpaethCo
This was also my first thought when reading this. It almost makes me wonder if
it was really a SMS exploit at all — when someone has the user, pass, and 2FA
code, that sounds to me like the target clicked on a convincing URL and
readily supplied all the things their attacker would need.

------
JoblessWonder
This incident report glosses over the depth of what access was given to focus
on the user data that was compromised... but it sure seems like they got
pretty deep:

* A complete copy of an old database backup containing user data from launch in 2005 through May 2007 including:
    
    
      -usernames,
    
      -salted/hashed passwords,
    
      -e-mails,
    
      -all content including private messages
    

* Reddit source code

* Internal logs

* configuration files

* other employee workspace files [?]

~~~
bredren
This is a serious breach and I'd suggest "gloss over" does not characterize
Reddit's statement appropriately.

Given how the report is structured, it seems like the amount of leaked data is
purposefully being hidden behind red herring info about SMS 2FA that is not
important to users who want to know where they stand.

When this DB is leaked, there should be more than enough weak passwords to
both pwn and dox many, many reddit users. Do we know the encryption scheme
reddit used to encrypt their password database involved in the leak?

Also, how is it that Reddit gained a head of security 2.5 months ago? Who was
in charge of this prior to that date?

~~~
Deimorz
> Do we know the encryption scheme reddit used to encrypt their password
> database involved in the leak?

At the time of this backup, it would have been SHA1. Here's the relevant
hashing code:

[https://github.com/reddit-
archive/reddit/blob/4778b17e939e11...](https://github.com/reddit-
archive/reddit/blob/4778b17e939e119417cc5ec25b82c4e9a65621b2/r2/r2/models/account.py#L244-L248)

Edit: reddit's confirmed this here:
[https://www.reddit.com/r/announcements/comments/93qnm5/we_ha...](https://www.reddit.com/r/announcements/comments/93qnm5/we_had_a_security_incident_heres_what_you_need_to/e3f8og0/)

~~~
zokier
randstr has this goldnugget of a comment:

    
    
        """If reallyrandom = False, generates a random alphanumeric string
        (base-36 compatible) of length len.  If reallyrandom, add
        uppercase and punctuation (which we'll call 'base-93' for the sake
        of argument) and suitable for use as salt."""
    

[https://github.com/reddit-
archive/reddit/blob/4778b17e939e11...](https://github.com/reddit-
archive/reddit/blob/4778b17e939e119417cc5ec25b82c4e9a65621b2/r2/r2/lib/utils/utils.py#L41)

The function has specifically a flag for use as salt, but the hashing code
does not actually use it. Whoops. Of course the loss here is not really that
significant (~4bits of entropy), but I find it still bit funny.

~~~
zokier
Picking on the code bit more while we are at it, random.choice that randstr
uses is of course not backed by CSPRNG so its not ideal for salts. Although I
would be shocked if attackers would be able to exploit that in any way.

Kinda interesting is how they decided on specifically 3 characters for salt,
which seems really low. Its not like the characters cost anything, why not 30
characters instead?

------
Deimorz
The scary part of this is probably for people that had accounts on reddit in
2007 but later deleted them, or just completely forgot they existed. Reddit's
not going to be able to contact the owners of those accounts.

Did you have an account 11 years ago? Did you vote on anything embarrassing,
or send any compromising messages? How sure are you?

I don't even know the answer to those questions for myself.

~~~
techscruggs
Reddit can't contact them because Reddit didn't require an email address to
create an account back then.

If they did require an email address, they could restore their database
backups to retrieve that information.

~~~
PSZD
Reddit still doesn't require an email, the signup form is just a well executed
dark pattern - you can hit next and skip providing an email.

~~~
deaddodo
I wouldn't call a form requesting your email (the only entry field on a
dedicated page) a "dark pattern". They out right imply an email is necessary
and knowing it's not requires knowledge otherwise or accidentally clicking
"next".

~~~
crtasm
I've made a couple of reddit accounts in the last 1-2 years and it was always
explicit that an email was optional. Is this all since the recent redesign?

~~~
irishsultan
The only indication that an email is not required is that the email field
doesn't contain a blue dot, as opposed to the username/password fields.
However those are on the next page, so you have no indication that it's not
required. That's actually on the new design, on old.reddit.com there is not
even that indication.

~~~
crtasm
I see. that's a shame, last time I used the register form it was all fields on
one modal popup and said email wasn't required.

------
newman8r
If the logs contained IP addresses, they could be used to correlate multiple
accounts, leading to throwaway accounts being doxxed.

It doesn't sound like IP address data was compromised, but I wouldn't be
surprised.

~~~
fpgaminer
Could probably also correlate by password. I'm sure lots of users re-use the
same password at least for their throwaways.

~~~
bredren
They certainly did 10 years ago.

------
lsllc
Alright, 2FA tokens came up the other day on HN and now we have this. Time to
make the switch.

Yubikey 4 / Feitian looks interesting, but it seems it only works in Chrome
with Gmail etc. etc.

Anyone have any thoughts on solutions that include Safari on Mac and/or iOS?
The NEO claims NFC support but I doubt that works on iOS.

~~~
teilo
2FA does not usually mean U2F hardware such as Yubikey. Reddit does not
support hardware keys.

Most of the time, 2FA means using a token generator, such as Google
Authenticator, Authy, or similar. They are just apps. This is much safer than
SMS because one would need physical access to your unlocked phone to generate
a token.

~~~
jjnoakes
TOTP has a downside when compared to SMS as well.

In cases where server security was breached and databases (or database backups
or dumps) were accessed, if the TOTP seeds were part of the database (not sure
how likely that is, but I'm guessing it's likely), then TOTP is doing nothing
for security.

TOTP protects against things like credential stuffing and weak passwords, and
is safer than SMS (no hijacking/intercepting), but for database security
breaches things aren't so cut and dry.

I wonder if there should be a TOTP-like app which you still register with a
site when you first log in or create your account, and which codes are sent to
when new logins are needed, but which uses a more secure communication channel
than SMS. This gives you the best of both worlds, no? One-time codes not
generated from a single plain text seed, communicated to a known client over a
secure channel, to prove the initial user is still in possession of the known
client?

~~~
vel0city
A standard called SQRL is similar to many of the concepts of U2F and can be
done with an app scanning a QR code.

[https://www.grc.com/sqrl/sqrl.htm](https://www.grc.com/sqrl/sqrl.htm)

~~~
jjnoakes
Looks like SQRL is aiming higher than just 2FA, they try to be the entire
login process. Thanks for the link, but not sure it's quite what I'm looking
for.

------
Canada
For what reason was a decade old backup kept online for? That is insane. If
they have hygine that poor I'm really worried about what other problems they
have.

~~~
ajross
Good grief. You're talking about a decade old data set.

Try this: Go to your own site backup for whatever you've got, be it a personal
disk backup or something you made for a customer or friend or whatnot. Now,
tell me which files might contain sensitive information to third parties. I'll
wait.

This isn't "hygine". This is "we have a 11 year old backup mounted somewhere
that we all forgot about and we honestly don't know what's in it". Yeah, it
sounds dumb, but it's not reasonably avoidable by internet pontification
regarding "best practices" unless your "best practices" involve eidetic
memories or time machines.

~~~
Canada
It's simple. If it's a couple of years old and I haven't accessed it then it
gets archived offline. Metadata about it is kept hot so I can track what I
have. I might be a bit sloppy personally and have a _limited_ amount of 4 year
old stuff still internet connected but certainly not anything approaching 10
years and it's sure as hell not large archives of other people's data I have a
duty to protect.

There is just no excuse for that, it serves no business purpose, ancient
backups that have no recovery value should not be online if they are kept at
all. This incident shows an appalling lack of care by reddit technical
leadership. Obviously they are not systematically tracking and reviewing the
data they keep. Given this incident I would not be the least bit surprised if
they have copies of this and that all over the place with no awareness or
oversight.

~~~
ajross
> It's simple. If it's a couple of years old and I haven't accessed it then it
> gets archived offline

How is that remotely simple? This is a medium size company. They have
thousands or tens of thousands of storage devices mounted "online" in some
way. How do you purport to audit every single one of them to determine when
the "last access" time was for all the relevant data?

I suspect, as mentioned earlier, that your answer is going to involve a time
machine to go back to 2007 and make sure reddit was doing things "right".

~~~
Canada
It's one thing to get owned by 0 day and lose the stuff you were working on
last month. If you lose stuff from 10 years ago you absolutely had it coming.

The way you protect old data is by routinely auditing what you have. You make
sure each department is on top of organizing its data. If you're not sure what
it is, you offline it. It can always be brought back online if necessary. Even
lowest-common-denominator schemes like ISO 27001, a system designed to allow
management that doesn't even know how to turn on a computer to manage
information security, covers this basic idea. It would be one thing if a non-
technical department had leaked some ancient folder full of reports containing
some sensitive data, but this is a database dump of one of the most highly
trafficked sites on the net. Reasonable people should expect the custodians of
those sorts of things to know better. To anyone with technical knowledge,
minimizing your data exposure should be as natural as breathing.

And yet again this time we get the usual "the attack was so sophisticated"
refrain. Oh, the defenders were so careful, and tried to take every precaution
for sure! The attackers hacked the 2FA! If that's true why didn't the
attackers get the 2018 data? Frankly I don't believe the Reddit management.
They probably left that old database dump on some old system they forgot about
that tons of people had access to.

How many breaches to we need to remind us to be aware of what data we are
managing and take precautions? How many more is it going to take before we
collectively stop being so careless?

~~~
ajross
> If you're not sure what it is, you offline it

I've literally never worked at nor heard of an employer that tried this. You
have a case study example? You seriously think IT departments are in the
business of finding an archiving the contents of every random PC on the
network?

~~~
Canada
No, of course I don't believe IT departments do that. It's management's
responsibility to make sure someone is responsible for the data on every
random PC on the network. Management is responsible for putting systems in
place to manage risk effectively. Random PC users need to be compelled to
comply by policy and enforcement because otherwise they usually don't have the
knowledge or incentive to do what must be done.

People working in development or operations on the other hand, should
instinctively know and do what must be done with their own data. And reddit
didn't do that. Management failed and even worse the technical leadership
inside the company was directly responsible.

Remember this next time you're working at a place with a poor management of
data and a culture of indifference. Do something about it, sound the alarm
instead of sitting on your hands waiting for the inevitable leak.

~~~
ajross
So... who exactly does this again? You seem to be "solving" this problem by
turning it into a simplified academic exercise. Real security happens in real
companies with real people.

------
Dowwie
If you are using SMS based 2FA, understand the risk:. "Already having our
primary access points for code and infrastructure behind strong authentication
requiring two factor authentication (2FA), we learned that SMS-based
authentication is not nearly as secure as we would hope, and the main attack
was via SMS intercept. We point this out to encourage everyone here to move to
token-based 2FA."

------
NVRM
In comment from the admin: « In other news, we hired our very first Head of
Security, and he started 2.5 months ago. » No comment.

«Old salted and hashed passwords» This sentence mean: All hashed were
readable. It also mean, if they are still needed on their servers, that they
are probably still in use. It would had been easy to salt this hashes.

First fix holes, then redesign...

------
jandrese
SMS is not about securing an account. It's only use is as a proof of work
(money) to make it harder/more expensive to make a bot account.

Using it as a security measure is a mistake.

------
tluyben2
I keep telling my bank SMS 2fa is bad but they say it is not. Many banks
replaced tokens with SMS unfortunately.

~~~
adrr
I don't know any bank that supports token-based 2FA and most support both SMS
and email based 2FA. Email is a terrible 2FA method since most users re-use
passwords.

One way to help protect you is to visit your carrier's retail store and have
them turn off online access to your account and require all changes to your
account to be done in person with a valid government ID. This should make it
more difficult for number porting attacks but they can still sniff the SMS
message when goes over the cell network. As far as I know, mobile network
control messages aren't protected.

~~~
Symbiote
Several British banks use Chip+PIN cards to provide a token — not necessarily
for login, but for authorizing a transaction.

Like this: [https://c7.alamy.com/comp/CYGATP/online-banking-security-
chi...](https://c7.alamy.com/comp/CYGATP/online-banking-security-chip-and-pin-
card-reader-for-authorising-transactions-CYGATP.jpg)

------
samstave
I got the alert to change my PW. I had had the same PW for 12 years!

Edit: 12 years, not 13.

\-------------------------------

 __ _Account credentials from 2007 compromised_ __

 __ _from reddit_ __

 __ _[A] sent 35 minutes ago_ __

 __ _Hi,_ __

 __ _TL;DR: As part of the security incident described here, we 've determined
that your account credentials may have been compromised. You'll need to reset
your password to continue using Reddit. Details below._ __

 __ _On June 19, Reddit was alerted about a security incident during which an
attacker gained access to account credentials from 2007 (usernames + salted
password hashes)._ __

 __ _We 're messaging you because your Reddit account credentials were among
the data that was accessed._ __

 __ _If there 's a chance the credentials relate to your current password,
we'll prompt you to reset the password on your Reddit account. Also, think
about whether you still use the password you used on Reddit 11 years ago on
any other sites today. If there's a chance the credentials relate to the
password you're currently using on Reddit, we'll make you reset your Reddit
account password. You can find more information about the incident in the
announcement post linked above. If you have other questions not answered
there, feel free to contact us at contact@reddit.com._ __

~~~
shawn
Has reddit been live for 13 years? Jeez. I thought my 11 year account was old.

~~~
oldManRiver
I got on reddit in 2005, before you could comment.

Got on thefacebook as well in 2005 because the college kids in my classes
(taking for fun as an adult) told me I needed to be on there to get invited to
parties and so they could write on my wall. Good times.

~~~
samstave
With those comments and your username, you can tell people to CD .. off your
lawn.

------
hyder_m29
How would an attacker go about intercepting an SMS?

~~~
peterwwillis
By taking control of your phone number or the radio network your phone
connects to, or attacking the signaling network itself, to intercept
information going to your phone number.

Basically, imagine every conceivable way any human or computer might at any
point interact with a plaintext signaling packet designed to be passed around
the world by different companies and eventually read by people. Now attack all
of them. Something somewhere will give it up.

~~~
bdamm
It's fairly easy to claim the general case, and indeed you're right. But the
challenge is that not all attackers have infinite resources, and the ones that
effectively do us small fry really can't protect against anyway, because
they're already where they need to be.

So specific information on known attack paths is an interesting conversation,
because part of the SMS 2FA security is the belief that while 1-off SMS 2FA
attacks are possible, they generally don't scale, and so that puts a high cost
on carrying out the SMS 2FA, or informs a limit on the value that can be
protected by SMS 2FA.

So, good for reddit? Maybe yes. Good for your bank? Maybe not, but maybe yes,
depending on the diligence of the customer, the robustness of anti-fraud
measures, and the cost of fraud insurance.

~~~
oarsinsync
> So, good for reddit? Maybe yes. Good for your bank? Maybe not, but maybe
> yes, depending on the diligence of the customer

Good for Instagram? Maybe no, without much dependence on the diligence of the
customer.

[https://motherboard.vice.com/en_us/article/vbqax3/hackers-
si...](https://motherboard.vice.com/en_us/article/vbqax3/hackers-sim-swapping-
steal-phone-numbers-instagram-bitcoin)

~~~
bdamm
Alrighty then. Thanks for the enlightening read.

------
erikb
You mean they tell us 1.5 months after the event that our emails and passwords
might be compromised?

------
ojosilva
Would someone kindly explain how a SMS can be intercepted during 2FA and
how/why tokens otoh are safer?

A friend and I were brainstorming the design of a fraud prevention app/startup
just this week and we naively thought SMS would be the way to go. Yikes!

~~~
tptacek
SMS ties security to phone numbers. Phone companies can trivially move numbers
to different people, accounts, and SIMs. When you rely on SMS for security,
you are relying on the customer support staff of giant mobile phone companies
for your security.

There are other more technical weaknesses with SMS, because the phone networks
themselves are also insecure. But the big issue is phone companies themselves.

Don't use SMS 2FA.

~~~
chris_wot
Google pretty much foists it on all our employees.

~~~
criddell
For Google, it's not a text message though, is it? IIRC, it's tied to your
Google account and not to your phone number.

~~~
chris_wot
They enforce it via SMS.

------
danbtl
How does SMS interception actually work in practice? Wouldn't this require
physical access to the phone/SIM, or are there any known remote exploits?

~~~
tptacek
How about "teenager calls phone company, gets number reassigned"? That's the
level of assurance we're dealing with in SMS.

------
barking
So what's to stop a hijacker persuading the website to take off 2FA or switch
you from TOTP to SMS.

Seems just as possible as hijacking your phone.

------
vxxzy
SMS Interception is what got them. Moving to offline 2FA needs to happen. SMS
Interception is on the rise.

~~~
Silhouette
And yet bizarrely, the number of organisations with serious security
requirements that are adopting SMS messages or other methods dependent on
phone numbers as 2FA just recently is quite noticeable.

Stripe use it for logging into your business's account.

HMRC (the UK government tax office) also uses it for logging in.

Various banks and financial services I use in a personal capacity rely on
secondary phone authentication to set up things like new recipients for paying
bills online.

------
vram22
Was the notification to Reddit users about the incident, sent from
noreply@redditnewsletters.com ?

------
empath75
>In other news, we hired our very first Head of Security

wow...

~~~
sdinsn
> we hired our very first Head of Security, and he started 2.5 months ago

He started before the hack happened

~~~
jandrese
I'm pretty sure he didn't fix every security problem on his first day of work.
Once a company gets big enough every change ends up being an ordeal of
organizing all stakeholders and getting them to agree and giving them time to
update their own systems so they won't be broken when the change happens,
etc...

------
pandasun
Edit: nevermind

~~~
always_good
They didn't say write-only access.

They said they only got R access instead of RW access.

------
hindsightbias
>In other news, we hired our very first Head of Security, and he started 2.5
months ago.

Uh huh.

------
Alex3917
SMS hijacking? Really?

How is it that Reddit’s security team is continually learning security lessons
that have been common knowledge among non-technical people for 5+ years? They
seem to treat their production systems more carelessly than the average person
treats their Nintendo switch account.

~~~
gmjosack
> been common knowledge among non-technical people for 5+ years

Who are these non-technical people that you know that are not only using MFA
but also know that SMS is insecure for MFA?

Rather than putting them down I'm happy they're willing to share and bring
knowledge, that some communities already know, to even more people.

~~~
Alex3917
> Who are these non-technical people that you know that are not only using MFA
> but also know that SMS is insecure for MFA?

Anyone who reads pretty much any mainstream newspaper? At this point it would
be easier to name mainstream media publications that _haven’t_ covered this
issue extensively. E.g. just google:

site:nytimes.com sms hijacking

site:wsj.com sms hijacking

site:latimes.com sms hijacking

Not to mention the fact that it’s been discussed on Reddit itself hundreds of
times. And on the front page of HN dozens of times as well. E.g.:

[https://news.ycombinator.com/item?id=14480191](https://news.ycombinator.com/item?id=14480191)

