
Dropbox Cofounder & CTO Arash Ferdowsi responds to yesterday's bug - chaz
http://blog.dropbox.com/?p=821
======
rdl
I think Dropbox would benefit from bringing in a competent PR person to advise
them on making this kind of announcement and communicating with customers.
(which is saying a lot, since I generally disdain PR people). Commenting on a
problem you've caused people without actually apologizing is kind of a dick
move. Even the US military does a better job when it accidentally kills
people!

The statements should come from the top, but should probably only be written
by founders if they're actually good at this kind of thing.

There are a lot of companies who are relatively good at crisis communications
with affected parties. Network carriers are often good (at least when talking
to other networking professionals), although they often try to restrict the
spread of information via NDA.

There are some reasons to limit disclosure immediately after an incident (if
you think the vulnerability still exists and might put you at risk, especially
for a security threat and not a regular outage). You absolutely want to
include "why this won't happen again" in your message (which Arash kind of did
this time), but you also want to accept blame at an emotional level -- you can
do this in ways which make you look perfectionist and hyper-professional, vs.
weak.

I don't actually care that much about Dropbox security myself, but if anyone
in the cloud computing space fucks up badly enough and frequently enough to
make users distrustful of cloud resources, it makes life harder for everyone
else in the space. That is a problem for me, and for other startup founders.

~~~
gjm11
> Even the US military does a better job when it accidentally kills people!

What does the US military do when it perpetrates a boneheaded security screwup
that might or might not have compromised a small amount of data belonging to a
few people?

(No question, Dropbox's response to this screwup is totally inadequate, but
comparing it to how another organization responds when it has _killed people_
is a bit silly.)

~~~
rdl
I kind of view it the other way.

Shooting someone for speeding toward your checkpoint who turns out to have
actually been deaf or in a hurry to rush a sick kid to the hospital is sad,
really bad for everyone, should be avoided if at all possible, etc. There are
also a lot of liability and cultural concerns. If you're willing/able to make
a real apology for that, it should be really easy to do so for something
comparatively minor.

------
tshtf
_If you’re concerned about any activity that has occurred in your account, you
can contact us at security@dropbox.com._

I'd like the full access logs, including timestamps and IP addresses of every
time my account was accessed in this timeframe. I've written
security@dropbox.com about this, and am waiting to hear back.

~~~
phillco
In fact, they should add this permanently to the UI, like Gmail.

~~~
reaganing
They should. Could also do what Facebook (optionally) does: require you to
send them a 4-digit code via your phone to allow access to any new
unrecognized devices.

------
mef
Somewhat disconcerting that I am a Dropbox customer, yet neither the breach
nor the explanation has been communicated to me by email; I've had to find out
about it by reading HN.

~~~
tomjen3
I don't think that was a wrong decision actually -- most people who use
dropbox are like my mom, not at all computer savy.

There is no way dropbox would be able to explain to them what happened without
scaring them silly.

And if nobody logged in, there can't possibly be any danger to their account.

~~~
bxr
>There is no way dropbox would be able to explain to them what happened
without scaring them silly.

Opposed to who? People who do know what it means and _should_ be scared silly
but aren't because they've been beaten into submission by breach after breach
after breach.

~~~
tomjen3
If you account hasn't been accessed in that time, there is nothing to be
afraid of.

~~~
owenmarshall
You're making the rather large assumption that this was a one time goof on the
part of Dropbox.

IMO, this is reflective of a corporate culture that places testing and
security on the back burner. And while some people may be OK sending their
data to such a company, the rest of us might not be.

------
ctide
I don't understand why Drew doesn't either make these posts or find a PR
person to filter Arash's comments through. Arash never comes off well in these
blog posts or in his comments here surrounding these issues.

His comments always come off feeling like 'sorry bros, it's really not that
bad, and we fixed it so no problemo' unlike Drew's comments which generally
have a much more personal feel to them.

~~~
phillco
Agreed. This would be like a major bank saying "for a 4-hour window yesterday,
anyone could withdraw money from anyone's account. But don't worry, it's not a
problem, because only a few people actually did."

~~~
minikomi
Or a husband explaining to his wife about last nights escapades...

------
raganwald
To each their own, but I am far more interested in what happened, what they
are doing about it, whether it can happen again, and why I heard aboutnit on
HN instead of directly from Dropbox.

I personally am not interested in an apology or a gaveling tone. These people
are not my personal friends, I don't have an emotional investment in whether
they pretend to care about my feelings. I have an objective intrest in how
they choose to _act_ and the information they give me so that I can make my
own informed choices.

~~~
ra
I would really like to know the exact details of what happened.

Ideally I'd like to see the actual bug in code.

~~~
arethuza
"I'd like to see the actual bug in code"

That makes it sound like it was primarily the fault of the developer who
created the bug - things like this indicate an inadequate _process_ or
_culture_ not simply a mistake at the developer level.

Edit: Does Dropbox have any testers or is all testing based on automated tests
created by the development team?

~~~
bmj
Agreed. Sounds like the deployment wasn't QC'ed at all.

------
orofino
Just emailed them the following:

Dropbox Team,

For the duration of the event described in the post on your blog on Sunday I'd
like the time of login and IP address of any authenticated sessions during the
window. I'd also like to understand why a post from one of your customers
linked to via hacker news was the only notification I received as one of your
paying customers. If you know which user accounts we logged into during the
event it seems rather straight forward that you would notify those impacted.
It seems clear that had a 3rd party not brought this to light you'd have felt
it unnecessary to notify your customers.

I look forward to your prompt response.

~~~
thehigherlife
I'd love for you to post their reply if you're willing.

~~~
orofino
As Requested, I received the response about an hour after I posted this/sent
the email. Can't say the response is terribly reassuring, but I suppose if no
authentications have happened I don't have to be concerned about THIS
incident.

We're working around the clock to gather additional data. We will notify
affected users if we detect any unusual logins or activity in their account.
We are reviewing our logs that record password authentication events in
accounts. We have not been able to detect any relevant account activity for
your account during the time period in question, so we believe that your
account was unaffected by the bug.

Regards,

------
epistasis
I haven't kept anything that absolutely had to be secret on Dropbox, but I
have kept stuff that I'd slightly prefer not to be public.

I'm going to change my behavior, the only things that go on Dropbox now will
be completely public. They've completely lost my trust.

~~~
bennysaurus
Security leaks happen. The lesson here is treat anything that you put on a
machine you don't completely control as being at-risk, even if you pay for top
notch security and the vendor guarantees it. The guarantee (and any
compensation) isn't a lot of comfort if the information hits the wild. This
includes everything from web servers on Amazon to your Gmail account.

In the case of Dropbox, I would suggest PGP or Truecrypt anything sensitive
and keep the keys locally or in another location that is completely unrelated
to the box.

------
Timothee
Not that it would really change anything, but there isn't much wording around
apologies or being sorry about letting that happened.

More details would have been nice too. Allowing anybody to log into anybody's
account is _a big deal_ , even if in the end a small percentage of people were
likely affected. It's not like I couldn't access my account for a few hours or
that the sync got messed up somehow.

Also, it'd be nice to know how the bug was discovered on Dropbox's side: did
they realize it themselves or was it from nice people who found the problem?

~~~
kanak
I believe Chris Soghoian was the person who discovered the bug (there was an
HN submission about this earlier).

<http://twitter.com/#!/csoghoian>

~~~
Timothee
Yeah, that's kind of what I'm curious about: did Dropbox learn about it
through that guy's discovery? If so, we're lucky that that guy came across it
the very same day the bug was introduced. I'd assume there aren't that many
people who would have found the security hole, been nice not to abuse it and
cared enough to let Dropbox and the world know…

------
tptacek
It took almost 4 hours to discover that anyone could log in without a
password? There has to be more to it than that, right?

~~~
brown9-2
Perhaps the idea of monitoring the number of failed authentications and
placing both an alert when the rate goes over a certain threshold _and_ when
it goes under a certain threshold doesn't seem so apparent until after the
fact.

Might be a huge lesson here to learn for other services to monitor their
authentication system in both directions.

~~~
bryanallen22
Better yet, basic unit tests that verify passed and failed authentication of
test accounts. They should run before and after each code deploy (among other
unit tests).

It's very disconcerting that they don't seem to be doing this.

~~~
brown9-2
I don't like to be pedantic but code that tests multiple components and how
they interact together aren't really "unit tests" anymore since they are
testing more than one unit - it's pretty reasonable to assume that an
authentication system is comprised of many modules / classes / etc.

~~~
bryanallen22
Good point.

It still shouldn't even be possible to deploy without some basic sanity tests.
I'd put authentication very high on that list.

------
georgemcbay
The blog post mentions that they ended all logged in sessions, but did they
also rollback any new sync settings set from the start of the incident to when
they did their cleanup? eg:

User Joe's account is logged into by attacker Tom. Tom sets his computer as
one of Joe's "My Computers". Does what they did to clean up the problem
invalidate this or does Joe have to log into his account, look at his list of
"My Computers" and remove the ones he doesn't recognize manually to stop Tom's
system from automatically syncing all his stuff until he does finally notice
(which is likely never for most users, I'd assume)?

~~~
jerrya
IIRC from the last dropbox security fiasco, they also have to revoke the magic
cookies that permit access without a password.

------
smoody
Doesn't this completely blow their "only a few employees have access to your
data" stance out of the water? Doesn't mean that, for example, every one of
their programmers has access to user data? If a presumably simple bug can give
the whole world access to your data, then it can't be that difficult for any
dropbox employee to access it (again, from the inside).

~~~
robryan
Doesn't matter how much encryption you have on data if your decrypting it for
the wrong people.

------
jerrya
Compare and contrast how Dropbox has communicated this breach with the storm
that rained down on LastPass when they forced password changes because of an
activity they couldn't explain.

This might explain why Dropbox's poor communication of this security breach is
the #winning move.

~~~
w1ntermute
As a result of what LastPass did, they won my respect. As a result of what
Dropbox did, they lost it.

~~~
palish
What did LastPass do? Sorry, I missed that story.

~~~
jerrya
Previously on Hacker News: <http://news.ycombinator.com/item?id=2526868>

LastPass Disclosure Shows Why We Can't Have Nice Things

08 May 2011

A few days ago, LastPass announced they would be forcing their users to change
their master passwords in response to what was essentially "something weird":

"We take a close look at our logs and try to explain every anomaly we see.
Tuesday morning we saw a network traffic anomaly for a few minutes from one of
our non-critical machines. These happen occasionally, and we typically
identify them as an employee or an automated script.

In this case, we couldn't find that root cause. After delving into the anomaly
we found a similar but smaller matching traffic anomaly from one of our
databases in the opposite direction (more traffic was sent from the database
compared to what was received on the server). Because we can't account for
this anomaly either, we're going to be paranoid and assume the worst: that the
data we stored in the database was somehow accessed."

LastPass acted exactly like we wish most companies would act: responsibly. And
the media's response? Declaring LastPass "hacked" and "vulnerable", and
placing them in the same category as Sony—who definitely were hacked—with
sensationalist headlines like:

WARNING: Your Web Browser's Master Password May Have Been Stolen -- Change It
Now LastPass Has Been Hacked And Asking Everyone To Change Their Master
Passwords LastPass Hacked, Change of Master Password Urgent LastPass Is Hacked
– Change Your Master Password, But Don't Panic Should the LastPass, Sony hacks
make you fear storing data in the cloud?

------
g0atbutt
There has been a lot of comments bashing Arash's response. His post certainly
wasn't personal (and should have had a more apologetic tone), but beyond that
what would you've liked to see him communicate? He seemed to do a good job
explaining what happened and outlined a plan moving forward.

I'm not trying to defend him but I am genuinely curious what other people
think his response was missing.

~~~
random42
1\. The tone MUST have been apologetic.

2\. The error is presented like _casual_ news. Sort of downplaying the
incident and not acknowledging that they screwed up. If the accounts were
freely accessible for 4 odd hours, it is pretty serious.

3\. No actual details of bug introduced were provided.

4\. The communication was done on blog, and emails were not send. Again, it is
an attempt to downplay the incident.

~~~
Angostura
But he specifically said that e-mails would be sent to accounts accessed
during the bug period.

Seems reasonable to me.

~~~
albedoa
So, to take it to one extreme, you think that if zero accounts were accessed,
then it would be reasonable for zero users to be directly notified about this?

That doesn't sound reasonable in the least bit.

~~~
kwis
Some people might also be concerned that the note about sending emails wasn't
added to the post until after there was mass outrage at the lack of
notification. Makes it feel very inauthentic, as though they were really
hoping they'd be able to sweep it under the rug.

------
dacort
Client-side encryption feature request:
<https://www.dropbox.com/votebox/21/client-side-encryption>

Ya know, just in case you want to upvote it.

------
famousactress
I'm not sure I totally understand. Is he saying that the users who were
already logged in during the code update were the ones who had access to any
account? Or is he wrangling words and letting us know that < 1% of Dropbox
users had active web sessions at the time?

I understand the challenge that the Dropbox team finds themselves in here, but
I'd very much like more details as to exactly what happened and what was
possible for 4+ hours. I'm less interested in they're eventual report of the
bad stuff they think did or didn't happen during that time.

~~~
georgemcbay
He's saying that for any account to have been potentially compromised due to
this bug it would have had to occur during the ~4 hour window the bug was
active. During that window less than 1% of all dropbox users logged in, so
that puts a cap on how many users could potentially have been compromised. Of
course, of that "less than 1%" most were probably valid logins, so that 1%
doesn't represent only compromised accounts but rather a ceiling on how many
could possibly have been compromised.

~~~
famousactress
Thanks.. That makes perfect sense.

That said, I can't help but feeling misdirected. I mean, obviously the cap on
the number of compromised accounts is relevant, but I think more relevant is
the fact that 100% of the accounts were completely insecure for _hours_.

Misdirected, because as a user I don't care at all how many accounts were
_actually_ compromised. This isn't a no-harm-no-foul incident. It's an
enormous breach of trust that causes me to completely rethink what I'd be
willing to do with their service.

------
acangiano
The lack of "sorry", "apologies", "rest assured" and similar keywords is
astounding. Where is the heartfelt apology for the screw up?

~~~
sunchild
Deleted by lawyers? Dropbox is doing a bang-up job of squandering their
amazing reputation lately.

------
ary
As it could so easily be seen as bashing/trolling I need to preface this
question by saying that I'm asking in complete seriousness.

How does one accidentally set the failure mode of _any_ piece of
authentication code to escalate privileges instead of denying them? Even when
I was first learning to code web apps I never found myself accidentally
accepting _any_ password.

It's shocking (to me, at least) because it seems like such a beginner mistake.

~~~
dunham
On possibility: A developer short circuits permissions checking for local
testing and accidentally checks in the change. But automated testing should
catch these kinds of mistakes.

------
JohnsonB
They made an apparently appalling security error; in the name of transparency
they should be more specific about what code was the problem and how this bug
worked so people have a better idea of really how bad they screwed up. Right
now it could be something that could have blindsided anyone or something
incredibly obvious that they should have seen or had some process to detect.

------
mark_h
Since a few people have commented on finding out via HN rather than Dropbox
directly (as did I), I thought I'd post that I received an email from them
just then.

In my case though it included "we noticed some potentially suspicious activity
during the period", which for me was linking the desktop app. I don't know if
everyone will be receiving a note, or just those who fit the potentially-
suspicious profile. A quicker notification would have been nice, but I
appreciate that they're also obviously looking for any aberrant behaviour.

------
getsat
Sounds like they need to add a spec where a valid user tries to login with an
invalid password. That's quite embarrassing.

------
biot
From their FAQ: <https://www.dropbox.com/help/27>

    
    
      "We have strict policy and technical access controls that
       prohibit employee access except in these rare circumstances."
    

Perhaps they accidentally published an internal version of their
authentication code which allows a Dropbox employee to view files for any
account?

~~~
ary
If such code even exists why would it require them to enter a password?

~~~
biot
You're assuming their internal tools have the same fields and field validation
requirements as public-facing login mechanisms, while is likely not the case.

------
Rabidgremlin
I'm pretty sure that accepting any old password during login should be
considered more then a "bug" !

------
ca136
What are people storing on dropbox that they're getting so upset by this? The
chance that someone took advantage of this bug and accessed YOUR account are
so small. I thought the blog post was fine, but I guess it's a good lesson for
founders to take these kind of security issues more seriously than they they
are - or else your users will freak out... It shouldn't have happened, but
they'll learn from it and improve their system so it won't happen again.

------
gst
If Dropbox would have used client-side encryption (in addition to their
server-side security measures) you wouldn't need to worry about issues such as
server security.

That's why you shouldn't trust statements like: "Theoretically we can read
your data, but we make sure only you can access it". If there's the
theoretical potential for abuse it will happen sooner or later. Either
deliberately or not.

~~~
guan
This is true, but I don't wish for this. Dropbox with client-side encryption
would be a different product where a lot of their features would be very
difficult to sustain. For example, easy web access or having iOS and other
apps easily sync using the Dropbox API.

I am happy with not putting sensitive files on Dropbox, and encrypted the few
I do.

------
donohoe
2 major issues in 4 years (unless I missed something?). Not bad - can't say
the same for some 'traditional' companies.

As long as this average doesn't spike I am very happy with Dropbox overall.
They have enabled me to do so much and to work better.

~~~
fredoliveira
Sure. But here's the thing: people don't usually _forgive_ things like this.
People heard about an issue that potentially affected their accounts through
the media before they heard it from Dropbox - that's the disappointing bit.

On top of not letting something like this happen (where were their tests?),
they should be diligent in talking about something like this immediately so
that people know they're not being lied to, ever, and that they can trust the
service with their data. By not doing so, we have to wonder about how many
other bugs do we not know/hear about that might affect our accounts too?

------
plusbryan
Remember folks, unit testing is your friend.

~~~
leftnode
Wouldn't a functional test have caught this better? Ideally a unit test would
use mock objects for all external resources, so it probably wouldn't have
caught this. A full end to end functional test would have.

~~~
benatkin
Yes. Functional test, integration test, request test (as rspec-rails calls
it).

Better if it tests the production site directly, in addition to testing in a
test environment.

------
dawson
I'm kind of glad this happened as it's forced me to finally get an AES volume
setup on Dropbox using TrueCrypt.

------
andrewcooke
someone at work just asked me: what's a better alternative? any suggestions?
they are reasonably technical, so can cope with something a little more
complex than dropbox, but they want secure, easy-to-use, long-term reliable...

------
st3fan
You would think that proper unit tests will catch these kind of mistakes.

~~~
jerrya
Yeah, well that will come in their next round of funding. This round is all
about expanding the base.

------
zecg
rms, still winning.

------
ltamake
They're getting torn apart there. Come on guys, it was a bug that existed for
a short amount of time and nobody was affected afaik. I know I'm gonna get
shit for defending them, but come on. If you're using Dropbox to store your
passwords or confidential documents, you deserve to have your account hacked.
It's harsh, but it's an unfortunate reality.

~~~
ca136
Totally agree... It sucks that it happened, but come on, take a deep breath.
I've been trying to think of any documents I might have that would really
cause me to get so upset about this... Some might be a little embarrassing,
but I really can't think of anything that would make me this pissed off.

~~~
ltamake
I just had a friend tell me that he stores PORN on Dropbox.

------
nhashem
Arash, man, I don't know you, but I think it's totally badass that you guys
apparently deploy code on Sunday afternoons. This was a pretty terrible
oversight, but whatever your 'safeguards' are, don't do silly things like
"only production releases on Wednesdays" or all the usual things that add up
and cause a company to shrivel into a petrified husk.

