
Update on Coinbase Data Security - nathancahill
http://blog.coinbase.com/post/81407694500/update-on-coinbase-data-security
======
downandout
While not ideal, I think this is being blown out of proportion by someone that
doesn't like Coinbase. For starters, of the 2042 "leaked" emails, 1153 are
unique. That means the person that posted it was trying to pad their results,
which combined with the possible but unfounded FBI/Fincen accusations,
illustrates that someone is mad at Coinbase and is lashing out.

Enumeration isn't a fantastic idea, but given its ubiquity in various forms on
major sites throughout the internet, I don't think it's worthy of all of this
negative attention directed specifically at Coinbase either. I once wrote a
program that could take a list of random emails and use Facebook to turn it
into a CSV matching each email to a name, a list of their friends, their
location, and interests. That should have been scandalous, but it wasn't.

We are acting as pawns in someone's revenge scheme against Coinbase.

~~~
bushido
_> That means the person that posted it was trying to pad their results_

Not necessarily. The duplicates are in exact order(quick check using sublime).
Could have just been a double paste, happens very often.

The fear mongering (FBI et al) definitely seems unfounded.

 _> illustrates that someone is mad at Coinbase and is lashing out_

That's an ad hominem attack.

 _> I don't think it's worthy of all of this negative attention directed
specifically at Coinbase either_

Agreed Coinbase is the target of a lot of negative attention. That does not
discount that enumeration deserves any less attention, it's unnecessary and
poses more risks than benefits (again I don't know of a single benefit when
requesting funds - when sending funds it is understandable but still
problematic).

 _> We are acting as pawns in someone's revenge scheme against Coinbase_

You're giving people too little credit. Coinbase is bad at communicating
(timing and message), bad communication pisses people of. They are also in a
business that gets more scrutiny than other payment processors.

~~~
300bps
>> illustrates that someone is mad at Coinbase and is lashing out

>That's an ad hominem attack.

No it isn't. I honestly wish that people would stop erroneously calling out
logical fallacies. An ad hominem attack is refuting someone's argument by
attacking their character in a way that has nothing to do with the discussion.
For example, this is an ad hominem attack:

Obama: ObamaCare has insured 7.1 million people through the exchanges.

Sally: Oh sure, but what difference does that make - you're a muslim and want
this country to fail!

That is an ad hominem. Because Obama being (or not being) a muslim is
irrelevant to the discussion and is only used to impugn the character of
Obama.

What is not an ad hominem attack is evaluating evidence of someone padding
numbers to make Coinbase look bad and then determining that they must be
biased against Coinbase. If you think evaluating evidence and coming to
negative conclusions about someone is an ad hominem attack, then you are
seriously mistaken.

------
zorpner
_You’ll find that user enumeration is possible on Facebook, Google, Dropbox,
and nearly every other major internet site._

And yet, most banks & payment processors do not do this, for good reason.
Seems like Coinbase is suffering from some domain confusion.

~~~
barmstrong
As mentioned in the blog post, payment services also commonly allow user
enumeration, including Paypal, Venmo, Square Cash, and others.

The reason you don't see it with banks is that they don't allow you to send
money to an email address.

~~~
waterside81
Maybe in the US they don't, but in Canada you can. I'm quite sure in most of
Europe & Australia you can as well.

~~~
patmcc
Interac e-Transfers (the only widely used method for doing this I'm aware of)
do let you send money to someone via an email address, but it's a notification
channel, nothing more. An account enumeration isn't possible with it, the
actual email is sent some time after the money for the transfer has left the
source account, and the sender doesn't get any information about the delivery
of the email.

I suppose you could send e-Transfers to random email addresses and then see if
any are accepted, but that would cost you an absolute minimum of $0.01 per
attempt and would probably have a terrible response rate.

Source: this is my day job.

Edit: sorry, forgot to mention this is Canada-specific.

------
yid
I'm curious why, given the prior reports of security issues at Coinbase and
the ongoing drama with Mt Gox, you guys didn't _immediately_ hire, say,
tptacek's company to do extensive penetration testing and a full security
audit. It appears that not all API calls were rate-limited, as they probably
should have been, and there certainly doesn't seem to be any sort of
monitoring of brute-force attempts like this in place. With all the negative
publicity around Bitcoin exchanges, you should have doubled down on security
weeks ago, or at least explained the privacy tradeoffs in your design
decisions clearly.

~~~
FredEE
A few thoughts. I agree with you, which is why we are currently going through
a third party security audit in addition to the impromptu peer review by
Andreas the day MtGox went down and our normal reviews by accountants. We also
hired a director of security from FB. Also, there were rate limits, just not
well tuned enough. So it's definitely in focus for us.

Hope this helps clarify

(edited for formatting)

~~~
dobbsbob
What precautions have you taken against meatspace robbery? What's to stop 3
thugs with guns walking into your office(s) and cleaning out all the coins?
Can you get insurance against this?

Do you also have measures to prevent evil janitor attacks like hardware
keyloggers being planted at 4:00am? Do you have screens facing an open window
to watch from across the street? Can I rent beside your offices, drill holes
through the walls and set up spycams or gain entry? Not to sound alarmist but
seems no exchange has given a thought to physical security meanwhile bank
execs are dropped off at work by private guards specializing in counter-
kidnapping operations, even though their money is fully insured and extremely
difficult to steal. Bitcoin's are easy to steal.

~~~
mcherm
> meanwhile bank execs are dropped off at work by private guards specializing
> in counter-kidnapping operations

Perhaps there are some bank executives for which this is true, but it is
absolutely NOT the case for all banking executives. I work with some bank
executives and they drive themselves to work in their own cars. The buildings
DO have alarm systems and it is quite possible for the FBI to respond to
physical threat incidents (because it is treated as a bank robbery) but
otherwise there is little that is special in the way of physical security.

And for Coinbase, I believe the lack of special physical guards is
appropriate. A high percentage ("up to 97%" according to
[https://coinbase.com/security](https://coinbase.com/security) ) of their
coins are in cold storage and while I am not privy to the details of
Coinbase's arrangements, keysharing and multiple physical storage locations
that are off-premises are a reasonable precaution. They are vulnerable to
hostage-taking or "3 thugs with guns" to the exact same extent (no greater) as
any other company with a similar amount of protection.

I can't comment on protection against hardware keyloggers: it's a threat that
they need to be prepared for. Cold storage is one major way of protecting
against this threat, business insurance is another.

~~~
dobbsbob
Any other company doesn't need to worry since robbing their head office and
demanding online bank transfers is a waste of time. A cryptocoin fixed rate
exchange with millions in storage you can instantly transfer is a different
story. It's like Ft. Knox being located in a regular office building with gold
piled on the desks. Bank vaults have physical security so why don't Bitcoin
based businesses.

I did read through their security about the backups being spread around
different locations, but those are backups. They would need access to the cold
wallet on a regular basis if 97% of funds are truly in there. Unlikely to
happen but then again police here didn't expect criminals would remove huge
concrete barriers with a stolen tractor, ram a shopping mall entrance, drive
through the mall and ram a gated jewelry store but they did.

~~~
mcherm
> They would need access to the cold wallet on a regular basis if 97% of funds
> are truly in there.

Not true. First of all, that would only be true if their net daily turnover
were more than 3% of their total amount stored -- which it may not be. Even
then, I would expect graduated levels of cold wallets: imagine one with
another 2% that is down the street in a bank safe deposit box, 5 wallets with
50% of the deposits stored in a way that can only be accessed with cooperation
of 4 people in different parts of the country ... that sort of thing.

I am, of course, just speculating: I don't know how Coinbase runs their
system, I just know that they seem competent and that this is how _I_ would
run such a thing.

------
danielweber
_We’d also like to address the claim of a “leaked” list of Coinbase emails and
user names. This list (the size of which is less than one half of one percent
of Coinbase users) was not the result of a data breach at Coinbase. This list
of emails was likely sourced from other sites - probably Bitcoin related ones.
It’s clear there was no data breach because no other user information is
provided._

That last sentence is doing a _lot_ of lifting, out of its weight class.

This immediate "no it's not true" might reassure some folks, but it scares me
because of how quick it is. Have you looked at the audit systems on your
database?

"We believe this information is bogus but are investigating to make sure" is a
better response, assuming you actually do investigate to make sure.

------
virtuabhi
I think many users will assume, with a finance site like Coinbase, that the
name they have provided is for $ transactions (taxes, Visa, MasterCard, etc.)
not for social aspects. In my opinion, getting behind the Privacy Policy and
claiming that users know their names will be publicly shared is unethical.

Why people want to hide their names? Personally I don't hide my name. But it
is not too hard to understand that on "web-scale" there will be someone who is
stalked, who has posted on suicide help forums, etc.

------
peterwwillis
(Bug report at
[https://hackerone.com/reports/5200](https://hackerone.com/reports/5200))

From Ryan McGeehan, director of security (user magoo):

    
    
      > This behavior is mostly informational to an attacker and does not
      > directly increase risk in any significant way
    

All information leaks are useful to an attacker. By themselves they are
harmless, but can be combined with other information to successfully exploit a
system.

From bug reporter Shubham Shah (user zero):

    
    
      > This request can now be replayed unlimited times, with unlimited email
      > addresses inputted. Coinbase does not limit the rate of POST requests
      > to /transactions/request_money
    

This should not be possible _at all_. The reporter must have made a mistake
and forgot to mention the X-CSRF-Token needs to be updated each time. If it
didn't need to be updated, this would be a basic CSRF vuln.

All this being said, the real flaw here is the lack of rate limiting on
transactions, for three reasons:

1\. The spam will eventually mount up and ISPs will block their servers for
days or weeks.

2\. Their network and app stack is subject to DoS attacks unless they rate-
limit transactions.

3\. Harvesting of e-mail addresses would be stopped by basic rate limiting of
email<->user queries.

------
tcarey83
I was phished for coinbase just recently with an email telling me "You just
received 0.08525920 BTC" and just "Click here to sign in and view this
transaction" and I stupidly I did click on the link and did try to log in. The
login failed (as it would with a trojan and the coinbase 2 factor
authentication I have enabled). But even so, the phishing site was able to
attach 3 Android apps to my account with full access. I deleted the apps and
notified coinbase, but they were totally less than helpful.

~~~
meowface
People should upvote this much more. This shows that the reported exposure has
resulted in at least one successful phishing.

The fact that the 2 factor auth can apparently be bypassed by attaching apps
is another security vulnerability entirely. If that is what you are claiming
is the case, then they should be immediately fixing this as soon as you
reported it to them.

------
pbreit
Less sympathetic than I was hoping for but copacetic. Could they have nipped
this in the bud with a faster response? Perhaps. However having dealt with
reports like this, I cannot recall a decent interaction with a reporter.

~~~
bertil
copacetic : in excellent order.

(For the lazy like me, who still want to learn new and useful words.)

~~~
bbrian
Three-finger click on OS X defines the word. Or right-click and Look up in
Dictionary.

~~~
bertil
Indeed, and a great trick too — but that appears to be a great but non-default
option, only working within Apple software (namely Safari browser when surfing
HN) and uses the System language (not English for me).

I personally prefer Right click “Search with Google…” in Chrome: it has the
upside of coming up with a definition when the word is actually rare -- so it
prevents me from defining an word I didn't know simply because I’m not a
native English speaker.

~~~
MBCook
It works in any software in OS X using standard text controls. It's _insanely_
handy.

The few programs I use that don't support it actually drive me nuts because
I've become so used to it.

The system language thing is a little annoying. It would be fantastic to be
able to look up the random Spanish or Japanese word, but I understand the
limitation.

------
revelation
_Rate limiting_

Do we have to spell it out to them?

~~~
FredEE
We are pushing some changes to rate limiting - this wasn't clear in the
original post and I just edited. Thanks for the heads up.

~~~
korzun
So why are you blowing this off and now all of a sudden writing rate limiting?

The only reason he got 1000+ emails is because you guys messed up.

Not even going into the whole idea of you releasing that end point with name
leakage without somebody going 'oh hey.. do we have rate limiting?'.

Mistakes like this are signs of amateur hour.

~~~
MichaelGG
Suppose they limit it to 100 emails before blocking your account. The guy can
just sign up with 10 accounts. Or 5. This "attacker" would still post it and
make a big fuss.

Most likely they're implementing rate-limiting to appease people and prevent
an ongoing spam issue. Or perhaps it was on their list for a while and just
hasn't been an issue until now.

~~~
korzun
In regards to rate limiting, it would be a much smaller number prior to block.

If it's IP based at let's say 10 over X attacker would have to lease 100 IP's.

In any case, rate limiting is the quickest mitigation prior to actual fix of
the data leak in question.

------
wes-exp
If Coinbase can't admit any amount of fault whatsoever for enabling the large-
scale harvesting of their customer list, I'm sorry, but I've lost faith in
their security.

This is a service that stores _digital cash_. It should be like an online Fort
Knox, not "safe as Facebook" like that's some kind of high bar.

~~~
tptacek
If you read the post even a little bit carefully, they refute the idea that
this was a harvesting of their database. One compelling bit of evidence they
present is that the list is tiny, and their customer list is very large.

It's not just that this isn't a "large scale" leak; it's that they say it's
not a leak _at all_ ; that this data was made available through some other
combination of services that exposed it, not Coinbase. They don't provide any
additional evidence (but few companies would) --- but it's a plausible
argument.

~~~
wes-exp
Coinbase's tone at
[https://hackerone.com/reports/5200](https://hackerone.com/reports/5200)
convinces me that they just don't care about user account enumeration.
Combined with the blog post, my sense is that Coinbase does not deny that
systematic enumeration is possible; rather, they deny that we should worry
about it. ("it's not a bug, it's a feature")

~~~
tptacek
I don't know if they do or don't, but I wouldn't be surprised either way,
because I sure don't care about user account enumeration. We doc it, but I'm
always embarrassed when we do.

------
korzun
Do you smell it?

They believe it's not a risk to their users (never minds those users who are
now targeted via e-mail leak because they have BitCoins).

"You’ll also find many leading payment services allow user enumeration"

They also do pattern monitoring and rate limiting. And instead of saying 'but
they do it!' they should be saying 'they do it too, but we think this is a
valid privacy issue that we need to fix'.

This is more or less 'If you don't get privacy implications , we will bullshit
you so you don't panic and go elsewhere with your money. Everything is fine!'

------
Ryel
I also think this whole thing is overblown but I hope it will help to further
humble Coinbase in realizing that they really need to focus more efforts on
squashing things like this before they become a problem. Coinbase has a large
responsibility in whether or not Bitcoin is to become "accepted" in the USA
and several small events like these left unsettled or left to fester could
prove catastrophic(IMO).

I don't believe Coinbase should consider this a real "security threat". I
believe that this is a negative side-effect of what may be a feature. It is
certainly something that needs improvement, as I'm sure all of the people
whose email has been leaked will tell you...

I'm not sure if Coinbase has an engineering blog or a similar outlet where
they can speak to more developers directly but if they do not have one
already, this may be the time to start. This could've been squashed entirely
within a small development community but when left unsettled for so long, it
is things like this that the news will latch onto and run with We all know how
blown out of proportion things get when that happens.

Anyway, long story short... I hope Coinbase improves. As much as I hate to say
that a tech company needs more representatie

------
tobias2014
I don't who is wrong or right here (Coinbase saying there was no breach or
alledged hackers via pastebin), but isn't having fake security breaches for
large bitcoin sites a good opportunity to manipulate the rates? If one wanted
to bring the rate down it might be possible to do this with creating a good
fake security breach of a large site.

------
scottlinux
Is there an option to opt your account info OUT of the api?

~~~
MichaelGG
I tried deleting my name from the user settings page. No problem. This "bug"
is literally about people being upset that information they opted-in to share
was shared.

------
jseip
Interesting to note that they did not dispute the claim (posted on HN with the
pastebin data) that they are providing transaction data to government
agencies.

------
FireBeyond
I always find my ‘skeptic’ meter ticks faster when I read of a data breach,
and find a company:

a) using language that is very specific when making a denial b) also
introducing a new Director of Security in the same post

~~~
dangrossman
No new director was introduced; the Ryan McGeehan hire was in the news weeks
ago.

~~~
FireBeyond
Thanks, I stand corrected. It was my gut reaction at the phrasing, all good.

------
broolstoryco
My email was not published anywhere with regards to bitcoin or coinbase, I
receive relatively little spam, yet I received 4 of these spam messages.
Smells fishy.

------
akennberg
You can add a soft rate limit with a captcha to make sure it's a human doing
the requests rather than a spammer's script.

~~~
dangrossman
You can't rate limit APIs with CAPTCHAs.

------
Aqueous
There was no mention of the IRS and FBI gag orders/data transfer in this post.

~~~
Mizza
I filed a Freedom of Information Act request to the FBI about Coinbase, and
they replied that they have no documents.
[https://www.muckrock.com/foi/united-states-of-
america-10/coi...](https://www.muckrock.com/foi/united-states-of-
america-10/coinbase-fbi-9727/) Of course, the FBI is explicitly allow to lie
in response to FOIA requests if it will protect an ongoing investigation.

However, I was also told at a party by a Coinbase employee that this is not
true (which is why I filed the request to begin with.)

I am certain that they have a relationship.

------
razfar
does anyone know what blogging software they are using for the coinbase blog?
is it tumblr?

~~~
dangrossman
If you view the source, you'll see tumblr right from the first line.

~~~
razfar
thanks boss

~~~
ParadisoShlee
You should write a book - Babies first pentest.

------
bowmessage
No mention of the claimed IRS / Fed gag order, interesting. (although I
realize its not their main focus right now)

~~~
MichaelGG
Well they can't mention it if they have it, but they can mention if they
don't. So this would be the only way Coinbase could communicate that they
_are_ under a gag order, barring a prior warrant canary. So it's probably
prudent to act as if they have acknowledged the gag order until they deny it.
Although it'd be really dumb if anyone was assuming the records were private.

I guess they could say "We have implemented a warrant canary at <url>" then
404, but perhaps their legal team wisely denied that one.

~~~
mcherm
Yes. And while we're at it, I wanted to point out that cbcbcb (who posted the
initial leak) is ALSO under a federal gag order. He/she either won't deny it
(because, obviously, they can't) or WILL lie and deny it (forced to lie by the
gag order).

Oh, and I'm under a federal gag order too... or at least there's no way to
prove that I'm not.

~~~
MichaelGG
Has it been shown that the USG can order an entity to lie?

~~~
mcherm
There has been speculation among legal scholars that the legal threshold for
ordering someone to lie might be greater than the legal threshold for ordering
someone to keep silent. This has not been tested in court (at least, not in
open court) so no one knows for sure. If I received a national security letter
ordering me to lie, I would have to think VERY carefully before deciding to
violate it and become a test case.

------
pearjuice
>there has been no data breach of names or emails at Coinbase

I have a Pastebin URL with 200 email addresses to prove the contrary. Why lie
in PR?

~~~
sp332
You have to have the email addresses before you can query Coinbase with them.
The addresses didn't originate with Coinbase, the attacker already had them.

~~~
MBCook
Were you certain they were Coinbase users? Did you have the first and last
names?

