
Ask HN: Does anyone know what's going on at Coinbase? - joezydeco
Maybe a Coinbase engineer can anonymously tell us what's going on over there, because nobody trying to trade with Coinbase can get a public answer.<p>Money is going missing.  Transactions have been delayed for days and, in some cases, over a week.  Even amounts already in BTC aren't leaving the system.<p>Coinbase claimed the other day this was all due to a database migration[1], but they're offline once again.<p>[1]: http://support.coinbase.com/customer/portal/articles/1051063-pending-transactions
======
FredEE
Fred from Coinbase here.

There are still a few lingering issues, which we've been working hard to fix
and will be looking at again today. To those affected, I'm very sorry for the
wait. Funds are safe, it's just a question of opening the pipes properly to
let them flow normally.

~~~
PommeDeTerre
Can you please describe the implementation of this system, and the problems
encountered recently, in more detail?

Which database systems are involved in the "database migration" described in
the support center article? Please list at least the product names and
versions.

What was the nature of the "refactor"? What programming languages and
implementations were involved, for instance?

What exactly happened during the "database migration"? Was there any data
loss?

What sort of backup infrastructure is in place?

~~~
jmharvey
Im a Coinbase customer. I'm waiting for my pending transaction to go through,
and for USD to hit my bank account. And I'd rather have Fred and the rest of
Coinbase working on solving that problem than explaining what, exactly, the
glitch is, and while they're at it, what their entire infrastructure looks
like. There will be time for explanations later.

~~~
StavrosK
False dichotomy. I'm sure someone in the entire company has five minutes while
fixing the problem to explain to us what's going on.

~~~
wilfra
You may be overestimating how many people 'the entire company' is ;)

~~~
StavrosK
Judging by the design, two? :P

------
niggler
This is absolutely an issue that needs to be handled immediately. When there
is real money involved, trust/faith is by far the most important factor. If
there is any concern about missing money, people will lose faith in the
service and usually stop using it. To give an analogy, consider what is going
on in the minds of the Cypriots while their accounts are in limbo.

~~~
znowi
I agree that they should have been more vocal on this issue, cause information
vacuum gives way to posts like this and surely it doesn't do them good PR-
wise.

As for Cyprus, given how volatile and wild the bitcoin ecosystem is, I only
tend to _somewhat_ trust Mt.Gox and I view the rest as potential frauds (even
if YC-backed).

~~~
epscylonb
MTGOX is the most trustworthy and secure bitcoin service out there. I still
wouldn't put a large amount of bitcoin through it all in one go.

------
smallegan
Not sure if my story syncs up with what everyone else is seeing: -I purchased
10 coins on the 14th. -Money came out of my account on the 18th. -They said
they were available to me on the 20th via e-mail -Have tried multiple times to
send money to external wallets with no success -My account balance went from
10 to 0 to 10 -Support has been horrible

~~~
tpsreport
> -My account balance went from 10 to 0 to 10

That could potentially indicate a database infrastructure problem. Eventually
consistent databases can issue responses that appear to travel backwards in
time. And [1] says this:

    
    
      Coinbase uses MongoDB for their primary datastore for 
      their web app, api requests, etc. Coinbase is a
      decentralized, digital currency that is changing 
      the world of payments.
    

[1] <http://www.mongodb.org/about/production-deployments/>

~~~
wheaties
As much as I love MongoDB it has way too many issues to use it as a primary
data store for financial transactions. I hope they backed up and tested their
backup recovery. Something tells me they're dealing with a data
corruption/loss which wiped out their master and slaves without a backup.
Perhaps if they've got decent logging they can piece it together with logs.

~~~
Xorlev
I barely trust MongoDB with my personal projects. It's shot me in the foot
enough given they're not a very stable vendor (case in point, see the 2.4.0
replication bug -- we didn't get hit by that thankfully).

We've had issues where databases can get on divergent paths, then MongoDB will
keep up to 300mb of the stuff it can't match in a directory and after that
you're hosed.

------
metaverse
It's possible that they've been undercharging for their service, and as a
result have simply run out of cash. With the recent price growth, their buy
price sometimes fails to catch up with the current exchanges rates, resulting
in losses for them that their fee of 1% does not cover for.

My friends trying Coinbase have raised concerns about the long delays as well,
and I have a bitcoin transfer to my personal wallet that's been pending for
multiple days. This needs to be resolved quickly to keep Coinbase and Bitcoins
reputation.

~~~
jessaustin
_...their buy price sometimes fails to catch up with the current exchanges
rates..._

This is an existential flaw in their business. Why don't they just reject
transactions at expired prices? (If they wanted to be slightly more evil, they
could do this in one direction only.)

~~~
metaverse
Or just raise more money hoping that the price of a bitcoin will stabilize in
the near future. Their current system is very convenient for users.

~~~
jessaustin
I wouldn't characterize the experiences of the customers on this thread as
"convenient". If a commodity-trading system works except in cases of 2-sigma
volatility, then the system doesn't work.

I'm just responding to metaverse's reasonable hypothesis here. Coinbase ought
to diminish speculation by providing information, unless there are legal
reasons for them not to do so.

------
rdl
IIRC, Coinbase has a founder and an engineer, which doesn't excuse non-
responsiveness, but is probably the reason. They're probably hard at work
trying to deal with this problem, and haven't prioritized support or
notification of affected customers. That might have been a good choice if the
fix only took a few hours, but as it's dragging on a bit, it's probably better
to take some time out from fixing it and communicate with customers.

------
ashray
I hope they figure this out soon. This is the kind of thing that scares me and
the primary reason that I decided against speculating in bitcoins. For someone
like me who can't really afford to lose a lot of money, I need to trade in
historically stable stuff (even if they are as rigged as they are..).

I don't really understand bitcoin very well, that is reason #2. I hope the
Coinbase folks though have a solid grasp of it and can fix this otherwise this
is going to look very bad.

~~~
wting
If you can't afford to lose your investment, _you shouldn't be investing at
all._

Scared money is essentially lost money if you can't make rational decisions
without being influenced by fear.

As for #2, that can be solved through self education. Why use an investment
vehicle that you don't understand?

~~~
jessaustin
_If you can't afford to lose your investment, you shouldn't be investing at
all._

I doubt they would have let Madoff use this excuse.

I'm not saying these dudes are anything like that jackhole, but rather that
there is a difference between the risks inherent in the security or commodity
(in this case bitcoin) and the counter-party risk of your broker screwing up
the bookkeeping. Coinbase have made the latter risk appear significant.

~~~
wting
Madoff is a straw man; he committed fraud.

~~~
jessaustin
If you said this was a poor example, I would at least understand what you're
trying to say, but how on earth is Madoff a straw man in this instance? Try to
to separate your [legitimately] dim view of Madoff from the fashion in which
his example illustrates the phenomenon of counter-party risk.

~~~
saraid216
Hold on. I got this:

[http://www.shortpacked.com/2013/comic/book-14/09-same-
planet...](http://www.shortpacked.com/2013/comic/book-14/09-same-planet-
different-dimentia/strawman-2/)

------
jonpaul
I'm a Coinbase customer, and as an additional data point, their customer
service has been wonderful. Now granted, this was about a month ago. I did
send them an email on Thursday and haven't heard back from them, but my
request wasn't urgent, so I haven't thought much of it.

Overall, I'm rooting for them, and being YC backed, I'm confident that they'll
figure out their issues. I do hope sooner rather than later, as I can't wait
to use their API.

~~~
joezydeco
Sure, we're all rooting for them. But a mod just deleted the "YC ('12)" off
the headline. What should I think about that?

~~~
pg
Must mean that we're trying to hide the fact that we funded them, right? It
could only mean that.

~~~
niggler
It's a very relevant fact (after all, this is a YC site and Coinbase is a YC
company) that was explicitly removed by a moderator, and the action feels like
dissociation. Can you present a better explanation?

~~~
pg
That convention is for company names that appear in headlines of stories.

HN is a news site, not a customer support forum for companies funded by YC,
and in fact the site guidelines explicitly ask that it not be used that way:

<http://ycombinator.com/newsguidelines.html>

Strictly speaking I should have killed the post entirely, but that would just
have inflamed conspiracy theorists further.

~~~
joezydeco
I'm happy the post achieved it's intended effect, which was to get someone at
Coinbase to respond. So yeah, I'll be the first to admit that I abused the HN
post by turning it into a defacto customer service request instead of
something newsworthy.

But, on the other hand, this _is_ somewhat newsworthy. You have a company
trying to start up in a very tough space where you need to be super-reliable
and super-transparent. Coinbase's actions this entire week were neither. Let
me repeat that important middle part. An _entire week_. This is not an "OMG my
transaction took a day instead of an hour!" for one or two people.

It's not like I was posting a customer service request for myself. Look at
twitter and you'll see a lot of people with troubles greater than mine.

Perhaps I should have prompted some Bitcoin-related blogger to write a nasty
post and then link to it? Would that have turned the post into a "news" one
and passed the gate?

~~~
niggler
"Perhaps I should have prompted some Bitcoin-related blogger to write a nasty
post and then link to it? Would that have turned the post into a "news" one
and passed the gate?"

As per PG's guidelines, that is the only compliant way. If I were him I'd much
rather have the conversation on HN than to have the blogosphere or financial
media light up over a controversy.

------
famulus
This makes it pretty worthless: "Sorry, the maximum number of purchases on
Coinbase has been reached for today"

~~~
cobrabyte
Sounds like they got outside of their reserves in the BTC run-up this month.

------
waterlesscloud
I'm rooting for them, but they definitely need to communicate more.

~~~
merinid
Am rooting for them too - they are in beta folks, that's quite an honest
communication on its own.

~~~
dabeeeenster
This is ridiculous. If they are trading real money, they are not "in beta".

------
jensenbox
Their lack of transparency is not a new problem.

I had a problem with their API and it took days for them to even acknowledge
that I had even emailed them about it.

I doubt they will be around for much longer.

When it comes to money, customer service must be the first priority even if it
is to say "sorry".

------
jboggan
That's too bad, I had actually set aside today to learn their API and
integrate it into a project . . . nevermind.

On a related note, check out the insane price action on the exchange rate
overnight.

~~~
newnewnew
I would give them six months to migrate off of mongodb first.

------
djengineerllc
I've had a pending payment transaction to a couple places for about a week
now. :( Still waiting....

------
taylorhou
I tried to exchange for 10 coins a while back and they said somehow my bank
didn't approve of the ACH... that was when it was like $21 to 1 btcn.

------
mef
Perhaps they were running the client that got rolled back and got screwed on a
big double spend. Pure speculation but that did happen recently.

~~~
im3w1l
I thought the transactions were migrated over successfully?

------
phormula
I wonder if part of the current price drop is lack of transactions from
coinbase

------
wilfra
They are actively hiring for customer support if anybody wants to tackle the
job of improving this situation: <https://coinbase.com/jobs>

I know it's real money and people flip out but try to be understanding of
their situation. Their business is exploding and they're moving 1,000 miles an
hour in an industry that is literally being invented as we speak. Hiccups
should be expected.

------
yarou
Strange indeed...I've never used Coinbase, but I had heard good things about
them before. Hopefully they can resolve this issue in a timely manner.

------
michaelochurch
Wait a minute. Y Combinator funded a fucking _BitCoin_ startup?

f is Fonz/YCombinator, w* is water, and S is a fucking vicious shark.

Frame 1 (before funding a BitCoin startup):

    
    
        f
        wwwwwwwwwwwwww
             S
    

Frame 2 (decision to fund a BitCoin startup):

    
    
             f
    
        wwwwwwwwwwwwww
             S
    

Frame 3 (after funding a BitCoin startup):

    
    
                  f
        wwwwwwwwwwwwww
             S

~~~
tptacek
How does that attempt at snark even make sense? In frame 1, before YC funds a
Bitcoin startup, the Fonz is already on a trajectory that requires him to jump
over a shark.

Try harder next time, nascent evil 'edw519.

~~~
michaelochurch
I'm cynical, not evil.

I like what Paul Graham is trying to do, and I don't actually think one
obvious bad call constitutes "jumping the shark". If terrible, stupid
decisions constitute shark-jumping, then I shouldn't have any limbs left.

However, I strongly dislike Bitcoin (obvious scam) and don't know _why_ YC
would gamble its reputation in such a degenerate way as to associate with it.
Also, I felt the world could use some crude ASCII art.

~~~
SkyMarshal
PG already answered your exact criticism [1]. He's playing the extreme long
tail "black swan" odds.

 _"I don't know what fraction of them currently raise more after Demo Day. I
deliberately avoid calculating that number, because if you start measuring
something you start optimizing it, and I know it's the wrong thing to
optimize. [5] But the percentage is certainly way over 30%. And frankly the
thought of a 30% success rate at fundraising makes my stomach clench. A Demo
Day where only 30% of the startups were fundable would be a shambles. Everyone
would agree that YC had jumped the shark. We ourselves would feel that YC had
jumped the shark. And yet we'd all be wrong."_

[1]: <http://paulgraham.com/swan.html>

~~~
michaelochurch
Good-faith business failure is fine. It's excellent, insofar as it's a
byproduct of something very good: taking risks in a world of convex payoff.

Associating with a Bitcoin startup is a different story. Bitcoin is a sleazy
scam designed to enrich people who got in early and sold at the top.

It's not a problem of making a call that fails. That's going to happen in this
business. It's the taste issue. Unless they were something else in their YC
application and became a Bitcoin startup, they should have known better.

~~~
SkyMarshal
_> Bitcoin is a sleazy scam designed to enrich people who got in early and
sold at the top._

That's quite an accusation. Is there some smoking gun evidence we should all
be aware of?

Let's assume for a minute you're right, and it is a sleazy scam. How do you
see it playing out and ultimately

Disclaimer: I think it's certainly possible, but not a certainty.

~~~
tptacek
Let's count all the ways 'michaelochurch is being fatuous here:

* Even if you agree with him that Bitcoin is a scam, as I do, YC does not necessarily agree

* Similarly, Coinbase doesn't have to agree with us, in which case they're also not willfully furthering a scam

* _None of his points have anything to do with the thread_ , which isn't about intrinsic problems with Bitcoin but instead stability problems at Coinbase.

~~~
SkyMarshal
Fwiw I'm genuinely curious how people who are convinced BTC is a scam see it
playing out. Generally, once a scam is exposed, it collapses, so what does
that look like with bitcoin? But if that's too OT, I'll save it for another
thread.

~~~
tptacek
Early entrants in the market extract profits, leave the "currency" to deflate
back to Flooz valuations. At a certain point in the life cycle of things like
BTC, trading takes on a life of its own. I'm sure tulip traders could have
given you an impassioned defense on the intrinsic long-term value of tulip
bulbs during that bubble, too.

