
Coinbase down for maintenance - winterismute
https://www.coinbase.com/
======
wonderwonder
Just my opinion but I think the conspiracy theories are not realistic.
Coinbase is essentially a startup that is dealing with monstrous traffic. They
are still scaling their infrastructure and when the price starts moving
quickly everyone logs in to either get in on the action or take a look. This
cranks the traffic up and the servers cant handle it.

I am not excusing their behavior as they need to be able to handle it,
especially when their server status affects the bank accounts of real people.
But I feel that it is understandable with out condoning it. The simplest
explanation is likely correct in this case.

I don't think many of us could have predicted the volume and interest in BTC a
year ago and neither did Coinbase's purchasing and infrastructure teams.

~~~
gamblor956
Coinbase goes through this charade every time this happens with a major
digital currency. It wasn't a legitimate excuse the first time, and it's not
an excuse the 7th time.

Scaling is a solved problem. You have AWS, Google Cloud, and MS Azure to
choose from with financial institution/HIPAA-level offerings.

~~~
haburka
Lol scaling is not a solved problem. Garbage collection is a solved problem.
Scaling is always a challenge.

~~~
appdrag
Scaling is by design, seems it was not designed to scale (serverless
architecture and db)

~~~
nikanj
Alas, those silver bullets don’t work either. Scaling still remains a multi-
faceted, complex problem with a lot of variables.

------
pcf
My friend thinks the exchanges also go down when assets "flash-crash", because
it means normal users can get in to sell their assets. And when they can't get
in, they also can't cancel their stop-loss orders, so it means a TON of assets
are available to buy cheaply (by the exchange and/or certain VIP
users/collaborators) as the value goes down. Then these are bought up, and the
price quickly increases again, before the exchanges slowly come online again.

Therefore he thinks the people exchanges are either part of the "flash-
crashes" (and buy up assets themselves, because they know exactly where most
of the stop-loss orders are placed), or they get kickbacks from the whales
manipulating the market, by giving them access to the database with all the
limit and stop-loss orders.

~~~
twohearted
This has always been a worry in the traditional stock market. With a stop loss
order you're basically creating a "secret" order to sell at below market
value. Anyone who knows about it has a direct incentive to profit from it.

Stop-losses are very important in investing but I'd recommend always executing
them manually at market rate. Set a price alert to notify you. Of course if
the exchanges are down you're sol, but if BTC is on a bullet train to $10 your
exchange stop-loss probably won't fill at your price anyway.

~~~
rsync
"Stop-losses are very important in investing but I'd recommend always
executing them manually at market rate."

No, no, no, no ...

You _do not_ ever want to insert a market order[1] ... the market can be
manipulated and high frequency traders can buy/sell your public market order
at an artificially low/high price.

The answer to getting "stopped out" is not to insert a market order - it is to
create a fill-or-kill at a specific price or other more specific order type
...

[1] [https://www.marketwatch.com/story/this-is-why-you-never-
use-...](https://www.marketwatch.com/story/this-is-why-you-never-use-a-market-
order-during-a-selloff-2015-08-28)

~~~
twohearted
Stop loss means get out NOW to stop losses. Trying to finesse a nickel out of
a trade at that point is a sucker's game.

Getting "stopped out" is fine, it's part of investing. You need to stop your
losses. I'm just saying don't put the order on the books for the exchange to
pluck like a ripe berry.

------
siculars
Coinbase being unavailable during times of extreme volatility is not a new
thing. Today's swing was 14k up to 19k then down to 15k or so over a period of
an hour of severely degraded access.

I'll go out on a limb and say that much of the recent spike can probably be
attributed to the 100k++ accounts open on Thanksgiving weekend getting funded
today and everyone losing their shit mashing the Market Buy button which get
instantly executed. The mechanics of vast market buy orders with no liquidity
is well known.

~~~
nlperguiy
On some exchanges it was an opposite direction. Down to 12k then up back to
15k.

------
tome
Now what would happen if Coinbase went down at a time when Bitcoin was going
through a short period of decreasing value?

~~~
fullshark
This is exactly my question. What happens when these exchanges face the
inevitable panic/bank run?

~~~
rednerrus
We've been through this in the past. They just cancel your transactions
without apology.

~~~
emerged
That's what they did to me. Incredible how much naivety surrounds and excuses
their behavior. There is no recourse when they "accidentally" lose your
purchase at $200 once it jumps to $450.

------
ajcodez
Build a financial exchange with Ruby on Rails, MongoDB, and Redis. What could
go wrong?

[https://engineering.coinbase.com/november-29th-
december-1st-...](https://engineering.coinbase.com/november-29th-december-1st-
post-mortem-69a1810f9910)

~~~
wilde
Stripe is on Ruby and Mongo and they seem to be doing OK.

~~~
ajcodez
Good point.

~~~
nikolay
Why do people still use a document database with no transactions for domains
with a fairly slim and rarely changing schemas?

~~~
j4ship
the expertise of their engineers leans that way .... why do we have many techs
that can do the same thing? ... engineering is a social science

------
Animats
The big question now is, what happens when somebody sells big for dollars.
This would be an excellent time for someone with 10,000 Bitcoins to sell, take
their $160,000,000, and go home. Could the market handle that?

~~~
oneeyedpigeon
There was a _big_ multimillion dollar sale today, possibly the cause of the
small dip. Looks like it handled it.

e: actually, tbf, it was a buy and a sell:

[https://blockchain.info/address/366k4bqibNrbUyby4xtkDT1demdc...](https://blockchain.info/address/366k4bqibNrbUyby4xtkDT1demdcHS8fVA)

But it does demonstrate the market can just about handle large transactions.

~~~
Animats
But did it result in an exchange actually transferring millions of dollars out
of the exchange to a real bank? That's the question.

~~~
smokeyj
Compared to national currencies bitcoin is now top 20 in terms of market cap.

~~~
tomjakubowski
What kind of currency is measured in market cap? Isn't that a chicken-egg
problem?

------
ktta
Bitcoin hit ATH on coinbase, then crashed. Something's not right. The exchange
is still not really 'up'.

[https://www.reddit.com/r/Bitcoin/comments/7i7mix/great_gdaxc...](https://www.reddit.com/r/Bitcoin/comments/7i7mix/great_gdaxcoinbase_just_crashed_right_before_20k/)

Can we change the link to
[https://twitter.com/coinbase/status/938802155925913600](https://twitter.com/coinbase/status/938802155925913600)

~~~
sciurus
Also [https://status.coinbase.com/](https://status.coinbase.com/)

~~~
dwaltrip
That's a nice, detailed status page. Although, some would say that it's a bad
sign such a page is needed.

~~~
woolvalley
Apple:
[https://www.apple.com/support/systemstatus/](https://www.apple.com/support/systemstatus/)

Google:
[https://www.google.com/appsstatus#hl=en&v=status](https://www.google.com/appsstatus#hl=en&v=status)

AWS: [https://status.aws.amazon.com/](https://status.aws.amazon.com/)

~~~
dwaltrip
Fair enough, I see what you mean. I take back my statement.

------
shiado
As somebody who doesn't have a background in finance, what are the legal
obligations of an exchange to stay online? How can a user of the service be
certain that an exchange is offline for technical or non-technical reasons?
Let's imagine the likely scenario when the price suddenly drops like a rock
and the exchange goes offline completely (for technical or non-technical
reason). Could a user make a claim that Coinbase is liable for their losses
because they were unable to access their coins?

~~~
gamblor956
A financial institution has fiduciary obligations to its clients to maintain
the assets (here, the Bitcoin, ether, etc.) entrusted to them.

However, financial institutions are not currently obligated to maintain their
clients' ability to _transact_ those assets, let alone within a specific time-
frame. Many people have sued their brokerages for not letting them sell stock
in a timely fashion to take advantage of price changes; essentially all of
them have lost or settled for pennies on the dollar.

------
simonsarris
It's a matter of time until Mt. Gox part 2 I think. It may not even take
nefarious workings at the top, just sloppy work somewhere + one bad employee
or outsider.

Is this going to end in a re-discovery of exactly why we have FDIC? Or the
SEC? How do people even know how much bitcoin activity is real and not
essentially faked?

Popular Google Searches by year, by my estimation.

2016: "what is bitcoin"

2017: "what is ethereum"

2018: "what is a wash trade"

2019: "can you eat squirrel"

~~~
baoha
>> How do people even know how much bitcoin activity is real and not
essentially faked?

Define fake

~~~
Rmilb
Ignore exchanges where there is 0 trading fees like the big one in Korea.

~~~
runeks
Why? Volume isn’t important, it’s order book depth that matters.

------
SilasX
You really shouldn't be surprised, at this point, by Coinbase going down
during times of high activity. It happened on November 29th (when, in
fairness, everyone was going down), and happened four times last June, which
should have been their canary-in-a-coal mine.

------
TFortunato
I do wonder if after the Ether "flash crash" recently
([https://blog.gdax.com/eth-usd-trading-
update-5d8142b5bdc1](https://blog.gdax.com/eth-usd-trading-
update-5d8142b5bdc1)), if Coinbase / GDAX instituted some sort of "circuit
breakers" on their exchange to prevent this type of thing from happening
again. Bitcoin and the other cryptos, trades at a pretty low volume relative
to their market cap, so if a lot of people we're trying to dump large orders
this morning to take a profit at the new ATH, I could see that causing similar
issues that they may have been trying to prevent.

~~~
bhouston
Circuit breakers do not matter if other exchanges continue to trade the same
currency. It would just cause the exchanges that do the circuit breaker to
possibly hurt their customers.

The only effective circuit breaker is some form of synchronized method, but
will the different exchanges cooperate? They really should put in circuit
breakers this week across all exchanges so that when really bad stuff the
panic doesn't destroy tons of people. It will be bad, but at least if the
trades can complete and things unwind in a moderately slow fashion, it won't
just 100,000 attempted transactions per second with random successes.

~~~
FireBeyond
> They really should put in circuit breakers this week across all exchanges so
> that when really bad stuff the panic doesn't destroy tons of people.

"Decentralized! Deregulated! But coordinate to protect us from ourselves,
don't let us lose money!"

~~~
TFortunato
Haha, yeah I can't see the BTC crowd all being on-board with this.

That said, while I don't think the exchanges need to (or realistically could)
coordinate on circuit breakers, I do think it's entirely reasonable for there
to explore this type of mechanism on a per-exchange basis in the case of GDAX
and other exchanges where margin is allowed, to prevent the whole "cascading
margin call" thing that happened with ETH to happen again. That could take a
lot of forms, including limiting the size of market orders to not eat 30% of
the book, or not allowing 1 market order to trigger margin calls if (insert
criteria here)

(That said, they could also just say "that's how margin and trading goes on
low-volume, highly volatile assets" and technically be correct, but much like
their decision to reimburse some traders, this is more of a business decision
and wanting to keep your customers. Not sure what the "right" answer is.)

------
prawn
If this phase of Bitcoin is likened to each goldrush around the world, isn't
it fairly inevitable that there will be hiccups? Robberies, banged together
shopfronts, people using any tools they can find, etc. And the internet has
long been a bit of a Wild West already.

Assuming they're adding thousands and thousands of customers quickly, I'm not
surprised that their systems and support are lagging. Comes with the (Wild
West) territory.

------
hoodoof
I wouldn't want to be in a business in which the worlds absolute finest
hackers and crackers - the elite of the best of the best - make it their full
time job - forever without end - to get into your systems and steal everything
of value.

~~~
lunaru
That sounds like any bank. Computer systems as the source of truth for
billions of dollars of assets is not new.

~~~
will_hughes
Banks have the advantage that if they see a huge spike in their outbound
inter-bank transfers they can suspend traffic while they investigate it, and
then if necessary call up the recieving bank(s) and issue a freeze on those
funds. There's already established relationships and agreements for these
things.

With a cryptocurrency there's nobody to call, you have to convince 50% + 1 of
the mining pool to undo any major breach - which is almost certainly never
going to happen.

------
syntaxing
I'm curious what the implications are that all these exchanges go offline
whenever there is a huge spike in bitcoin.

That being said, anyone care to elaborate how the price of bitcoin is
determined? I mean it's easy when there is a large central bank monitoring
this figure. However, how exactly does it work for bitcoin? What makes it
worth the $15K value when there is no "goods" to back it up. Are there anyone
here buying bitcoin as an investment?

~~~
wskinner
What makes it worth 15k is that people will pay that much for it. The price is
determined by buyers and sellers, trading with each other on markets called
exchanges.

~~~
syntaxing
I get the "market" determines the price but I guess I'm confused with who is
agreeing to the value? Is there a small amount of big players that are playing
with the price? Or is there a general consensus that the price is $15K. And
wouldn't there have to be people that we willing to pay this price? What
prevents the exchanges from colluding and raising the price artificially?

~~~
frgtpsswrdlame
>Is there a small amount of big players that are playing with the price?

We don't really know.

>Or is there a general consensus that the price is $15K. And wouldn't there
have to be people that we willing to pay this price? What prevents the
exchanges from colluding and raising the price artificially?

Well the exchanges could be running buy-bots but I think the more likely
collusion is going to be big-players selling to themselves, if a big player is
filling up both sides of the order book they can make it look like there is a
lot of activity and drive the price up that way. Now is this that or is it
organic? I'm not sure we have a good way to tell.

~~~
mancerayder
Naïve question here: since wallet addresses are fixed, couldn't you detect
transaction activity unusually repetitive between a few wallets? Couldn't some
clever learning algo detect anomalies such as what you hypothesized could be
going on?

~~~
frgtpsswrdlame
Probably. Could be an interesting data science project. One thing I'm not sure
about is how easy it is to make new wallet addresses. Do you know?

~~~
teraflop
A Bitcoin address is just the hash of a public key, so generating new
addresses is extremely cheap (on the order of microseconds) and doesn't
require interacting with the blockchain at all.

------
scarmig
What's the likelihood here that someone is in the middle of running off into
the sunset with a giant ɃɃɃ bag?

~~~
darawk
Pretty unlikely for coinbase. They always go down during high traffic periods.
Plus their principal owners are well known US citizens/residents.

------
rdp
This is another one of those moments in cryptocurrency where time seems to
freeze and everybody holds their breath.

~~~
djhworld
In regards to the volatility? sure.

But coinbase goes down for maintenance all the time

~~~
nemacol
When CB went down for a bit today I had the recent news of NiceHash in my
mind. Sure was a breath holding moment.

------
dewiz
how many things can go wrong when the service is overloaded... looks like
Coinbase doesn't know what stress testing or monitoring is.

* ID verification by either webcam or id pic upload fails, and eventually one reaches the daily quota, locking the feature for 24 hours (seems fair though)

* removing and re-adding bank accounts fails, the site says the account is already linked and it shows some ghost duplicates (i.e. the same bank account multiple times with different statuses)

* bank account verification fails, the site says the small amounts deposited don't match

* some buying operations fails due to API timeouts when collecting money from linked bank accounts

------
pfarnsworth
How can anyone reliably trade Bitcoin? With Coinbase immensely unreliable, I
can't see how anyone would tolerate running algos on their exchange, given
they could be in a bad position unexpectedly.

What are the other exchanges that you can trade BTC and are they more reliable
than Coinbase?

~~~
kruhft
yobit.net

------
nodesocket
Serious question, would it ever be possible to implement circuit breakers like
the stock market does for Bitcoin? Without safeguards, and the nature of
24/7/365 trading of Bitcoin, this all feels like teetering on the brink of
catastrophe.

~~~
londons_explore
Not really. Unlike stocks and shares which are traded on regulated exchanges,
bitcoins can be traded anywhere on any terms.

If one exchange implemented circuit breakers, people would simply move their
trading elsewhere.

------
sk2code
I never want to reuse my comments from other thread but this time I just can't
resist. No one really anticipated what we all are currently experiencing and
Coinbase is not different.

It seems Coinbase struggling to cope up:

$0000 - $1000: 1789 days

$1000- $2000: 1271 days

$2000- $3000: 23 days

$3000- $4000: 62 days

$4000- $5000: 61 days

$5000- $6000: 8 days

$6000- $7000: 13 days

$7000- $8000: 14 days

$8000- $9000: 9 days

$9000-$10000: 2 days

$10000-$11000: 1 day

$11000-$12000: 6 days

$12000-$13000: 17 hours

$13000-$14000: 4 hours

$14000-$15000: 10 hours

$15000-$16000: 5 hours

$16000-$17000: 2 hours

$17000-$18000: 10 minutes

$18000-$19000: 3 minutes

~~~
Snackchez
How much money needed to come into the bitcoin market for it bump from 15k to
19k? Where is this money coming from??

~~~
dx034
Not much. 1% of asset allocation for an institutional investor is enough for
that.

EDIT: You currently need $70m to bump the market above 20k on GDAX. That's a
lot for retail investors but not much if institutional investors get involved.

~~~
0xCMP
In the grand scheme of things sure, but definitely a large influx as compared
to the size of the market in Coinbase.

We're talking probably low 2-digit millions to make big moves I think.

~~~
dx034
Yep, around $70m to get it above $20k.

------
thedragonslayer
For how powerful we announce bitcoin/blockchain and its possibilities, the
faultiness lies in the exchanges and that is my biggest con with Bitcoin

------
Shoothe
Bitstamp.net was also unavailable for a while. Weird coincidence.

~~~
glorkk
Not really... websites regularly go down when they are under a load they
cannot handle

------
krisives
Coinbase doesn't support Tether so it should be okay.

------
bob_theslob646
Maintenance xor DDOS?

------
benmarten
it's back up; but not working

~~~
bonestamp2
Strange, the price data from coinbase never stopped working on
tradingview.com. I wonder what was affecting the price changes when the order
book was down?

~~~
asciimo
The books are on [https://www.gdax.com/](https://www.gdax.com/), Coinbase's
exchange intended for big trades. Coinbase is more of a consumer retail site.
I assume the latter is what is going down, but it's hard to tell as they both
seem fine at the moment.

~~~
TFortunato
As of now it is back up. If you take a look at the GDAX or Coinbase charts
over the past 1-2 hours on a short timescale though (i.e. 1 || 5 minute
candles on GDAX, or Past hour on coinbase app), you'll notice after the big
dump, there was a large flat period where no trades were occuring, even though
the site and app appeared to be up for a lot of people.

------
Snackchez
All of this is anecdotal:

These past couple of days, whenever a coin soars a ridiculous amount, on any
given exchange, that exchange seems to mysteriously go down. For example:
Binance and Bitfinex were down for "maintenance" (or just plain down), while
IOTA was experiencing a surge in price (from ~1.10 to 6$). Similarly this
happens with Bitcoin today (it seems Quadriga had some issues and Coinbase;
not sure about others).

Something is up, and it's weird.

My conspiracy theory: big money institutions are testing pump and dumps for
when short contracts come into play. On the one hand, the cryptocurrency world
wants to remain as unregulated as possible; on the other, a lot of individuals
are going to get reamed by big whales.

~~~
crazypyro
Isn't it just the simple explanation that big swings in the market bring
insane traffic spikes to market websites?

Add to the fact that people are also DDOS websites (doesn't take a "big money
institution" to do that) and it seems the explanation is clear...

~~~
superfrank
Yea. I work as a software engineer at a company that sees huge swings in
traffic on our product (by nature of the product). It's not uncommon for us to
see 20x swings in traffic day-to-day. It's taken us a long time to be able to
handle that and even then shit still goes wrong.

I'm much more inclined to believe the downtime is caused by scaling issues
rather than some big conspiracy,

EDIT: Just to give some insight, one of the problems we run into is that
usually only one issue is exposed during an outage. For example, lets say we
have 10 functions (or services or whatever) in a row that all call each other.
All the odd numbered ones have issues where they fail at scale. During the
first downtime, function one will fail and start to throw errors, but this
means the full traffic load isn't reaching the rest of the functions. By the
time the problem is solved, traffic has usually died down, so function 1 is
fixed, but 3,5,7,9 still have issues that haven't been exposed.

Additionally, at a quickly growing company like Coinbase, the code isn't going
to stay the same for very long, so even once those functions are fixed, new
issues are introduced.

It's easy to say they should be load testing at 10x their expected load to
prevent these type of issues, but it's really hard to replicate production
traffic. Users do weird and unexpected shit that can cause problems.

~~~
dmix
There's an article on the frontpage of HN from NYTimes that mentions Coinbase
has struggled with the massive spike in traffic recently.

It's funny how quickly people jump to overly-complex or borderline
conspiratorial explanations when the obvious is staring us in the face...

Coinbase might be a large well funded organization but it's still run by
developers with giant backlogs of tickets and conflicting priorities to build
new stuff vs maintain old stuff.

------
themihai
so it got hacked

