Hacker News new | comments | show | ask | jobs | submit login
Coinbase down for maintenance (coinbase.com)
199 points by winterismute 4 days ago | hide | past | web | favorite | 239 comments

Just my opinion but I think the conspiracy theories are not realistic. Coinbase is essentially a startup that is dealing with monstrous traffic. They are still scaling their infrastructure and when the price starts moving quickly everyone logs in to either get in on the action or take a look. This cranks the traffic up and the servers cant handle it.

I am not excusing their behavior as they need to be able to handle it, especially when their server status affects the bank accounts of real people. But I feel that it is understandable with out condoning it. The simplest explanation is likely correct in this case.

I don't think many of us could have predicted the volume and interest in BTC a year ago and neither did Coinbase's purchasing and infrastructure teams.

I've used Coinbase on and off for a while. The service always felt exactly as you describe it: startup busting their collective ass to keep up with demand.

Given the recent increase in news about cryptocurrency I've expected "technical difficulties" with Coinbase. Heck, most mining pools are sweating bullets right now.

Hypergrowth = hyperpains.

missed opportunity to say:

hypergains = hyperpains

no hyperpains no hypergains

hyperpains are just hyperweaknesses leaving your body.

I would guess they're also likely the first stop for new entrants, as they're (or were?) the only US based exchange, easiest to use, and insured by FDIC/Lloyd's.

Coinbase has been my recommendation to anyone new to the scene who wants to buy/use bitcoin.

And obviously once finished, move all funds to paper wallet.

Which paper wallet software do you suggest?

Is it really that much traffic compared to your typical consumer-facing start up? There can't be that many people buying BTC compared to the number of people that say, Stripe or Whatsapp handled in their early days...

I don't know about Stripe or Whatsapp, but I know coinbase has had an average of 30,000 new users a day for a while, with spikes of over 100,000 new signups in a single day around thanksgiving.

It's absolutely bonkers amounts of growth!

Edit: more concrete numbers:

In January of 2016 they had around 2.6 million users.

On November 29th they announced they had over 13 million users.

Whatsapp famously had ~450 million users when they were acquired by Facebook. I assume that the average user sent quite a few texts, some pictures, occasional video... they also supported voice calls...

I don't know how much computing power is actually needed when you're just buying BTC. When I used Coinbase recently to take a small BTC position, I only made maybe 4 transactions in total. Is this really a lot to handle?

I don't have any experience in this area, but I imagine that fintech is a bit different than messaging. It's not the amount of data, but the nature of the data.

I'd imagine you can't use scaling tricks like eventual consistency or simple sharding here. Traders are going to be pissed if their most up to date information from the exchange is a second behind what they can see.

It's already at the point where they have a line in their FAQ for the GDAX API that tells people where to colocate from for the lowest latency.

And all of that data needs to be stored damn-near indefinitely, and securely. And integrated with traditional financial systems.

I'd love to see some information on what it actually takes, as i'm just guessing here, but seeing as there aren't any bitcoin exchanges that can seem to keep the servers running smoothly during spikes, i'm guessing it's not a trivial problem to solve.

Thanks, I'm not in finance but it seems obvious in retrospect that not being able to rely on things being eventually consistent is a bottle neck. Before thinking about this, the number of users that people threw around when talking about BTC exchanges seemed very small.

And yet the NYSE keeps on ticking.

... from 9 to 5, monday through friday, with a fairly predictable load, and decades more experience.

I'm not saying it's impossible, but the amount of growth they have seen is substantial and unprecedented. Nobody could have predicted it, and the rate of increase just keeps going up.

They are clearly new in this space, as all cryptocurrency companies are. I don't think that means that they should just give up and go home because they couldn't keep the servers up during spikes of insane activity...

> ... from 9 to 5, monday through friday, with a fairly predictable load, and decades more experience.

And a lot more operating capital, too.

There’s a lot less dangers involved in double-delivering Whatsapp messages, delivering them out of order, etc. Less constraints=easier optimizations.

That, followed by the fact that many of their users probably sit on the site a good part of the day constantly refreshing the page/taking up websocket bandwidth to watch price changes. Yeah, load is a serious issue.

I'm genuinely amazed that they can do streaming updated to GDAX as fast as they do.

Jan 2016 to now is 23 months. Still big but half as much.

Coinbase probably deals with lots of bot trading. Most consumer-facing startups deal with human-scale behavior, not bot-scale.

Ah that's good point, they probably generate a lot of traffic and could increase rapidly while the number of users remains somewhat small.

I think its also a matter of funding and company financials as well. You have to make money to justify spending money and each company has a different idea on what an acceptable infrastructure spend is. While it might be great to spend a ton of money and build a bullet proof infrastructure, if traffic slows down or revenue dips they are in the very negative position of having a lot of hardware and not a lot of money in the bank.

They have to scale at the rate their revenue or financial backers will be able to sustain. Based on the number of new users signing up to the service the down time is not yet a significant deterrent. If use drops because of downtime they will likely need to reevaluate a large capital spend to solve the issue.

"We want to run an exchange that the entire world will use for this crypto-currency that's become very popular and has increased in price 1000% since it became widely known."

I think I would have predicted that they'd need to run at a very large scale, and be able to handle massive spikes. Whether or not I could have convinced them to make the investment is another story.

The problem is what happens if that growth stops? What happens if bitcoin tanks right after you hired 300 new customer service reps and tripled your server hardware?

It's a gamble no matter what, but moving too quickly or moving too slowly results in the company going under.

In that case, I think straddling the line of "overwhelmed when there is a big spike" is a good spot to be in all things consitered.

You have layoffs and turn servers off. This isn't rocket surgery. They're skimming a ton of money off these trades every day, down time is a loss.

The optics on "servers are down due to a massive spike in volume" are much better than "Coinbase had another round of layoffs today..."

Currently they are signing people up at a dramatic rate despite their server issues. If they start laying people off because they spun up to fast they could start a panic that would kill them.

They have to weigh all of this in when deciding on an infrastructure spend.

I would also like to think from a purely personal level that they don't want to hire and fire people like that as these are real people with real families involved.

Coinbase goes through this charade every time this happens with a major digital currency. It wasn't a legitimate excuse the first time, and it's not an excuse the 7th time.

Scaling is a solved problem. You have AWS, Google Cloud, and MS Azure to choose from with financial institution/HIPAA-level offerings.

Bitcoin doesn't scale with "The Cloud". If anyone can get the private key, they have the money... sending it off to any other provider is going to end badly. Sure any other company doesn't care if "The Cloud" leaks.. it's just the customers problem, cause it's their data... However Bitcoin leaks, it's the companies money, so it's now again their problem.

Spoken like someone who truly knows nothing about scaling.

Imgur was reporting being over capacity just this morning.

By most engineers in the field it is a difficult problem with ongoing work.

Lol scaling is not a solved problem. Garbage collection is a solved problem. Scaling is always a challenge.

Scaling is by design, seems it was not designed to scale (serverless architecture and db)

Alas, those silver bullets don’t work either. Scaling still remains a multi-faceted, complex problem with a lot of variables.

The thing about scaling is that you can't justify the engineering involved in overscaling until you're reasonably sure you'll need the capacity. Otherwise that's called overengineering. Managers get fired for approving spending on things like that when the demand does not appear. It's also hard to simulate real traffic because they don't always behave how you predicted they would.

Also, your system gets more and more complex as you scale. More monitoring systems needed, more attack vectors to watch for, deployment systems, backup systems... Imagine the security level coinbase needs to have. It's somewhat unprecedented since a hack could make off with a lot of valuable assets in an instant. (even just from their hot-wallet)

You saying scaling is solved makes me question what you think their infrastructure looks like. It's not just an AWS loadbalancer with some shards behind it. It's A LOT more complex than that.

> They are still scaling their infrastructure and when the price starts moving quickly everyone logs in to either get in on the action or take a look.

Are they not using cloud, which should enable easy scaling? Or is their architecture not designed for good scaling?

No architecture scales infinitely, you merely shift at each step what the next bottleneck will be. Even with small scale sites you'll see this: first, you need to move from everything on one server to a load balancer plus app servers talking to a single database. Then you need to share your database. Then you need to move to multiple load balancers. Then so on and so forth, depending on the specifics of your service, mixed in with various rounds of local optimization at the application level that buy you just a little more time before having to ship another architectural change.

Most distributed systems don't really scale ass you add nodes beyond a factor of 10 to 100. If they did, you wouldn't need to hire super expensive people to handle those systems.

The cloud enables easier scaling, it's not a silver bullet. No matter what, scaling plans have a region where they break down, so it doesn't imply they aren't designed for good scaling.

You're not a programmer, right?

Cloud solves everything guys!!! Yes buy more Cloud now that's 10% down in the exchanges lol...

Slapdash startup-ism in an exchange for a Brave New Currency is just as scary as a conspiracy. It may be more scary.

At least with the Cabal theories you know someone knows what's going on. You might even have someone to blame/jail at the end of the tunnel!

You really seem to be wringing your hands pretty hard about an overloaded NGINX instance. Wouldn't a conspiracy that was stealing everyone's money be a lot worse?

No I'm saying that any reply approximating "its fine guys, they're new to this and won't anticipate everything" is disquieting when lots of coin is on the line.

My friend thinks the exchanges also go down when assets "flash-crash", because it means normal users can get in to sell their assets. And when they can't get in, they also can't cancel their stop-loss orders, so it means a TON of assets are available to buy cheaply (by the exchange and/or certain VIP users/collaborators) as the value goes down. Then these are bought up, and the price quickly increases again, before the exchanges slowly come online again.

Therefore he thinks the people exchanges are either part of the "flash-crashes" (and buy up assets themselves, because they know exactly where most of the stop-loss orders are placed), or they get kickbacks from the whales manipulating the market, by giving them access to the database with all the limit and stop-loss orders.

This has always been a worry in the traditional stock market. With a stop loss order you're basically creating a "secret" order to sell at below market value. Anyone who knows about it has a direct incentive to profit from it.

Stop-losses are very important in investing but I'd recommend always executing them manually at market rate. Set a price alert to notify you. Of course if the exchanges are down you're sol, but if BTC is on a bullet train to $10 your exchange stop-loss probably won't fill at your price anyway.

"Stop-losses are very important in investing but I'd recommend always executing them manually at market rate."

No, no, no, no ...

You do not ever want to insert a market order[1] ... the market can be manipulated and high frequency traders can buy/sell your public market order at an artificially low/high price.

The answer to getting "stopped out" is not to insert a market order - it is to create a fill-or-kill at a specific price or other more specific order type ...

[1] https://www.marketwatch.com/story/this-is-why-you-never-use-...

Stop loss means get out NOW to stop losses. Trying to finesse a nickel out of a trade at that point is a sucker's game.

Getting "stopped out" is fine, it's part of investing. You need to stop your losses. I'm just saying don't put the order on the books for the exchange to pluck like a ripe berry.

why would they cancel the stop orders when the sole purpose for stop order is to protect profit?

Coinbase being unavailable during times of extreme volatility is not a new thing. Today's swing was 14k up to 19k then down to 15k or so over a period of an hour of severely degraded access.

I'll go out on a limb and say that much of the recent spike can probably be attributed to the 100k++ accounts open on Thanksgiving weekend getting funded today and everyone losing their shit mashing the Market Buy button which get instantly executed. The mechanics of vast market buy orders with no liquidity is well known.

On some exchanges it was an opposite direction. Down to 12k then up back to 15k.

Now what would happen if Coinbase went down at a time when Bitcoin was going through a short period of decreasing value?

If there's a very large value swing, either up or down or both, services such as this are guaranteed to have to drink from the godzilla-scale firehose and learn to like it.

If there's a run on the bank, the little people are gonna get screwed, no doubt about it, no matter what store of value we're talking about (seashells, gold, paper currency, cryptocurrency, gold-pressed latinum, etc).

Hope you know Coinbase is only 8% of the total Bitcoin volume each day.


A run on one bank frequently has knock-on effects on other banks.

Bank runs are bad when there isn't a full reserve. Coinbase says it has 100% reserves.

Bank runs are how fraudulent exchanges are caught: Bitcoin is self-auditing, and either you have it or you don't, there aren't IOUs or paper receipts.

Fractional reserve banking is a prisoner's dilemma, we all marginally benefit from the status quo of credit/debit/interest, but in a crisis, the first people to pull out win, and everyone else loses.

What if Coinbase is the one buying large amounts of bitcoin? Then using reserves so they can buy/sell on both sides. See where it gets risky?

I wonder how quickly CoinBase could release funds from cold wallet(s)? Could be hours... which could pretty bad.

Right, "only 8%" is rather a lot.


Please read the guidelines and then start commenting civilly and substantively. We ban accounts that won't.


Yes but what is the % of withdrawls? I know most U.S. members transfer from the exchanges over to Coinbase to withdrawl so they need them eventually

This is exactly my question. What happens when these exchanges face the inevitable panic/bank run?

We've been through this in the past. They just cancel your transactions without apology.

That's what they did to me. Incredible how much naivety surrounds and excuses their behavior. There is no recourse when they "accidentally" lose your purchase at $200 once it jumps to $450.

Bank runs are bad when there isn't a full reserve. Coinbase should carry a full reserve.

Fractional reserve banking is a prisoner's dilemma, we all marginally benefit from the status quo of credit/debit/interest, but in a crisis, the first people to pull out win, and everyone else loses.

Coinbase isn’t a bank it’s an exchange. Why would they carry a reserve of anything?

It stores your shares for you, but it’s not loaning them out. The only way you get money out of Coinbase is if someone buys your shares.

They presumably have on hand the full amount of cash and Bitcoins that their participants have given them and don't loan any of them out. That means they have a full reserve.

They only need(* depending on TOS guarantee) to keep the full amount of Bitcoins matching your Coinbase wallet. They can take all the cash they've been given, pile it up and set it on fire if they like.

1 BTC at Coinbase is only worth whatever Fiat currency Coinbase is willing to buy it from you for. If that's $1 for all your Bitcoins, tough luck.

They're under no obligation to guarantee a sale price for BTC.

> They can take all the cash they've been given, pile it up and set it on fire if they like.

Then how do they pay you when you come to withdraw your cash?

Hopefully and unlike the exchanges using Tether.

You aren't buying the bitcoin from Coinbase...

No? My understanding was that if you want to sell Bitcons on Coinbase you have to give them custody.

I wonder how apt the comparison to a bank run is, given that AIUI Coinbase holds the Bitcoin of the exchange participants. Is there any legal form of sequestration of those assets?

Build a financial exchange with Ruby on Rails, MongoDB, and Redis. What could go wrong?


Stripe is on Ruby and Mongo and they seem to be doing OK.

Good point.

Why do people still use a document database with no transactions for domains with a fairly slim and rarely changing schemas?

the expertise of their engineers leans that way .... why do we have many techs that can do the same thing? ... engineering is a social science

The big question now is, what happens when somebody sells big for dollars. This would be an excellent time for someone with 10,000 Bitcoins to sell, take their $160,000,000, and go home. Could the market handle that?

There was a big multimillion dollar sale today, possibly the cause of the small dip. Looks like it handled it.

e: actually, tbf, it was a buy and a sell:


But it does demonstrate the market can just about handle large transactions.

But did it result in an exchange actually transferring millions of dollars out of the exchange to a real bank? That's the question.

Compared to national currencies bitcoin is now top 20 in terms of market cap.

What kind of currency is measured in market cap? Isn't that a chicken-egg problem?

How big is big?

The one that was linked as 14400 BTC. At $15k/BTC that's $216m.

On GDAX the trade volume for BTC/USD is now about $1.4 billion per day. And there are other exchanges too. So yes the market could fulfill that trade. Would probably bring the BTC price down but it wouldn't be catastrophic.

Given the completely unregulated nature of these exchanges, that 1.4 billion USD could be legit, it could be wash trades, it could be inhouse bots doing crazy things to rig the market. , or it could just be a complete fabrication.

Looking at the chart, you could liquidate $130 million and it would move the price down to $5200.

Is that based on the live buy orders? If the price dropped below 10k then I think you would see a ton of new buyers (including me)

You'd also see a lot of new sellers when peoples stop-losses trigger.

Catch a falling knife

Flash crash.

Bitcoin hit ATH on coinbase, then crashed. Something's not right. The exchange is still not really 'up'.


Can we change the link to https://twitter.com/coinbase/status/938802155925913600

That's a nice, detailed status page. Although, some would say that it's a bad sign such a page is needed.

Fair enough, I see what you mean. I take back my statement.

> some would say that it's a bad sign such a page is needed

Not me. Detailed status pages are awesome. Visibility into the service is awesome.

As somebody who doesn't have a background in finance, what are the legal obligations of an exchange to stay online? How can a user of the service be certain that an exchange is offline for technical or non-technical reasons? Let's imagine the likely scenario when the price suddenly drops like a rock and the exchange goes offline completely (for technical or non-technical reason). Could a user make a claim that Coinbase is liable for their losses because they were unable to access their coins?

A financial institution has fiduciary obligations to its clients to maintain the assets (here, the Bitcoin, ether, etc.) entrusted to them.

However, financial institutions are not currently obligated to maintain their clients' ability to transact those assets, let alone within a specific time-frame. Many people have sued their brokerages for not letting them sell stock in a timely fashion to take advantage of price changes; essentially all of them have lost or settled for pennies on the dollar.

In this case, it's completely up in the air - bitcoin isn't an official financial product, doesn't fall under any formal juristiction, etc. At best any gains get taxed now, but besides that, exchanges can't give any guarantees about uptime or e.g. getting your money back if they're not available.

I'm expecting this all to go wrong, up to the point that I'll just let my coins ride it all out. The first gamble is bitcoin, the second gamble is the exchanges.

It's a matter of time until Mt. Gox part 2 I think. It may not even take nefarious workings at the top, just sloppy work somewhere + one bad employee or outsider.

Is this going to end in a re-discovery of exactly why we have FDIC? Or the SEC? How do people even know how much bitcoin activity is real and not essentially faked?

Popular Google Searches by year, by my estimation.

2016: "what is bitcoin"

2017: "what is ethereum"

2018: "what is a wash trade"

2019: "can you eat squirrel"

"Why have I seen this comment before if it's only an hour old?"

Oh, you copied and pasted it from yesterday...

Yeah, but that story got flag-killed quick and I still thought this was funny.

(I actually originally tweeted it, like two weeks ago, but no one reads my twitter, which is to their credit.)

>> How do people even know how much bitcoin activity is real and not essentially faked?

Define fake

Ignore exchanges where there is 0 trading fees like the big one in Korea.

Why? Volume isn’t important, it’s order book depth that matters.

I do not think it would be difficult for someone to setup a website and scam newbies into buying 'bitcoin', handing over credit card info and taking wire transfers. Perhaps even act legit to start and then segue into only processing part of transactions as real. Quite a ponzi. I don't think is a stretch either. There isn't anything comparable that people would give you money for and the key really is the overwhelming interest and publicity combined with people who have no clue what they are buying. As long as (as with any ponzi) people don't withdraw more than they actually have this could go on for quite some time. Madoff had it much much harder.

You can't easily pull a scam like that using credit cards. Wire transfers could be potentially easier as we know the union transfers are hard to revert (but there's work being done there as well).

Essentially, if you interact with banks and have too many chargebacks or complaints, you'll be dropped. Or your fees will get massive. And that's if you find a payment processor who's happy with you taking money for cryptocurrencies in the first place.

True but if you are running a ponzi and you have enough money coming in then you can handle redemptions if you need to.

It's not about refunding the customers though. It's the payment gateway that will drop your as a customer. Neither them nor Visa/MasterCard want anyone running scams via them.

If I were running a scam, I'm pretty sure at the moment I would have a positive cash flow: a lot of people are transferring their USD/EUR into my bank account because they want to get on the Bitcoin train. If a few people are cashing out, I'm pretty sure I would be able to manage to give them their payouts. Exchanges also have a withdrawal limit of a few K's per day, in my ponzi scheme if someone wanted to withdraw 100K it would take maybe 10 days, and one hopes there are enough new suckers that each day I'm able to pay out, out of my official bank account, my slush fund. The rest of the money can be transferred out where no one can touch them...

And when the whole thing blows up, I disappear. My payment gateway will drop me, but what do I care?

Nah, you're thinking about it the wrong way. Government regulation is pointless overhead! </s>

Last week, their rates were about 4 BTC to rent 1 TeraHash/second of mining capacity for 24 hours. Today, according to a mining calculator[0], that works out to about $2.62 in BTC per day.

[0] https://www.cryptocompare.com/mining/calculator/btc?HashingP....

You really shouldn't be surprised, at this point, by Coinbase going down during times of high activity. It happened on November 29th (when, in fairness, everyone was going down), and happened four times last June, which should have been their canary-in-a-coal mine.

If this phase of Bitcoin is likened to each goldrush around the world, isn't it fairly inevitable that there will be hiccups? Robberies, banged together shopfronts, people using any tools they can find, etc. And the internet has long been a bit of a Wild West already.

Assuming they're adding thousands and thousands of customers quickly, I'm not surprised that their systems and support are lagging. Comes with the (Wild West) territory.

I do wonder if after the Ether "flash crash" recently (https://blog.gdax.com/eth-usd-trading-update-5d8142b5bdc1), if Coinbase / GDAX instituted some sort of "circuit breakers" on their exchange to prevent this type of thing from happening again. Bitcoin and the other cryptos, trades at a pretty low volume relative to their market cap, so if a lot of people we're trying to dump large orders this morning to take a profit at the new ATH, I could see that causing similar issues that they may have been trying to prevent.

Circuit breakers do not matter if other exchanges continue to trade the same currency. It would just cause the exchanges that do the circuit breaker to possibly hurt their customers.

The only effective circuit breaker is some form of synchronized method, but will the different exchanges cooperate? They really should put in circuit breakers this week across all exchanges so that when really bad stuff the panic doesn't destroy tons of people. It will be bad, but at least if the trades can complete and things unwind in a moderately slow fashion, it won't just 100,000 attempted transactions per second with random successes.

Maybe not, I'm just speculating that GDAX or coinbase may have added this as some sort of "sanity-check" in their system (regardless of the insane volatility in crypto right now!)

In the situation with ETH, a bunch of automated margin calls happened because of one large order on the books, dumping the price down to 10 cents! GDAX/Coinbase repaid a lot of people who lost money then, even though they didnt have to. They may now have some dumb heuristics that slow down trades / whatever if a bunch of large orders are going through, or price delta over a short time period goes too high. I remember looking at the charts today on GDAX and the 5 minute candle went from 20k open to 16k close before trading halted! I also heard from others that there were a lot of ~19k sells of 100 coins each on the books and executing.

So yeah, just speculating that this may bave been the case of someones boss at GDAX/Coinbase telling engineering "please don't make us do that again", after they ate the cost last time! :-)

> They really should put in circuit breakers this week across all exchanges so that when really bad stuff the panic doesn't destroy tons of people.

"Decentralized! Deregulated! But coordinate to protect us from ourselves, don't let us lose money!"

Haha, yeah I can't see the BTC crowd all being on-board with this.

That said, while I don't think the exchanges need to (or realistically could) coordinate on circuit breakers, I do think it's entirely reasonable for there to explore this type of mechanism on a per-exchange basis in the case of GDAX and other exchanges where margin is allowed, to prevent the whole "cascading margin call" thing that happened with ETH to happen again. That could take a lot of forms, including limiting the size of market orders to not eat 30% of the book, or not allowing 1 market order to trigger margin calls if (insert criteria here)

(That said, they could also just say "that's how margin and trading goes on low-volume, highly volatile assets" and technically be correct, but much like their decision to reimburse some traders, this is more of a business decision and wanting to keep your customers. Not sure what the "right" answer is.)

This is what I thought as well but haven't been able to articulate yet! It was actually on the opposite side -- a flash rise. I watched this happen and the order book on the sell side was getting extremely thin. The price rose thousands in minutes. I managed an 18k limit sell and bought back in at 16.5k

I wouldn't want to be in a business in which the worlds absolute finest hackers and crackers - the elite of the best of the best - make it their full time job - forever without end - to get into your systems and steal everything of value.

That sounds like any bank. Computer systems as the source of truth for billions of dollars of assets is not new.

Banks have the advantage that if they see a huge spike in their outbound inter-bank transfers they can suspend traffic while they investigate it, and then if necessary call up the recieving bank(s) and issue a freeze on those funds. There's already established relationships and agreements for these things.

With a cryptocurrency there's nobody to call, you have to convince 50% + 1 of the mining pool to undo any major breach - which is almost certainly never going to happen.

I think bitcoin attracts more hacker attention than banks.

I'm curious what the implications are that all these exchanges go offline whenever there is a huge spike in bitcoin.

That being said, anyone care to elaborate how the price of bitcoin is determined? I mean it's easy when there is a large central bank monitoring this figure. However, how exactly does it work for bitcoin? What makes it worth the $15K value when there is no "goods" to back it up. Are there anyone here buying bitcoin as an investment?

As others said, its determined by what people will pay for it. If you go to an exchange like GDAX (or any stock exchange), you can look at what is called the "order book". If you are a trader, you can act as either a:

- "Market Maker" - By placing an entry in the order book, which says something like: "I have up to 100 Bitcoin I'm willing to sell at $15,000.01 a piece", or "I'm willing to buy up to 100 bitcoin at 14,999.99 a piece".

- "Market Taker" - You can do this by looking at the book and saying: "I will take that deal". You can also do a "market order", saying: "I want to buy 100 BTC at market price", in which case, the exchange will match you with as many sellers as it takes to fulfill your order of 100 BTC, working from the lowest offered price on up.

The price is determined by either whatever the last trade executed at, or a weighted average of the buy side and sell side of the order book. If you hear someone talking about the spread, that is the difference between the highest buy price offered on the book, and the lowest sell price offered on the book. (2 cents in my above fictitious example).

The exchanges make their money by running the order book, matching makers and takers, and charging a small fee for the transaction. (On GDAX for example, market makers get charged no fee, market takers get charged some fraction of a %). The exchanges make money on the volume of trades going through them.

It can get a bit more complicated, but I hope that gives you an idea!

Is there a term for those who take advantage of the spread to make their money, essentially placing buy orders and then turning around and placing sell orders? Seems like there must be people making money off those who only "Take" deals and never "Make" them.

Those are just called "market makers". As a market maker you might be exposed to the risk of always ending up on the wrong side of the market over a longer time. For example if BTC is generally moving up, market makers might eventually have sold all their BTC and end up with USD. Sophisticated market makers will thus try to maintain a neutral position to avoid that risk. Some exchanges will even offer market makers a fee rebate to encourage parties to become market makers.

If they're just doing so by essentially flipping between bid and ask prices they may also be called scalpers.


That is what a Market Maker is.

Exactly. GDAX and other exchanges charge less or no fees to the market makers to increase liquidity in the market. (Basically lessen the spread and increase volume on the order book).

From the exhanges POV this is useful because more orders = more money for then.

From the market traders POV (the taker) this is useful because a lower spread and more volume means there is less chance of buying something they can't later sell.

For the market maker, they make their money on volatility (aka the movement if the price). If they think the price is going up, they will offer to buy at a price higher than the current highest buy offer. This reduces the spread and increases the odds of someone taking their offer. They will then later sell those coins, and make a small profit on the movement.

If they think the price is going down, they will place a sell order cheaper than the lowest sell, and then buy back later, making small profit whole keeping the same amount of coin / stock / whatever.

Basically a "market-making" trader is less in it for long-term, and more interested in making money on the movement.

Aside: You may have also heard of "short selling". This is when a market maker is betting the price is going to go down, but doesn't have anything to sell. In a short sale, they will borrow the stock/coin from someone else (such as the exhange or their broker), and sell it, promising to buy it back later to repay the loan of said stock / coin. Easy way to bet on perfornance of an asset you don't owb, but dangerous since if you are wrong and price goes up, the person who loaned it to you will want you to repay it back sooner rather than later, costing you money!

What makes it worth 15k is that people will pay that much for it. The price is determined by buyers and sellers, trading with each other on markets called exchanges.

I get the "market" determines the price but I guess I'm confused with who is agreeing to the value? Is there a small amount of big players that are playing with the price? Or is there a general consensus that the price is $15K. And wouldn't there have to be people that we willing to pay this price? What prevents the exchanges from colluding and raising the price artificially?

Overly simplified explanation: You have to have a buyer willing to purchase at that price. Lets say the exchanges set the price at 100k when 3 minutes ago it was 15k. No one would buy.

A lot of the market movement is caused by people setting fixed prices on their transactions. Eg: Sell 5 BTC at X Or buy 5 BTC at Y. If X or Y is never reached the transaction never takes place.

Generally the exchanges just connect a buyer to a seller. You are not buying the exchanges coins but rather the coins of another individual/trader. The exchanges are just middle men facilitating the sale.

The exchanges make money on a per transaction basis on fees. If the market grinds to a halt because there are no transactions the exchanges make no money.

Now there are many possible reasons for the recent spike, some malignant such as potential pump n dump scheme or artificial inflation due to the upcoming futures exchanges and the desire to profit from shorting and some benign such as all the media attention and fear of missing out or just genuine belief in the potential of BTC as a currency or asset.

No idea which reason is accurate, could be all of them or none.

Hey, I hope you saw my above price about how the market works.

So to give you a little more info: 1) Yes, there are big players (aka "whales"), and the rest of the "small fish", all playing in the same pond. The interesting thing about cryptos is that the are more "money-like" than "stock-like" when it comes to trading. Meaning just like money, where I can buy a fraction of a $, I can also buy a fraction of a bitcoin. So there are a lot of smaller players who will buy, say .01 BTC for $150 right now.

But while there is a general consensus that bitcoin is $15k by virtue of people paying that much for it, the "whales" (people with a lot of money or coins), can cause issues because they can put a lot of volume on or take a lot of volume off the order book at once. This happened to Ether this year, dumping the price for a few minutes from >$300 down to about 10 cents for a few minutes! https://blog.gdax.com/eth-usd-trading-update-5d8142b5bdc1

In fact, there is some speculation that a lot of the rise right now is from Wall St. / big firms buying up a lot of coin, as futures trading on Bitcoin is coming in the coming weeks! (But I don't know enough to comment on that theory)

As far as preventing exchanges themselves from colluding... it would be difficult but not impossible, because again the price on the exchanges comes from the traders, not the exchanges themselves. (Theoretically it is possible that an exchange could do a bunch of "wash trading" / make up a bunch of fake trade volume to make demand look higher than it was, but given that there are a number of exchanges in a number of different countries, coordination there might be difficult)

>Is there a small amount of big players that are playing with the price?

We don't really know.

>Or is there a general consensus that the price is $15K. And wouldn't there have to be people that we willing to pay this price? What prevents the exchanges from colluding and raising the price artificially?

Well the exchanges could be running buy-bots but I think the more likely collusion is going to be big-players selling to themselves, if a big player is filling up both sides of the order book they can make it look like there is a lot of activity and drive the price up that way. Now is this that or is it organic? I'm not sure we have a good way to tell.

Naïve question here: since wallet addresses are fixed, couldn't you detect transaction activity unusually repetitive between a few wallets? Couldn't some clever learning algo detect anomalies such as what you hypothesized could be going on?

Probably. Could be an interesting data science project. One thing I'm not sure about is how easy it is to make new wallet addresses. Do you know?

A Bitcoin address is just the hash of a public key, so generating new addresses is extremely cheap (on the order of microseconds) and doesn't require interacting with the blockchain at all.

People place orders saying they will buy it at x price and these are matched with sellers. I recommend checking out poloniex to understand how a crypto exchange works. Coinbase isn't actually an exchange rather they have to buy Bitcoin from elsewhere and then sell it directly to you and that's part of why the price on coinbse is always higher than on real exchanges

This is true. It also helps that their is a finite supply, so if demand remains constant, the value theoretically should rise over time.

Look at it this way: these services are like banks, but their traffic is more like a news site. It's literally the most challenging aspects of both worlds.

simplistically, the exchange manages an "order book", which records the amount each person wants to buy or sell, the price at which they are willing to trade, and the time the order is received. if there is enough "buy" volume at a given price to fill a given "sell" volume, then a trade happens. the price of the latest trade is the "market" price. this happens for every order that has a volume and price matched "other side" of the trade.

there are many orders that are above or below the current market price. those wont get cleared until the trades that are closer to market clear, and they may not get cleared at all if the market moves the other direction

Like any traded asset it's just based on the current buy and sell orders, same as with the stock market

What's the likelihood here that someone is in the middle of running off into the sunset with a giant ɃɃɃ bag?

Pretty unlikely for coinbase. They always go down during high traffic periods. Plus their principal owners are well known US citizens/residents.

As always with Bitcoin, I'd say 50/50.

Then the market loses faith in the currency and that bag is suddenly a lot smaller, perhaps gone.

Considering that it has appreciated by 15% in one hour, not really a good sign

This is another one of those moments in cryptocurrency where time seems to freeze and everybody holds their breath.

In regards to the volatility? sure.

But coinbase goes down for maintenance all the time

When CB went down for a bit today I had the recent news of NiceHash in my mind. Sure was a breath holding moment.

Generally speaking. For example, Nicehash, which had been chugging along for a long time, went belly up yesterday after what seemed like a regular maintenance period.

how many things can go wrong when the service is overloaded... looks like Coinbase doesn't know what stress testing or monitoring is.

* ID verification by either webcam or id pic upload fails, and eventually one reaches the daily quota, locking the feature for 24 hours (seems fair though)

* removing and re-adding bank accounts fails, the site says the account is already linked and it shows some ghost duplicates (i.e. the same bank account multiple times with different statuses)

* bank account verification fails, the site says the small amounts deposited don't match

* some buying operations fails due to API timeouts when collecting money from linked bank accounts

How can anyone reliably trade Bitcoin? With Coinbase immensely unreliable, I can't see how anyone would tolerate running algos on their exchange, given they could be in a bad position unexpectedly.

What are the other exchanges that you can trade BTC and are they more reliable than Coinbase?


Serious question, would it ever be possible to implement circuit breakers like the stock market does for Bitcoin? Without safeguards, and the nature of 24/7/365 trading of Bitcoin, this all feels like teetering on the brink of catastrophe.

Not really. Unlike stocks and shares which are traded on regulated exchanges, bitcoins can be traded anywhere on any terms.

If one exchange implemented circuit breakers, people would simply move their trading elsewhere.

I never want to reuse my comments from other thread but this time I just can't resist. No one really anticipated what we all are currently experiencing and Coinbase is not different.

It seems Coinbase struggling to cope up:

$0000 - $1000: 1789 days

$1000- $2000: 1271 days

$2000- $3000: 23 days

$3000- $4000: 62 days

$4000- $5000: 61 days

$5000- $6000: 8 days

$6000- $7000: 13 days

$7000- $8000: 14 days

$8000- $9000: 9 days

$9000-$10000: 2 days

$10000-$11000: 1 day

$11000-$12000: 6 days

$12000-$13000: 17 hours

$13000-$14000: 4 hours

$14000-$15000: 10 hours

$15000-$16000: 5 hours

$16000-$17000: 2 hours

$17000-$18000: 10 minutes

$18000-$19000: 3 minutes

It would make more sense to put this on a logarithmic scale. The move from $18000-$19000 is nowhere near as meaningful as the move from $0 to $1000.

Also, the move to $19,000 seems to have been some kind of liquidity issue on GDAX, the price did not get that high on other exchanges.

I agree, so putting this together from the parent's info:

$0000 - $1000: 1789 days

$1000- $2000: 1271 days

$2000- $4000: 85 days

$4000- $8000: 96 days

$8000- $19000: 20 days

Edit: Even on a logarithmic scale, bitcoin is experiencing exponential growth.

Thanks, this is a really interesting way of looking at it. The change from $0-1000 is still not comparable to the other 2x gains. I looked up $500-$1000 and it looks like it took about 1000 days.


That being said, does price matter or does volume matter, in terms of Coinbase? Their infrastructure should be indifferent to price. I understand there's a correlation there, but daily trading volume would be the ultimate determinant of Coinbase's performance.

This unfortunately reminds me too much of stellar evolution timetable[1]:

> Star burns through a succession of nuclear fusion fuels:

    Hydrogen burning: 10 Myr
    Helium burning: 1 Myr
    Carbon burning: 1000 years
    Neon burning: ~10 years
    Oxygen burning: ~1 year
    Silicon burning: ~1 day
(...and then it goes supernova.)

I hope Bitcoin's end is not as violent. :/

[1] http://www.astronomy.ohio-state.edu/~pogge/Ast162/Unit3/supe...

While I agree with you that the amount of users probably raised a lot to the point were servers are stressed out, there's something slightly off with the data you present: the latest entries are actually smaller percentage moves.

Here is the same data with (approximated) percentage moves:

$0000 - $1000: 1789 days

+100% ($2000): 1271 days

+50% ($3000): 23 days

+33% ($4000): 62 days

+25% ($5000): 61 days

+20% ($6000): 8 days

+16% ($7000): 13 days

+14% ($8000): 14 days

+12.5% ($9000): 9 days

+11% ($10000): 2 days

+10% ($11000): 1 day

+9% ($12000): 6 days

+8% ($13000): 17 hours

+7.5% ($14000): 4 hours

+7% ($15000): 10 hours

+6.5% ($16000): 5 hours

+6.25% ($17000): 2 hours

+5.8% ($18000): 10 minutes

+5.5% ($19000): 3 minutes

Anyway, it's more the volume that is of importance here than the price when it happens.

How much money needed to come into the bitcoin market for it bump from 15k to 19k? Where is this money coming from??

You don't need money "coming into" to increase value of Bitcoin.

All you need is back and worth trades in th market. It can even be market manipulation where some buy and sell between themselves.

Bitcoin has massive market liquidity problem. Once people want to sell, the price drops like a stone. This is why it would be convent to have "technical problems" or do DDoS attack to prevent sells and allow markets to calm.

Had to be a lot. Should have screen recorded it, but instead recorded with my phone. The order book entries and chart were going nuts. Entire sections of the book disappearing at once. The chart would go almost entirely flat several times.

Almost always you see minor movement on the order book except maybe sometimes at the actual mid point it may update a bit. The chart is almost always balanced until the past few weeks it's be HEAVILY weighed towards the sell side.

71K+ BTC had been already traded by the time it hit high 19k.

I saw the same thing. It was insane

Technically, not much. Let's say people stopped trading except for two people that made a series of trades of .0000001 btc at $100k, then the price would be $100k.

Not much. 1% of asset allocation for an institutional investor is enough for that.

EDIT: You currently need $70m to bump the market above 20k on GDAX. That's a lot for retail investors but not much if institutional investors get involved.

In the grand scheme of things sure, but definitely a large influx as compared to the size of the market in Coinbase.

We're talking probably low 2-digit millions to make big moves I think.

Yep, around $70m to get it above $20k.

What institutional investor is going to put 1% into Bitcoin?

How do you determine the $70m?

You look at the "Depth Chart" -- it shows you how much people are selling (or buying if you go to the right) in a way that escalates as you get further away from the mid-market price that is currently executing.

"How much money needed to come into the bitcoin market for it bump from 15k to 19k?"

Almost none.

If there are just a few transactions at a price, well, we assume that is the worth of all bitcoins in existence.

Which is a crazy assumption, but hey.

right now, as i'm looking on gdax, the price is 16.3k/btc, and it would take 42m to fulfill all of the sell orders to push it up to 19k. that's not a lot (but it's only one exchange). i think <100m to swing the market by that much across all of the exchanges.

New money is coming in that's for sure. But a limited resource and a lack of sellers will also push the price.

Tangent: it would be cool to have a browser extension that could notice that this data could be presented visually and offer to show you e.g. the scatter plot.

Can you describe what this data is you're sharing? As is it's not clear what phenomenon you're describing.


The insane rise in the valuation of Bitcoin. How many times we have seen a currency or an asset that represents this kind of momentum? Clearly no one has seen this coming (that soon).

Well, there was the tulip bubble of 1637.

I think its the amount of time taken for the price of Bitcoin to rise from the first amount to the second.

>No one really anticipated what we all are currently experiencing

Actually, those in the Bitcoin economic space have been saying this was going to happen for ~3 years

Hyperbitcoinization - Daniel Krawisz - March 29, 2014 http://nakamotoinstitute.org/mempool/hyperbitcoinization/

The article you've linked to does not address the situation being described.

How many 9s do you think Coinbase has?

2 9s, or 99% uptime? Or is it worse than that?

They are rarely completely offline. But, they've been experiencing all sorts of "minor outages" and partial service degradation. And when you can't trade or transfer between wallets, the fact that their portal is "up" doesn't matter much.

During normal times? Uptime is maybe 95-99% as they're down for maintenance for a few hours every few days, usually during off hours.

During major price swings? About 10% uptime at best. They're almost always down when the price jumps or drops.

For how powerful we announce bitcoin/blockchain and its possibilities, the faultiness lies in the exchanges and that is my biggest con with Bitcoin

Bitstamp.net was also unavailable for a while. Weird coincidence.

Not really... websites regularly go down when they are under a load they cannot handle

Coinbase doesn't support Tether so it should be okay.

Maintenance xor DDOS?

it's back up; but not working

Strange, the price data from coinbase never stopped working on tradingview.com. I wonder what was affecting the price changes when the order book was down?

The books are on https://www.gdax.com/, Coinbase's exchange intended for big trades. Coinbase is more of a consumer retail site. I assume the latter is what is going down, but it's hard to tell as they both seem fine at the moment.

As of now it is back up. If you take a look at the GDAX or Coinbase charts over the past 1-2 hours on a short timescale though (i.e. 1 || 5 minute candles on GDAX, or Past hour on coinbase app), you'll notice after the big dump, there was a large flat period where no trades were occuring, even though the site and app appeared to be up for a lot of people.

All of this is anecdotal:

These past couple of days, whenever a coin soars a ridiculous amount, on any given exchange, that exchange seems to mysteriously go down. For example: Binance and Bitfinex were down for "maintenance" (or just plain down), while IOTA was experiencing a surge in price (from ~1.10 to 6$). Similarly this happens with Bitcoin today (it seems Quadriga had some issues and Coinbase; not sure about others).

Something is up, and it's weird.

My conspiracy theory: big money institutions are testing pump and dumps for when short contracts come into play. On the one hand, the cryptocurrency world wants to remain as unregulated as possible; on the other, a lot of individuals are going to get reamed by big whales.

Isn't it just the simple explanation that big swings in the market bring insane traffic spikes to market websites?

Add to the fact that people are also DDOS websites (doesn't take a "big money institution" to do that) and it seems the explanation is clear...

Yea. I work as a software engineer at a company that sees huge swings in traffic on our product (by nature of the product). It's not uncommon for us to see 20x swings in traffic day-to-day. It's taken us a long time to be able to handle that and even then shit still goes wrong.

I'm much more inclined to believe the downtime is caused by scaling issues rather than some big conspiracy,

EDIT: Just to give some insight, one of the problems we run into is that usually only one issue is exposed during an outage. For example, lets say we have 10 functions (or services or whatever) in a row that all call each other. All the odd numbered ones have issues where they fail at scale. During the first downtime, function one will fail and start to throw errors, but this means the full traffic load isn't reaching the rest of the functions. By the time the problem is solved, traffic has usually died down, so function 1 is fixed, but 3,5,7,9 still have issues that haven't been exposed.

Additionally, at a quickly growing company like Coinbase, the code isn't going to stay the same for very long, so even once those functions are fixed, new issues are introduced.

It's easy to say they should be load testing at 10x their expected load to prevent these type of issues, but it's really hard to replicate production traffic. Users do weird and unexpected shit that can cause problems.

There's an article on the frontpage of HN from NYTimes that mentions Coinbase has struggled with the massive spike in traffic recently.

It's funny how quickly people jump to overly-complex or borderline conspiratorial explanations when the obvious is staring us in the face...

Coinbase might be a large well funded organization but it's still run by developers with giant backlogs of tickets and conflicting priorities to build new stuff vs maintain old stuff.

> but it's really hard to replicate production traffic

This times 1,000.

Honestly, I don't think these exchanges can handle a lot of traffic. By a lot, I mean a few hundred trades a second. It is odd to me.

I have entertained using cryptocurrency for game betting at some point, while running some typical web stack. When I started looking into security and scaling, I realized there was a expensive and complicated overhead. When security is of very high concern, traditional performance seems to really suffer.

They are very well funded, and they can afford scalability. It's not that hard, given the number of transactions even at peak so far.

So yeah, I find these weird circuit breaker patterns suspicious. The worst part is "partial" degradation, when you can't execute trades, but someone else can. It's very easy to hide all sorts of irregularities behind these partial outages.

> It's not that hard

I'm sure they would pay you a lot of money to solve their scaling issues, if you were so inclined.

and I would do it.

Not really. While most exchanges went down during the Nov 29th rally, most others didn't during today's, nor did they during the four June rallies where Coinbase also went down.

I think you can take off your tinfoil hat. Lots of price movement motivates more traffic to the exchanges, resulting in DDoS.

As the bitcoin market grows GDAX, Bitfinex, etc are probably going to have to borrow some ideas/talent from NASDAQ & other big electronic exchanges to stabilize things.

Tinfoil hat off!

Shouldn't an exchange be able to handle volume? Seems like a basic requirement.

Regardless, I still believe that with the introduction of shorts, were going to see bigger swings.

> Shouldn't an exchange be able to handle volume? Seems like a basic requirement.

Exchanges have incentive to handle as much volume as possible to make as much via fees as possible, but no site can handle infinite volume. The coinbase app has been climbing the iTunes charts into the top 20 the last couple days. Just before the site went down the price was running up by hundreds of dollars per minute, attracting tons of trader traffic trying to profit from it. Even if they have basic scaling measures in place for the basic infrastructure, they most likely experienced unprecedented exponentially increasing volume that was inevitably bound to break something

Judging by the GDAX runbook, they barely have a couple of orders per second. They can't have performance problem at this scale.

I've seen exchanges running 10 or 100 times that amount with no problem, ran by only a couple of graduates out of school.

Either way, doesn't matter. Exchanges have no performance problem, it's a big queue, you just queue requests.

In my capacity as an infrastructure/scaling consultant I've spoken to multiple exchanges. Without disclosing anything I'm prohibited from, my strong expectation is that almost every exchange can't handle traffic spikes.

They're not all shitty me-too Rails apps built by junior engineers who figured "how hard can it be?". But most of the ones I've seen are.

> Shouldn't an exchange be able to handle volume? Seems like a basic requirement.

Exchanges are just websites like any other. They're build to handle a certain amount of traffic based on the average amount plus a little extra to handle spikes.

If they only get (And I'm pulling numbers out of my ass here just to make a point) an average of 10 trades per second with occasional spikes to 100 trades per second, how much infrastructure should they build? What capacity should they build to? 1,000 trades per second? 10,000?

There's a cost analysis involved. I think it's a bit unrealistic for them to have enough infrastructure to handle spikes an order of magnitude higher than their typical peak loads.

Clearly it isn't a basic requirement because they exist without being able to handle volume.

I agree, they _should_ be able to, but lots of times performance is a 2nd priority behind getting shit out the door - for better or or for worse.

I'm sure as they grow they'll harden their systems and/or do a total rewrite. In fact I wouldn't be surprised if they're doing it already since the big boys are starting to show interest with futures.

> handle volume

But what levels of volume? 100 quadrillion trades per nanosecond? Sure they could scale their infrastructure to those levels, but why? You overprovision, get estimate your worst-case scenario and work towards that. But in the end, if there is a huge rush, like a 10000% increase or more, even the largest exchanges would break down.

US stock exchanges don't run websites on which you can trade. However, during the flash crash in 2015 I was unable to place trades on either of the two brokerage sites I used at the time (USAA and Vanguard), probably due to other people rushing in to trade like myself.

I imagine they've spent a lot of time already beefing up their infrastructure, you can anticipate high loads but until you know what that looks/feels like, you cant compensate for it. They'll get there eventually. It'll just take some time.

No, most stock exchanges aren't exposed to the public like bitcoin exchanges are.

No need for conspiracy theories.

These services are like banks, but the traffic they handle is more like a news site, due to the extreme volatility of cryptocurrencies. Super-security combined with extremely large traffic fluctuations.

Whoever is on PagerDuty at Coinbase these days - I feel for ya.

Coinbase runs on Ruby and their devops aren't good enough to make up for it.

> big money institutions are testing pump and dumps for when short contracts come into play

Big-money institutions won't do this. Random individuals taking advantage of the fact that lots of value rides on relatively little volume? Much more likely. Combined with the fact that the exchanges, intentionally or unintentionally, lock liquidity when volatility spikes and it's 1907 all over again.

24 Hour Volume is north of 17 Billion.


Given the incentives involved (and history of exchanges, everywhere, before regulation) a good fraction of that is wash trading. (A better measure for the risk at hand is the effect of small out-of-market bids and offers on the overall price, i.e. how much it costs to move the price $1.)

[1] https://www.investopedia.com/terms/w/washtrading.asp

Disclaimer: This is not securities advice. Do not buy or sell anything based on this Internet comment.

On the 10th bitcoin will be listed on the chicaco exchange. This is the reason for the price swing lately.

Can you explain? You could also argue that when futures are listed on the 10th, there will finally be an easy way to go short on Bitcoin. For instance when you want to hedge a long position, or simply because you want to speculate on Bitcoin going down.

So the listing of Bitcoin futures on the Chicago exchange might not be a good thing for the Bitcoin price at all.

The 10th is a Sunday

The exchange opens Sunday evening.

No, No, No! Conspiracy theories attract attention because they make the world seem more legible, when it's not. In this case, exchanges are not being attacked by shadowy corporations for short contracts.

Attention from price surges increases visitor traffic, and exchanges are understaffed or unprepared for the 10-100x increase in traffic volume. There's no need for complicated stories, evil corporations, or any of that nonsense. Stick with straightforward cause and effect.

No, No, No! Don't pay attention to that thing! Look over there!

Seriously though, it's worth considering that this kind of thing could be possible even if we don't know whether or not it has anything to do with these specific outages.

No, it's not. There are an infinite number of conspiracy theories. There is no value in considering them.

You are coming off as very emotional/desperate in your attempt to convince people to act cool and logical. It's not very persuasive, and if I could downvote you I would.

so it got hacked

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact