
Robinhood Is Down Again? - jaredwiener
https://status.robinhood.com/incidents/h6xsxvsxg7rm
======
rococode
If this is a leap year problem, apparently similar issues happened 4 years ago
and they never fixed it...

Thread from exactly 4 years ago, March 2 2016:
[https://old.reddit.com/r/RobinHood/comments/48mep4/robinhood...](https://old.reddit.com/r/RobinHood/comments/48mep4/robinhood_not_working/)

Thread that pointed it out:
[https://old.reddit.com/r/wallstreetbets/comments/fcoaev/and_...](https://old.reddit.com/r/wallstreetbets/comments/fcoaev/and_now_a_brief_flashback_to_four_years_ago_where/)

~~~
ajphdiv
Wow. That is negligence, plain as day.

~~~
ketamine__
Except it's not:
[https://twitter.com/AskRobinhood/status/1234861941413351434](https://twitter.com/AskRobinhood/status/1234861941413351434)

------
kilroy123
As a fellow software engineer, I feel for the people who work there.

As a customer who lost a decent chunk of money yesterday (and today) because
their system was down, I'm angry.

I hate to say this, but I do not think they're competent enough to continue to
operate as a broker. I, for one, will move all my money off as soon as I can.
I'm also going to file a complaint with finra[1].

[1] [https://www.finra.org/investors/have-problem/file-
complaint/...](https://www.finra.org/investors/have-problem/file-
complaint/complaint-center)

~~~
ajphdiv
Right there with you. I only use RH for options -- so thankfully not a lot of
money in there for me, but I'll be switching everything over to tastyworks.

~~~
ishwarn
What's nice about Tastyworks? Never heard of it before

~~~
ajphdiv
It has a nice GUI for different option strategies. Great website and mobile.
Fairly cheap commissions too. If you are just buying/holding shares probably
not the right platform.

------
bob1029
I've been defending Robinhood since the beginning but this is just too much. I
will start looking at new brokers this afternoon.

At this point for me its not so much about the lost opportunity as it is about
what the hell actually exploded so hard that they need to be down for 2
consecutive days. From my perspective, it's looking less and less like some
one-off technical or hardware screwup and more like a fundamental limitation
of their core architecture.

~~~
rriepe
I haven't looked into the technical credibility of this at all [EDIT: I should
have more strongly indicated my doubt here], so this isn't any sort of
condemnation on my end, but I thought the Leap Day theory was interesting:

[https://twitter.com/jtech63/status/1234600045787394048](https://twitter.com/jtech63/status/1234600045787394048)

~~~
Humdeee
There was a discussion last night about this. Some of the more technically
competent posters dismissed it because it's unbelievable that a financial
platform would roll their own date-time implementation.

~~~
notyourday
That means people should reevaluate the weight of the opinions of those
technically competent posters - we have a screen shot that robinhood did roll
their own implementation because yesterday was March 2nd and its app was
requesting March 3rd.

That at least means that some portion of their stack used roll your own
datetime library. That would not actually be a problem as long as the entire
stack got the same library or the same rules for datetime. The problem is of
course that it probably did not so the libraries were not bug compatible which
of course caused errors and those errors need to be handled, preferably every
fast.

Error load and error handling is the least tested part of every system because
it can only be properly tested in production and, unless your company embraces
the Chaos Monkey approach to testing, the C-level would have a heart attack
when anyone proposes doing it.

~~~
vsareto
That particular API endpoint is just returning market open times, in which a
request for tomorrow might be perfectly reasonable.

[https://api.robinhood.com/markets/XASE/hours/2020-03-03/](https://api.robinhood.com/markets/XASE/hours/2020-03-03/)

vs

[https://api.robinhood.com/markets/XASE/hours/2020-03-07/](https://api.robinhood.com/markets/XASE/hours/2020-03-07/)

This whole discussion lacks context to the nature of the requests, aka a front
end code review.

~~~
notyourday
The issue is going to end up being related to some sort of date being injected
by the front-end and propagating that incorrect date into the infrastructure.

Somewhere within the infrastructure there's going to be an assertion such as

    
    
      if (max_drift_delta > delta(order_live_date,system_live_date) {
        # oh crap, something is completely broken in our system where did this come from?
        blow_up("terrible things are happening! how did we get this order")
      }
    

which is an excellent and correct catcher for "terrible things are happening"
since those things should never happen. That blow_up() code path is likely to
be very expensive which kills performance of the system, which in turn means
that it no longer can handle the load.

And since RH has lots of people who use apps, it is not that they can just
push an immediate bugfix.

------
eden_hazard
I'm a QA engineer and this is freaking infuriating. I just went on Linkedin
and 0 QA engineers. Just some QA associates that have no tech backgrounds. And
one QA Engineer opening. I wonder how long it's been open? How does a trading
platform not have a big team of QA engineers? Some dev managers, devs are so
freaking arrogant. They think since they aren't coding, they aren't smart and
they aren't important.

This isn't the first time that it's happened. Robhinhood should have been
prepared. This is unforgiveable.

~~~
btown
As someone who also needs to explain the importance and nuance of QA
engineering to my startup partners, do you have any go-to resources you use to
describe/sell the position? Genuinely want to expand my ability to speak to
the role.

~~~
eden_hazard
Point them to situations like this. Maybe small e-commerce sites can get away
with it but financial and health tech companies should always double, triple,
check everything. Move fast and break things is fine for new startups but once
you have a significant amount of users, the reliability of the product should
be the main source of pride for the company.

I know not every company needs it. I was the second QA engineer at a food-tech
startup in Boston 5 years back. QA got laid off after 10 months but they are
still successful without QA right now. They hire great devs and have a great
process. At the end of the day, if the site stops working, people will just
get some late food deliveries. One the other hand, I've a friend working at
Ford in Michigan and he says they have no code. You have hundreds of engineers
and everyone manually tests their own code. He says sometimes he will test for
a couple of minutes and just push his code. I don't blame him. Devs are great
at writing code. Let them write code but have someone else test their code
please.

I've a small options position open on Robhinhood. I'm not too worried about it
but I imagine some people have tens and hundreds of thousands. I've read some
rumors that this outage was caused by the leap year timing and it happened 4
years ago too. This is just unacceptable.

------
nate_meurer
Regarding getting some money back from Robinhood, here's an interesting
comment I found at Reddit:

> To those of you who deposited funds to Robinhood and are now down your
> deposit or more, these are technically ACH transfers, and if you genuinely
> believe Robinhood defrauded you, or otherwise failed to provide service as
> understood, you can issue an ACH chargeback through your bank’s web portal
> or by phone for any funds deposited in the last 60 days.

> RH may try to call you into arbitration, but more likely they’ll just ban
> you from the platform. It’s not a terrible idea if you plan on leaving the
> platform anyway. Just make sure you can substantiate your losses or they
> might have a fraud case.

> [If RH dings your credit], you‘re able to dispute with the credit reporting
> agencies or, if the agencies won’t delete, hire a debt lawyer to resolve. I
> doubt they would even if able. Any representation they have is likely about
> to be drowning in FINRA and arbitration prep, some people keep obscene
> amounts of money on their platform. They also likely can’t furnish any
> signed documentation of credit pending payment unless you signed up for
> margin trading.

> If the funds haven’t been deducted from your bank account or are in a
> pending state, you may also be able to just issue a stop payment. Again,
> anyone who tries this should ensure that they can substantiate losses
> related to the payment they stop or chargeback.

------
christophilus
It's a shame. I was rooting for Robinhood, since they really did disrupt the
brokerage industry. I suspect two days in a row of this will be a death knell
for them.

~~~
swiley
I don’t actively trade stocks but I’ve looked at it. What exactly does
robinhood do that etrade didn’t? (Other than the shiny app and front running
being a formal part of the business model.)

~~~
christophilus
As I mentioned, they were the driving force behind the industry moving to
zero-commissions. Also, I do really like their app compared to Fidelity,
Schwab, Vanguard, and e-trade. But I'm probably a Fidelity customer for life
at this point.

------
ceohockey60
Since it's Day 2, def appears to be an architectural scalability issue, not
some bugs. Quick scan of their engineering blog shows the following stack:

-Zookeeper, Kafka, Spark, Airflow, Faust (Kafka Streams in Python) -Kubernetes (was/is SALT + Terraform) -AWS Aurora, PG, ES, Influx -Presto via AWS Athena, Redshift -homegrown solutions for a data lake & managing k8s microservices

Fwiw, their team does appear to have a rigorous internal post-mortem process
(SEV reviews) [0] drawn from industry best practices.

Let's hope a public one will be made when issues are resolved. (FINRA will be
asking for one anyway.)

Not piling on. I genuinely feel for all the infra engineers staying up all
day/night fixing this.

[0] [https://robinhood.engineering/creating-a-sev-process-that-
sc...](https://robinhood.engineering/creating-a-sev-process-that-scales-with-
robinhood-4b433f9c439b)

------
koolba
Is this a technical issue with the platform or underlying financial issue with
their company and margin system?

I would not be surprised if a “move fast and don’t test things” approach to
margin stress testing would collapse in the market we’ve had for the past
week.

------
jennyyang
Is this due to their Kafka infrastructure being down? Their tweet last night
alluded to it.

~~~
dmead
link to the tweet?

~~~
mulcahey
> The outage was not caused by a failure to code for the leap year. We had
> instability in a part of our infrastructure that allows our systems to
> communicate with each other.

[https://twitter.com/askrobinhood/status/1234861941413351434?...](https://twitter.com/askrobinhood/status/1234861941413351434?s=21)

Sounds like Kafka / message bus to me

~~~
conduit242
Could easily be service discovery too, like Zookeeper

------
kyleblarson
Given the sorts of positions RH has allowed the wallstreetbets crowd to put on
I would not put 5 bucks onto the platform.

~~~
spectramax
I actually don't think any sort of gatekeeping to do whatever the hell you
want to do with your money is the broker's business. Sure, don't allow naked
calls and other margin shenanigans, but for the most part - these are fully
functional adults - if they decide to gamble with their money, so be it. Let
them do it.

~~~
kyleblarson
I tend to agree with allowing people to do whatever they want with their own
money, but I just cant picture a broker like IB for example allowing this sort
of thing without much more serious vetting of the client.

~~~
pfarnsworth
You probably don't know much about IB then. There's no vetting at all, by IB
or any other brokerage, except making sure that the money is in their account.
They have margin calls for a reason.

------
toomuchtodo
Please move to a real brokerage if you use Robinhood!

> Why?

EDIT: Because their systems are garbage and their support non-existent. You
can do better literally anywhere else (Fidelity, Schwab, Interactive Brokers).

~~~
lsiunsuex
Why ? Somewhat serious question - what will a real brokerage offer me on my
tiny (< 1000) investment - we're just testing waters (at a bad time no less)

Blanket statement like that without reason would equal please switch to self
hosting git instead of using github.com cause it went down last week or so -
both have their purpose / skill level / cost associated with them. So a 3rd
party app went down for a bit. Happens often (as most of us know)

~~~
crystaldev
Trading with < $1000 is an utter waste of your time. Interactive Brokers has a
$10k lower account limit for your own good.

~~~
paulgb
Investing under $1000, especially with commission-free trades, is a good way
to learn the basics mechanics of stock trading.

~~~
JumpCrisscross
> _Investing under $1000, especially with commission-free trades, is a good
> way to learn the basics mechanics of stock trading_

Individual investors are at a _huge_ disadvantage when it comes to intraday
trading.

For everything other than a child's hobby account, intended to teach emotional
stability through gains and losses, a <$1,000 stock-trading account is value
destroying.

~~~
lsiunsuex
so again

you want me to risk > $1000 when I don't know what I'm doing with a real
brokerage that I don't know what their benefit is over something like
Robinhood because.... ?

I'm willing to loose a couple hundred to learn and understand something vs
giving someone I don't know thousands and "trusting" their opinion.

More so - testing the waters myself may not make me as good as someone who
does this for a living - but maybe after a few hundred and a few months, at
least I have a better understanding of what a real brokerage tells me to buy
than to just blindly say take my money and quadruple it.

~~~
JumpCrisscross
> _what a real brokerage tells me to buy_

Brokers should _never_ be telling individual investors what individual stocks
to buy, at least not anyone with less than ~$500,000 in assets. If a brokerage
is giving you buy/sell lines for individual securities, that's a red flag.

A good investment platform (or adviser) guides you in portfolio management. In
encourages long-term strategic thinking over short-term trading highs. The
former builds wealth. The latter lines professional traders' pockets.

------
tomashertus
I already initiated transfer of all holdings from RH. For everyone interested
you can do the same:
[https://robinhood.com/us/en/support/articles/360001226666/tr...](https://robinhood.com/us/en/support/articles/360001226666/transfer-
stocks-out-of-your-robinhood-account/)

~~~
Operyl
Which, hilariously, will cost you 75 dollars. Man, Robinhood is in for quite
the lawsuit.

~~~
inferiorhuman
Most brokerage firms will charge similar fees.

~~~
Operyl
The problem is most brokerage firms don't have two days of downtime where you
couldn't trade at all, while the market is in such huge fluctuation.

EDIT: Why am I getting downvoted for this..? That's the reality of the
situation here.. If I missed something, please reply instead of just
downvoting.

------
SirLJ
Trading app Robinhood experiencing ‘major outage’ for a second day amid heavy
volume market action:

[https://www.cnbc.com/2020/03/03/trading-app-robinhood-
experi...](https://www.cnbc.com/2020/03/03/trading-app-robinhood-experiencing-
major-outage-for-a-second-day-amid-heavy-volume-market-action.html)

------
thdrdt
Some context (I had no clue what the site is about):

Robinhood, a pioneer of commission-free investing, gives you more ways to make
your money work harder. Robinhood, a pioneer of commission-free investing,
gives you more ways to make your money work harder. It's Time to Do Money.

~~~
jaredwiener
They were also down yesterday during the market's rally.
[https://markets.businessinsider.com/news/stocks/robinhood-
ou...](https://markets.businessinsider.com/news/stocks/robinhood-outage-app-
back-up-after-glitch-stock-market-rebound-2020-3-1028959673)

------
wk0
For the algotraders out there, figured their status page could be an
interesting indicator:

[https://github.com/wk0/is-millennial-retail-
trading](https://github.com/wk0/is-millennial-retail-trading)

~~~
opportune
Those are usually updated manually by a human potentially hours after the
outage started. It would probably be better to just probe their backend APIs
directly

------
a2h
Fed cut interest rate by 50 basis points. Volume greatly increased since 10AM
announcement.

~~~
ajphdiv
I think it actually crashed right before that.

------
kcommam
Related: I noticed Vanguard's site was having trouble handling requests last
week. It was generally slow and would frequently error out or log out. The
site is running well now and it appears at least part of the solution was to
switch back to a previous version of the UI. Maybe two years ago they switched
to a more responsive, web 2.0-ish interface (which, to my eyes, significantly
reduce the information density of their pages in favor of large text and
looser spacing). This says to me that someone implemented a poorly performing
UI on top of a well-performing backend. Would be interesting, though unlikely,
to get a technical response from Vanguard on what happened.

My company's 401(k) provider's site was also experiencing significant issues.
Seems like last week would have been a fun one to be a system administrator in
the investment/brokerage sector.

------
blackswan101
Unforgivable. Moving funds to Schwab. And that's saying a lot

------
champagnepapi
yes it is unfortunately

------
trickledown
Geesh, what do you want for free.

~~~
missedthecue
Brokers make little of their profit from commissions. Even legacy brokers like
Schwab. They make their money on advisory services and margin lending.

------
godlikeNoob

       Robinhood has the worst customer service and security. My account got hacked, I changed the email, phone number and enabled 2FA and it got hacked again. I was trying to resolve it for a month. 
        I really don't see why anybody would use Robinhood instead of an actual brokerage, now that all of them are free.

~~~
freehunter
Why did you write this in a format that requires mobile users to scroll
sideways? No one is going to read that.

~~~
freeAgent
I'm a desktop user and it requires me to scroll as well.

