
An Update from Robinhood’s Founders - Beowolve
https://blog.robinhood.com/news/2020/3/3/an-update-from-robinhoods-founders
======
czbond
On the profession side of this, if you're an engineer at RH in the thick of
this - many have been there. It seems dire now, but in a few years the fog,
panic, and haze of no sleep will become a story you tell your peers at happy
hour.

Many will cast stones - but they have been there too. If they haven't, well
maybe their day will also come. You may feel bad at the moment - but the best
way professionally forward is "We try our best tomorrow"

~~~
cheschire
If this were an outage directly caused by a natural disaster, I could
understand. This outage was an availability problem. This clearly points to
some prioritization problems within the leadership layers if robust and
resilient infrastructure was not emphasized.

The prioritization problems may not be due to ignorance or malice though, and
may be justifiable if there are other fires that are burning brighter. It's
still pointing to problems though, and I think it's completely legitimate for
engineers to question the stability of the company when this sort of thing
happens.

At the very least as an engineer I would be asking some pointed questions of
my leadership. Maybe not dusting off the resume yet, but still I'd want to get
reassurance from internally that the leadership problems that caused this are
being addressed.

~~~
malux85
Sometimes you just have to cut them some slack. Have you engineered a highly
available cluster before? I'm not talking about the hot-standby postgres
master that gets called on once every 2 years, but I'm talking about a 180
node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at
60,000 writes a second every day, and you have to do node replacements every
week or two because of the high load.

Or I'm talking about a 200 node hadoop cluster thats doing the electrical
metering and billing for 8 million people, and is NOT allowed to stop.

Or the trading platform thats running sub millisecond trades and downtime
means 300,000 $ USD per minute.

These are systems I have engineered over the last 10 years, and I can say:
These things are _complex_ and have failures in 1000 different ways, and while
you're monitoring 999 of them that one thing you're not looking at is
festering under the surface (your monitoring system _is_ tracking IRQ hardware
interrupt response times, right???)

Part of being in a team is everyone pulling together, and yes it's stressful
at the time, but even very good management cant see all ends, just like very
good engineering cant predict everything. I don't think it's useful to start
pointing the finger at management and "asking some pointed questions at
leadership" because sometimes everyone _is_ doing their best. Yes we should
analyse our failures so we can do better, but your tone is very accusatory,
and I believe that a better approach is an all inclusive chat about how we can
do better, and management saying "great job engineering" for fixing it, and
giving them a break after the stressful event.

~~~
techie128
Kudos, these are moderate sized systems you've built over your career. There
are lot bigger and more mission critical systems in the world and you might
build them one day.

I understand GP's tone wasn't exactly nice here. But here's the rub with RH's
outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation,
Food, etc.) where people _need_ to trust them to be successful. The
consequences of failure in these industries is very catastrophic not only for
them but their clients. Sure failures happen but the scale at which RH has
failed and the lukewarm response they've put out has pissed off people. I
don't recall any brokerage, old or new, that has failed so catastrophically
_and_ has responded to it so poorly. If you think you have a worse example, I
am all ears.

~~~
kasey_junk
Hard to judge _worse_ in this context but while I was trading all of CME
globex was down for 4 hours canceling active orders.

[https://www.profit-loss.com/cme-hit-by-globex-outage/](https://www.profit-
loss.com/cme-hit-by-globex-outage/)

I don’t remember them offering any apology or explanation at all.

That’s an exchange mind you where things like the global price of oil and s&p
futures trade. Not a small boutique brokerage.

Further they have planned downtime every week & at that point still had
planned daily downtime I think.

I think Robinhood screwed up. I think they should learn a hard lesson. But
people thinking that trading is some high reliability industry haven’t spent
any time in it.

The scary thing to me is are healthcare, aviation & food the same?

~~~
vsareto
Part of AWS's sell with elasticity is only spending what you need, but those
industries have redundancies or unused capacity.

Someone in one of these threads said there's a hidden DNS within VPCs that can
fail and isn't scaled, so if that's true, they might just have to architect
around that unless they can get AWS to change it. It's on RH for not knowing
that but it's also kind of on AWS too.

But as far as what you can do, you can really only split your cash across
brokerages if you want to engineer the same redundancy yourself. Otherwise, RH
would need to route everything to another exchange to keep satisfying orders,
and even that is just another system that could fail. Keeping all of your
money in one brokerage doesn't seem ideal if you want to completely avoid
downtime. Doing the same redundancy yourself with those industries isn't
really practical.

------
RayVR
Having worked as a professional investor since 2012, I can say these outages
can happen anywhere. I've seen day long outages at exchanges where tens or
hundreds of billions of dollars would have been trading, at brokers where who
knows how much would have traded. I've also experienced these outages at
retail companies that are more established, including TD Ameritrade (I become
a customer when ThinkOrSwim was acquired.) I have also seen brokers screw over
individuals on a significant scale without real ramifications.

The fact that Robinhood is telling people anything about the outage is only
because they are the company they are, operating in the startup
world/mentaity.

To the people thinking they should be compensated in some way...If you are
doing >$1m daily volume, maybe you can contact them to see what they can do
but even then, I doubt it. The way this should be handled is to have multiple
executing brokers. You can implement offsetting positions if needed and
transfer positions when your main account becomes available, if you are using
a broker that can clear. Right now it seems Robinhood is working to implement
clearing but you could still go to neutral or put on your positions.

~~~
twic
> The fact that Robinhood is telling people anything about the outage is only
> because they are the company they are, operating in the startup
> world/mentaity.

Yep. Intercontinental Exchange and Eurex, two huge capital markets exchanges,
routinely have multi-hour outages and don't even acknowledge that they've
happened, let alone explain them.

~~~
whb07
multi hour isn't day and a half.

------
Itsdijital
I have mixed feelings of sympathy about this whole RH thing.

Anyone who has used RH regularly should be well aware of how inept it is. Any
spikes in volume or volatility, even on a single stock, bring it to it's knees
pretty often. Like not just the last week, but even during calm periods. I've
personally lost 20-30% on positions solely because RH was bugging out,
thankfully I use RH just for "fun trades" usually <$100.

I cannot fathom having the balls to trade any real amount of money on the
platform while being aware of these long term issues.

On the flipside I feel for new users and perhaps even generally inactive users
who weren't aware of RH's incredible flakiness. I'd imagine (or hope to) the
losses of most of those users were small, assuming they were new or casual and
just testing the waters.

Even if one of my small plays hit it big on RH, the money would just go to my
main account on TD (which has been smooth all week shy of a few hiccups Fri
morning during record volume). It's been obvious for a long time that RH
should not and cannot be trusted. If you're trading options with a $60K
account on RH, well, I don't even have words for that level of ignorance.

~~~
stef25
I abandoned Coinbase after having difficulties getting a few 1000 bucks out of
there. It worked out in the end.

Problems with my data I can tolerate up to a point. Problems with my money I
absolutely can not tolerate. As you said, it's unfathomable how people can
trade money on a platform that's flaky.

~~~
robinson-wall
The interesting thing about working for a UK challenger bank - I now have
visibility into all of the outages going on at large, high-street banks here.

Complete outages are rare, and well-publicised, but things go wrong a lot
more[1] than you might think without any communications to customers that
anything is wrong, sometimes outright denying[2] that there's a problem.

1:
[https://twitter.com/nickrw/status/1141058572547215360](https://twitter.com/nickrw/status/1141058572547215360)

2:
[https://twitter.com/nickrw/status/1164162320672669696](https://twitter.com/nickrw/status/1164162320672669696)

~~~
twic
IIRC the SLA for FPS is 2 hours. So if a bank stops processing them for an
hour, that's within tolerance, and they don't need to tell anyone.

I think your point is that it's a very different mindset to the native
internet world, and that is certainly true!

------
jennyyang
I know quite a few people that were personally affected by this and lost money
due to the two outages and they are all pulling their money from Robinhood.
The fact that they can't offer any compensation might be a big problem for
them, since they already have zero trading fees, which is what most brokerages
offer as compensation.

Personally it doesn't pass the smell test for me. The load was much higher the
previous week and load problems go away once the load disappears. They
probably had a lot less load the rest of the day, so the fact they were down
the entire day suggests it was something else. I would need a fully
transparent post mortem before I believed anything they said.

~~~
rolltiide
> The fact that they can't offer any compensation might be a big problem for
> them, since they already have zero trading fees

Robinhood makes the most money than any known firm on Wall Street by getting
paid specifically to leak user's trades to other traders.

SEC requires a periodic report on that which shows compensation.

Can't believe people are still buying Robinhood's pitch of misdirection.

~~~
LatteLazy
Is there a source you can cite for that? Why would anyone want retail investor
order data? Especially since most of their orders execute immediately, so you
can just get the trade data from the venue...

~~~
a2h
Rule 606 disclosure for the source. And it's not about the trade data it's the
order itself. Second link is a very thorough explanation.

[https://cdn.robinhood.com/assets/robinhood/legal/RHS%20SEC%2...](https://cdn.robinhood.com/assets/robinhood/legal/RHS%20SEC%20Rule%20606%20Report%20Disclosure%20-%20Q4%202019.pdf)

[https://www.google.com/url?sa=t&source=web&rct=j&url=http://...](https://www.google.com/url?sa=t&source=web&rct=j&url=http://blog.themistrading.com/wp-
content/uploads/2015/03/What-Every-Retail-Investor-Needs-to-
Know.pdf&ved=2ahUKEwi475aysYDoAhWuY98KHR0HCPEQFjAAegQIBxAC&usg=AOvVaw0M_rFmz-
YkQZu5pOnkiYVK)

------
RestlessMind
This is such an empty update. At the very least, they should have published a
detailed postmortem or committed to one by a certain date. How are we supposed
to know that they have learned their lessons?

~~~
harikb
I don’t work for them, but I am pretty sure we can blame the litigious nature
of this industry for the lack of detail in the postmortem. Not everyone can
afford to be cloudflare :)

Even for Cloudflare, I thought the company will get sued out of existence
after the proxy data leak, but finance industry/SEC etc is a completely
different ballgame.

~~~
dx034
I believe it's the fear of litigation rather than actual litigation. Other
companies also manage to publish postmortems and don't get sued out of
existence.

------
dilly_li
Start from the email notification. They have been asking themselves the easy
questions.

Just look at the top questions in their email:

* Are the funds in my account safe? Yes, your funds are safe.

* Was my personal information affected? No, your personal information was not affected.

* Can I use my Robinhood debit card? Yes. If you have a debit card, you should have been—and should still be able to—use your card, but you may have had issues receiving notifications, viewing your balance, and seeing transactions in your app.

\------------

The real question is: How is Robinhood compensating for the missed trades?

Stop asking yourself the easy questions, RH.

~~~
throwsprtsdy
I think it's unlikely that Robinhood (or any brokerage) would compensate
people for losses on hypothetical trades that could have been made during an
outage. Such a policy would allow customers to pick their entry and exit
points, and extract money from the brokerage at will.

Even if the trades were well-defined at the time the outage occurred, there
would still be an asymmetry between people demanding compensation on their
profitable trades while eschewing losses on their bad trades. It's doubtful
any brokerage would be willing to eat that.

Execution risk is a risk.

~~~
asah
Are expiring in-the-money options a "hypothetical trade" ?

~~~
benmanns
Those are automatically exercised at expiration.

~~~
Itsdijital
If the holder can afford to exercise, and I can assure you that most RH users
cannot afford to exercise a single contract(at least on most commonly traded
stocks).

Otherwise it's on the broker to sell it at close to someone who can afford to
exercise. And who knows if RH pulled that off or not.

~~~
benmanns
I haven't seen official statement, but I did see a couple reddit threads where
Robinhood exercised ITM calls without enough funding in the account, then
collected the shares and paid the cash difference.

[https://www.reddit.com/r/wallstreetbets/comments/fcqkmo/so_r...](https://www.reddit.com/r/wallstreetbets/comments/fcqkmo/so_robinhood_just_excercised_my_spy_call/)

There's also some threads where it looks like it did not go so well...

[https://www.reddit.com/r/wallstreetbets/comments/fd2eko/upda...](https://www.reddit.com/r/wallstreetbets/comments/fd2eko/update_yesterday_i_said_i_lost_400_a_day_thats/)

------
jsf01
I’d be interested to read a deep technical post-mortem like those which have
become fairly standard among other big tech companies. Hoping Robinhood does
the right thing here.

------
0xy
Still silence on the traders who lost tens of thousands of dollars? Are they
going to be compensating or not?

This blog post doesn't appear to say anything. It's not an apology, it's not
an explanation, it doesn't say what they're going to do in response.

This is after the incident in which there was no status updates or support
availability for multiple hours of time. Why can't they commit to updates
every hour or every 30 minutes?

~~~
ska
I agree the level of feedback isn't great, but what would people be
compensated for? Did they misplace actual orders?

~~~
cdurth
Yesterday was the largest upswing in market history and the entirety of RH
missed out.

~~~
majormajor
"Missed out" doesn't seem like the right phrase here. If you already owned the
stock, you still held it, no?

So people who were going to continue to sell off got lucky that they couldn't
make that trade, and people who were going to buy got unlucky?

Does anyone seriously expect compensation, or think that it's deserved, or is
it group wishful thinking? How would it even work? Would they just take
people's word for their supposed intent? Or are people wanting some sort of
"here's a gift card" type deal?

This is not to defend RobinHood - I've personally kept my money with well-
established companies cause conservative, old, proven systems seem like a good
thing for a product in this space - but shit happens, no? There will be more
good days, and more bad days, in the market, it's a long-run game anyway, and
it's pretty easy to vote with your wallet in this space.

~~~
yesiamyourdad
This is just like the late '90s, up to 9/11 when day trading was the rage. Now
it's back in fashion, low-key.

I had never heard of /r/wallstreetbets or Robin Hood (well, barely) until a
couple weeks ago.

~~~
JumpCrisscross
> _Now it 's back in fashion, low-key_

There is an entire generation that has never traded through a crisis.

~~~
speedplane
> There is an entire generation that has never traded through a crisis.

Given that most crises seem to occur roughly every 7-15 years, there will
always be such a generation.

A hypothesis: the reason why crises occur roughly 7-15 years is because that
is approximately the length of society's collective memory concerning monetary
issues.

------
dang
Recent and related:

[https://news.ycombinator.com/item?id=22477567](https://news.ycombinator.com/item?id=22477567)

[https://news.ycombinator.com/item?id=22475019](https://news.ycombinator.com/item?id=22475019)

[https://news.ycombinator.com/item?id=22468361](https://news.ycombinator.com/item?id=22468361)

[https://news.ycombinator.com/item?id=22465178](https://news.ycombinator.com/item?id=22465178)

------
aloknnikhil
Genuine question: With no commission trading at places like Schwab and eTrade,
is it even worth trading on Robinhood? For as far as I could remember (about 2
years ago), Robinhood has always failed to scale.

~~~
manigandham
Options are completely free on Robinhood while they still have a per-contract
fee at other brokerages. If you don't care about that then no, there's no
reason to stick with Robinhood.

~~~
benmanns
Additionally Robinhood self clears options (or for some other reason?) and
does not charge the Options Clearing Corp fee of $0.055/contract or the
Options Regulatory Fee of $0.0388/contract which all other brokers charge
(incl. ones with $0 or flat rate commissions/fees like WeBull, Gatsby,
Tradier). All you pay is the FINRA and SEC fees on sells of about a penny each
for small trades.

Actually, if anyone knows of another broker who _doesn't_ charge these, please
let me know. If you're first for the broker I'll give you $20 for the tip.

~~~
Itsdijital
Trust me, please trust me, you really really really want to be paying a
competent broker when trading options.

If it's chump change you're trading, sure, use RH.

If it's serious money, the $0.65/contract or whatever pays for itself many
times over. Even if it's just the ability to regularly get filled between the
spread it pays for itself.

~~~
dehrmann
I've always been told trading, but especially trading options, is asking the
big boys to take your money.

~~~
throwaway9d0291
It's not at all so straightforward. There's definitely money to be made
trading.

In particular, there's plenty of opportunity to find a successful strategy
that just doesn't scale. It won't make you rich but it will make you money.

~~~
dehrmann
> there's plenty of opportunity to find a successful strategy that just
> doesn't scale.

True.

------
mjs33
Their DNS system failed? How?! Unless DNS stands for “Do Not Sell”

~~~
tbrock
This happened to us at Hustle years ago. Basically if you run on AWS there’s a
DNS server provided inside each VPC that usually works fine but which has no
observable load metrics etc... so you don’t really know you are slamming it
and are about to have a problem unless you audit your entire codebase.

Why? Well that tiny DNS server has certain capacity constraints and if you
don’t cache DNS lookups by using a http/https agent for example (in NodeJS)
you wind up looking up the same dns info over and over and churning sockets
like it’s going out of style. If you run really really hot the poor thing
falls over (rightly so).

The limits are high and DNS is fast so you usually don’t notice but when you
are under load bugs like this come out of the woodwork. When it falls down you
look up the AWS docs, lean back in your chair upon finding this isn’t an
“elastic” part of AWS and say “FUUUUUUUUCK” so loud it can be heard from outer
space.

If you are Robinhood though don’t you have some former Netflix SRE/DevOps
beast on staff that knows this and so you run your own DNS and monitor it?

~~~
ajsharp
Wait, what?? There's an invisible DNS server running inside your VPC? I get
what you're saying wrt cached DNS lookups but this seems wild.

~~~
ra1n85
It's a DNS resolver that runs on the hypervisor hosting every instance.

~~~
tbrock
Yes and they limit you to throwing 1024 packets per second per network
interface at it.

Of course you could run your own dns cache per host/pod whatever.

~~~
cocire
you've got me so curious, could you please point me to the aws docs?

~~~
tbrock
It’s the first thing on google when you google “aws dns vpc limits” but sure:

[https://docs.aws.amazon.com/vpc/latest/userguide/vpc-
dns.htm...](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-
dns-limits)

------
tempsy
Sad that there isn’t an actual apology anywhere to be found in the letter at
all.

And now with the fed rate cut the interest on cash is only 1.3%, with more
cuts expected later in the year, which was the last big differentiator. I
don’t see how they don’t see massive net withdrawals going forward.

~~~
CamelCaseName
> And now with the fed rate cut the interest on cash is only 1.3%, with more
> cuts expected later in the year, which was the last big differentiator. I
> don’t see how they don’t see massive net withdrawals going forward.

This isn't really an issue because the fed rate cut impacts everyone. Other
institutions will cut their interest rates as well. I know of a few banks
(Canadian) that have already lowered their GIC rates.

If anything, this is actually good for RH. Now instead of comparing 1.8% at RH
and 1% at another Financial Institution, you're comparing 1.3% and 0.5% -- a
much bigger multiple.

~~~
tempsy
most brokerages don’t actually pay anything. With another cut it’s going to be
<1% vs 0%. Hardly anything even with a six figure balance. That’s my point.

------
xyst
The boys down in the salt mines of WSB will want a blood sacrifice.

Founders should be fired. CTO/CIO should be replaced.

------
vinaypai
Historic... Unprecedented... Thundering herd, a bunch of excuses to explain
why they couldn't handle the volume that most real brokerages handle every
second.

------
alishan-l
I heard it was related to the leap year. Apparently they had downtime 4 years
ago as well.

------
ablekh
I'm curious about your thoughts on why a technical infrastructure, which, by
nature of being cloud-native, is supposed to be (and likely has been)
architected as a _highly elastic_ platform, have not stood the test of time in
this regard.

Based on the in information from Robinhood's careers site, their platform is
largely based on the following technology stack:

    
    
      - Python, Django, Django Rest Framework
      - Go
      - PostgreSQL
      - Container and container orchestration technologies (Docker, Kubernetes)
      - Microservice-oriented architectures and related OSS technologies (Kafka, Celery/RabbitMQ, nginx, Redis, Memcached, Airflow, Consul)
      - Cloud-native infrastructure (AWS, GCP)
      - Infrastructure as Code and configuration management (Terraform, SaltStack, Ansible, Chef, Puppet)
      - CI/CD and test automation frameworks (Cypress.io, Jenkins, Appium, UIAutomation, Bazel)

------
vbtemp
On Reddit I've been trying to ask this ELI5:

Why would you use RH instead of a normal, mainstream brokerage like Vanguard,
Fidelity, etc that already has (1) an app and (2) commission-free trades?

~~~
lkbm
Easy answer: As someone who's used Vanguard for index funds and the like for a
couple decades now, I had no idea they had an app or commission-free trades.
They don't market this at all.

As a secondary answer, normal, mainstream brokerages have pretty bad tech,
tbh. I don't expect it to be worse than Robinhood in terms of things like
security, and I expect UX to be worse. (Side note: I just discovered that
Vanguard actually has a secret security key option hidden under Account
maintenance, so I can finally switch from sms 2fa. +1 to Vanguard.)

~~~
infinite8s
> Side note: I just discovered that Vanguard actually has a secret security
> key option hidden under Account maintenance, so I can finally switch from
> sms 2fa. +1 to Vanguard.

It looks like you still need security codes setup:

"You'll need to register for both security codes and security keys, however.
That's because keys and codes go hand in hand—if you lose your key or don't
have it, we'll need to send you a code in order for you to log on. In
addition, you'll always need a code to access your accounts from a mobile
device."

If an attacker can skip the security key you might as well not use one.

------
acchow
Can't wait for the post mortem

~~~
joobus
I don't think we will get a postmortem. Their lawyers will kill it because it
will be an admission of guilt and open them up to even more legal liability.

~~~
wbl
The SEC did one for Knight Capital.

~~~
SkyPuncher
Being down for a trading day is not the same as actively selling $440 million
in assets.

~~~
wocram
I would argue that it is worse for a retail brokerage to be down for a day
than it is for a trading firm to blow themselves up, though I suppose the
latter was more about creating a disorderly market.

------
lemmox
I'm always amazed by how tricky DNS failures can be.

------
Ambele
I can't help but think this glitch was a good thing and Robinhood investors
would do better if they traded less anyhow. According to an OpenFolio
correlational study, traders who trade more than 12 times per year make 0.5%
less than traders who trade less than 12 times per year. OpenFolio was one of
the first three websites to have an API integration with Robinhood portfolios.

------
VWWHFSfQ
every time I see a company that has "co-CEOs" I always wonder what kind of
weird stuff is going on in that company

~~~
winrid
Wasn't Steve Jobs a "co-CEO" of Apple for a while? (Edit, I mean after he came
back from NeXT)

------
LaserToy
Blame the load. If only Robinhood were not famous for saying they are hiring
only the best...

The best do not go down like that.

~~~
vsareto
That’s always marketing speak and never to be believed.

------
dirtydroog
Companies like Robinhood regularly go down when markets are volatile. It was
quite frustrating when the financial crisis was in full swing not being able
to log in to my trading account. I reckon I would have made a killing.

------
0xDEEPFAC
"Multiple factors contributed to the unprecedented load that ultimately led to
the outages. The factors included, among others, highly volatile and historic
market conditions; record volume; and record account sign-ups. "

What a sad press release, I am sure people at their corporate office were
sweating over this. The long and short of it is that users trusted the service
would work and had possibly a great deal invested only to get a comment when
everything breaks down deflecting blame "OMG we weren't prepared for what our
users did!"

We live in a sad state of software. I expect things like this and the Equifax
scandal to continue if things like software security, reliability, and
performance aren't taken into account.

------
homero
This is common in Bitcoin exchanges. Never thought I'd see a stock broker go
down but i guess it's the same issues.

------
buryat
> Traditionally depicted dressed in Lincoln green, he is said to have robbed
> from the rich and given to the poor.

Does the name still stand?

------
shrimpx
Wow this is Coinbase in 2017. Trading mass hysteria takes down unprepared
Silicon Valley trading service.

------
shiado
Does Robinhood have stop-loss orders an if so did they execute when it was
down?

------
sitzkrieg
volatility always shakes out the trading companies with lame infrastructure

------
gadders
I mean this is OK as an apology, but is there an actual post mortem anywhere
was can read?

------
c930
dnsmasq is your friend.

------
egdod
Pretty light on information.

------
ilrwbwrkhv
wasnt this cause by leap year and them not taking that into account?

~~~
illnewsthat
They denied that on Twitter:
[https://twitter.com/AskRobinhood/status/1234861941413351434](https://twitter.com/AskRobinhood/status/1234861941413351434)

~~~
tomc1985
So what of the screenshot in the original twitter post? Was it doctored?
Showing GMT?

~~~
ajhurliman
I don't know if it had anything to do with leap year, but I also checked dev
tools and saw the same issue (requests for market data on March 3, 2020 on
March 2, 2020 8AM PST). However, it was busted for both the website as well as
the Android app (and I'm guessing iOS too) so it doesn't seem like it's purely
a client-side problem unless all of their clients were built from the same
source.

------
lowdose
Did any of the complaining people actually pay RH for their service or is this
1st world entitlement?

------
yaur
"That in turn led to a “thundering herd” effect—triggering a failure of our
DNS system."

I'm just a spectator but I can not imagine that this was somehow caused by a
DNS failure.

~~~
zippergz
I’m not sure I can even count the number of outages I’ve been involved in that
had DNS issues as at least part of the cause.

~~~
yaur
sure, I've seen outages that are caused by DNS config problems. But I don't
think I've ever seen one caused by a "thundering herd" overwhelming DNS
servers.

Another give away that this is a lie is that support emails were getting a
stock postfix error message which means that MX records at least were
resolving.

------
rhizome
Is every bitcoin company run by ex cellphone store employees? Just the
exchanges?

~~~
dickjocke
Robinhood isn't a bitcoin company. That's just a feature they have. Its main
product offering is the commision free trading--and their presence pushed a
lot of big players to adopt the same offering. The wallstreetbets gang is
silly and all, but I think they have really democratized stock trading, and
made the whole idea seem much more accesible. I think they founders are former
finance guys. I hope this doesn't sound like guerilla marketing. I don't even
use the app, I have used it but I'm just not that interested in picking
stocks. I just think it's cool as an ex-code monkey to entreprenur story.

~~~
triceratops
> I think they have really democratized stock trading

I would think Vanguard did that already. Most people should be trading ETFs,
not individual stocks.

~~~
idnefju
I believe you still need a broker for ETFs, which usually carry brokerage
fees.

~~~
ksherlock
With mutual fund companies, including Vanguard, generally you can open an
account directly with them and buy directly from them (including partial
shares, automatic monthly purchases, and dividend/capital gain reinvestments).

------
mkchoi212
It's cool that the founders of the company publish blog posts like this for a
short outage. Hope other CEOs learn from this and become even more transparent
in the future :D

~~~
manigandham
Short outage? They were down for almost the entire trading yesterday and hours
today. And there's barely any transparency in this post compared to standard
post-mortems.

~~~
rpdillon
Was this their post-mortem? I didn't see anything to indicate that.

------
bayonetz
Just use Square’s Cash App! Free stock trades AND you can buy fractional
shares AND a bunch of other stuff like P2P payments and bitcoin. I work there
and so can say with some authority that we can handle more volume without
going down than RH can.

~~~
minimaxir
If you actually work at Square, it's poor form to advertise in this manner.

~~~
redis_mlc
Actually, bayonetz's posting is the only useful one in the comments for this
article. Most of us are here for information from actual industry insiders,
and this qualifies.

Here's some more inside info ...

If your "financial app" provider doesn't have a banking charter, run. None of
the recent trendy fintech companies have a charter, and are thus clown cars.

~~~
astura
Fidelity offers banking services and doesn't have a banking charter but they
aren't a "clown car," they are one of the largest financial institutions in
the world.

