
A Billion User Load Test on Healthcare.gov - shashashasha
https://blog.navapbc.com/the-billion-user-load-test-ffb035aeb2d6
======
sixbit
Healthcare.gov is the absolute worst. They had a button "End coverage for
2017" I clicked it around the end of November because I chose a non-
marketplace plan for 2017, and the end result of that click was Cigna my
current marketplace insurer for 2016 received a transmission from them, and
interpreted it incorrectly, which made my 2016 plan inactive and retroactively
terminated me back to 08/31 despite having paid all my premiums and processed
claims up until end of November.

Their escalation process is useless, 30 days to resolve (if I'm that lucky).
Will try not to get injured in the meantime.

Glitches and data sync issues are unreal, this is not an isolated incident.
Every year it's been something. 2014 I was on Covered California and had to
take them to administrative court to resolve data and tax form issues
(canceled then due to a move and it completely wiped out my 2014 enrollment).
Then in 2015 on HealthCare.gov they had me enrolled in the same plan twice and
the insurer wanted double the premiums until they could resolve it.

~~~
jlebar
Now we see the real test of whether healthcare.gov has reached the level of
consumer tech: Does bug-report-via-top-ranked-HN-comment get anything done?

~~~
llimllib
AFAICT it's actually a Cigna bug report yeah? Happy to make sure it gets to
the right place if I'm wrong.

(/me doesn't work for nava, does work on healthcare.gov)

~~~
sixbit
There is a persistent error message on my 2016 profile on healthcare.gov after
that button was clicked (despite it also showing it's active on that side of
things). My understanding is that the bug is the integration point between the
two. From past issues and what I've been told on this one, is that the only
way things get fixed and made right is to have healthcare.gov retransmit the
correct data/status. Cigna can't fix it on their own from their side. Thanks
for the offer to help, didn't expect any action, just a post out of
frustration. I'll shoot you and email.

~~~
roninb
I'm almost positive you mean "I'll shoot you an email." but the idea of
someone being so frustrated with healthcare.gov they want to shoot the
messenger (yet still get the problem resolved) is in-line with what I've
heard. (and gave me a hearty chuckle)

~~~
sixbit
Haha, just a typo and not a subconscious slip, I swear! :-)

------
coldcode
I wonder if this healthcare.gov will exist a year from now. It is nice to see
how far it has come in 2 years.

~~~
LargeCompanies
Doubtful Trump is getting rid of it.

Personally I preferred the previous way of signing up for coverage ... call up
a private insurer and sign up? The cost was less too as I wasn't paying for
insurance for lazy sally or Joe Schmoe! So for me Obamacare/social medicine
forced those who do and work to pay and cover for those who don't. See ya
Obamacare(love obama though).

Also the $300 plan I paid each month for sucked.. they fought me tooth and
nail to cover things my doc prescribed.. I.e. Various scans(cat scan, etc).

~~~
LargeCompanies
I'm paying $400 a month now thru my employee just for myself with 1k
deductible... that's nuts.

My $300 Obamacare plan was horrid! Love Obama but see ya Obamacare!

~~~
sheeshkebab
nuts? I'm paying $700/month for a family through an employer plan. The cost of
the worse plan on Obamacare is about $1500 (oh, and before obamacare that plan
would cost the same $1500 and come with pre-existing conditions crap).

Medicare for all, $120/month/person - problem solved.

------
patja
From the information provided I am doubtful that this is really a load test of
healthcare.gov. It sounds like they are testing the login and account
management system in isolation from the overall workflow of entering your
financial data, dependents, etc. and shopping for plans, which is where the
sausage gets made. It is an interesting post but it seems like this is about
the most trivial component of the overall healthcare.gov solution, unless I am
missing something.

------
sulam
8K transactions / second is hardly extreme load. It's good they can support
it, and I doubt they need more, but many, many sites handle more load than
that.

~~~
notyourwork
Agreed, it is good they can support it. However, is 8k TPS really a big deal?
The team I work with handles 1M TPS.

~~~
ProAm
it's the government, we're thrilled they can do anything.

~~~
Certhas
I know it's just a throw away joke, but this persistent idea that government
is incompetent, no matter what is really damaging.

When it comes to health care systems there is no evidence whatsoever that less
government means more efficiency, if anything the US is a huge example to the
contrary.

~~~
ProAm
Im not really joking, their ability to do anything efficiently or correctly
for a reasonable sum of money is embarrassing. I know it's a joke as well, but
it's funny because it's true.

~~~
mozumder
My experience is the opposite. Private companies seem completely inefficient
compared to government, and I've worked as engineers in both.

Really it is only a right-wing profit-driven talking point to say government
is inefficient, mostly to protect against their profits that government takes
away.

Government is exceedingly efficient.

~~~
ProAm
I've worked in both as well and vehemently disagree.

~~~
mozumder
The numbers support my case. You can send a mail with the USPS for 50 cents
anywhere in the country, but it costs $5-$10 to do the same with UPS or FedEx.

Literally 10x-20x costlier.

Government just doesn't have to waste money on things like profit or marketing
that private companies need to do.

~~~
ProAm
USPS is a private entity under the government, they fund themselves. They
receive no funding from the government.

You can look at the military and at obamacare/heathcare.gov for examples.

~~~
mozumder
You can also look at the USPS as an example, since it IS a government entity
that doesn't have to serve profits.

Also, the military and healthcare is more efficient in government, but you
probably already knew that already.

------
lcw
I'm glad heathcare.gov is getting better. I would assume however the most
important part of tests for open enrollment are the writes to user management
where reads and writes simultaneously are exercised against datastores. Looks
like users were manually created not sure if they included updates in the
tests. I'm wondering if the tests were just auth based? That would make sense
why they could easily scale horizontally since auth is a lot of times just
limited to cpu capacity.

------
SilverSurfer972
A week to fill a DB and 70 m3x-large is way too expensive to find out how far
you can scale. Me and my friend built stacktical
([https://www.stacktical.com](https://www.stacktical.com)) so that with very
minimal load testing it's possible to understand and share how many concurrent
users your current and upcoming application infrastructure can handle. Testers
are welcome

------
mooneater
7754 tps over 1 hour is 28M. They may have 1B users in the db but they only
tested activity on at most 28M.

~~~
billsmithaustin
True, but running at 7754 tps over an hour with a database of 28M users is
easier than with a database of 1B users.

~~~
mooneater
It is much easier to test with a small # of active users and a large (mostly
unused) database, than with a large # of active users. To me the title implies
1B active users ("billion user load test").

~~~
user5994461
They are two different problems.

If you have many active users, you're testing for the write load of user
tokens and sessions.

If you have many accounts, you're testing for the read load of the account
database.

Technically speaking, user id and sessions should be two different database
systems, because they have different requirements and they're conflicting.

For the first one, the problem is to have lots of write and enough
performances. Plus having the storage if you keep tokens forever (typical for
auditing).

For the second one, the problem is to be able to store all the accounts, plus
read performances (which can always be hacked by adding layers of caching).

Funfact: if each account is only email + password (64 + 64 char), 1 billion
accounts consume 128 GB, just for storing the ASCII characters. A lot more in
practise with metadata, empty space and other fields.

They say that they only had 1 maria DB on a single EBS P-IOPS volume. That's
kinda weird that a 1 billion users database could fit in that -and zero
errors, really?-. Maybe they just had a single integer field (1,2,3,4) to
define the user and the password :D

------
jakupovic
Next thing to look at is why only 2k/sec users can be added to the DB.

------
pierotofy
The website is terrible; they keep spamming my e-mail and calling my number
with the same alerts over and over again.

------
user5994461
I'd like to see this kind of authentication done over OpenAM and OpenDJ.
That's the IAM systems done by Sun Microsystem (later Oracle, then Forgerock).
That should be rock solid.

IAM: Identity and Access Management

Forgerock: [https://www.forgerock.com/](https://www.forgerock.com/)

------
kasparsklavins
> The current observed peak load for HealthCare.gov’s Open Enrollment in 2016
> was on Dec 14

2016 Dec 14? Did I miss something?

~~~
ProAm
Probably Dec 14 2015 for the 2016 enrollment year

------
walter_bishop
What's it cost to run Healthcare.gov on AWS, how many people-hours were spent
in designing, configuring and deploying the website as compared to a non-cloud
solution?

------
rco8786
Why load test for > 3x the number of potential users?

------
0xdeadbeefbabe
> (of only 2 machines!) with a total of 6000 worker threads.

What about the bugs that arise from many connections from many machines? Those
must exist too.

------
rebootthesystem
I am sure my wife will be very happy to learn about this test. It will
definitely make up for our insurance going from about $12K per year to $26K
per year. That, along with keeping our doctor and saving $2,500 per year have
been fantastic.

News Flash: Engineers think technology while, in the real world, the problem
has nothing whatsoever to do with it.

It's like tuning your Ferrarri for more power while your reality is you
commute on the 405. In other words, irrelevant.

But, hey, congrats, glad to see what I am paying for.

~~~
guelo
The goal of technology is to be so flawless that it dissapears into the
background of our lives. When healthcare.gov was crashing the technology was
extremely important and it made all the news and became a political
controversy. Now that the technology is fine people like you can finger wag
engineers and tell them that their work is unimportant.

~~~
rebootthesystem
You missed my point. I am an engineer. When I was younger I, too, wasted lots
of time optimizing the irrelevant. Didn't fix a thing.

The ACA/Obamacare doesn't have a healthcare.gov traffic problem, it has a
"it's a piece of shit" problem.

Of course, this is not the purview of engineers working on healthcare.gov. The
point there is that we all get excited about building monuments to technology
while completely forgetting the problem that had to be solved in the first
place.

Someone will invariably say something like "20 million people have insurance
now that they didn't have before!" or some other fabricated number. Let me
address that now.

First. The President of the United States of America promised ME and my family
that we could keep our plan, our doctor and, on top of that, we would save
$2,500 a year. No need for a link, it's all over YouTube. He made that promise
dozens of times.

I don't care of one or a hundred million people got insurance. The POTUS made
a promise to families. If someone in business made such a promise and then
reality was exactly opposite, in an unbelievably grotesque way, the smallest
lawsuit for breach of contract and fraud would have them curl-up into a fetal
position and cry.

Oh, yeah, and the rich part is that they stick the IRS on us. You know, the
Gestapo-like, most-feared, we-can-ruin-your-fucking-life agency in the US now
has hooks into your healthcare. And this is OK, how?

The POTUS lied to all of us. Period. The fact that there are people with
insurance now is irrelevant. My family is playing the equivalent of a home
mortgage for health insurance. How could anyone even remotely tolerate this
from our elected officials.

Now, to address the "insured".

Most of these people have been shoved into Medicaid. The magic pill that
solves it all, right?

I am not going to go into a long explanation. ACH/Obamacare represents the
largest theft of personal property in the history of this nation. Why? Here,
go learn something:

[https://www.medicaid.gov/medicaid/eligibility/estate-
recover...](https://www.medicaid.gov/medicaid/eligibility/estate-
recovery/index.html)

Short version: These "insured" have signed over their personal assets to the
government. All care they receive --by law-- requires reimbursement and the
government is REQUIRED by law to recover costs.

Wonderful, isn't it? So many people now have insurance! So many people who
have no clue the government just sunk their hooks into their estate. What a
country!

I don't particularly like Trump but I hope he shreds Obamacare to little
pieces, it is nothing less than the largest criminal act ever perpetrated on
the American people.

