I have never worked on a high traffic site before, so I am just mostly talking out of my ass. But when I looked at the html source for the coveredca website, it had a lot of of basic problems: there were 3 DOCTYPES on one page, developer comments were still visible (you can actually see the names of all the developers who worked on the project), Java error pages were visible, they disabled the back button completely by enforcing javascript refreshing, they did not minify any of their assets (which would also remove the developer comments) AND they were actually loading many copies of the same assets multiple times because, like I said earlier, their site had 3-DOCTYPES in one html page.
As someone who has never worked on a huge site, I don't want to naively say that it could have been done better. But I feel like those are basic issues that could have easily saved them a lot of bandwidth.
Also, it seems to me that if they had so many issues on the front-end, it's a hint that maybe they also had many similar issues in the backend as well.
Yeah, the CA version (https://www.coveredca.com/) had so many visual glitches and various problems (404s and 500s), I threw up in my mouth a little when I found out that Accenture charged the state over 180M to build it and another 174M to operate it for four years. I don't know how that's possible.
I had a job interview with Accenture that had 2 parts a dinner and a technical chat.
1. At the dinner their representative spoke about how life at Accenture was a breeze and how as a undergraduate he spent most of this time drinking. He drank on the weekdays but his favorite time to drink was on Sundays. He said he used to do some SQL but now he had moved onto the business side of things. One of the candidates told us how his hobby of skydiving let him see the big picture and another told us that his desi background would enabled to come up culture specific features for software services. I probed this question further, and he said that in India showing your muscles was considered hubris, while in America it made you look strong. On his facebook page he wouldn't show his bare chest... Nobody at the table `looked` like they could do any programming work.
2. During the technical interview I was asked what `clubs` I was in. I told them I wasn't in many clubs and instead spent my time doing research work (which by that time had yield publications) and administrating websites. I told them my hobby was programming. I wanted to tell them how I dealt with EC2 sight outages, worked with clients and their clients to write WordPress plugins. I wanted to talk about my experience engineering our production and outsourcing workflow. I wanted to summarize the techniques I learned overseeing our production staff. They didn't comprehend and asked he what `clubs` I was in.
I was scheduled to have 3 interviews, I walked out after the first. Don't work for Accenture, they are a joke because their employees are a joke, which is much in agreement with this article.
I often wonder if this attitude makes the problem worse. If good programmers don't work for Accenture because Accenture is full of bad programmers, then how can Accenture ever become better? How can it evolve to provide better contracting services? If you were in charge of Accenture hiring strategy, what would you change?
> If you were in charge of Accenture hiring strategy, what would you change?
Is my goal to improve the ability of the developers? What does that gain me? If the ability of my talent is only loosely coupled with landing contracts, then I'd try to maximize the gulf between the salary I pay and the value they provide. If fresh grads can produce good enough work to satisfy contracts, and they cost less, they're my target.
Many firms work this way, and it seems to be a fairly successful strategy if you can land the contracts. Having a few senior, competent people in the mix probably helps hedge against any serious problems. Also, I'd make sure to get everybody as many useless certifications from big companies as possible, as they improve our ability to land contracts.
I'm just playing devil's advocate though. My real answer is I'd change my job.
If you were in charge of Accenture hiring strategy...
...then I would be well compensated for not changing a thing. They've had this model since they were "Andersen Consulting" (the rest of us on project teams with them called them Androids, before "android" was cool [this was also before "Andersen" became exceedingly uncool]), and they bank by the truckload. Apparently there is a type of customer that wants legions of safety-school graduates with lots of highlighter in the first three chapters of their "For Dummies" books.
Easy, rebrand and say they are gathering people for a new team. Maybe call it Accenture+. Maybe don't even use the Accenture name. Start a lean technical consulting service company that is 100% owned by Accenture. With 275,000 employee it is clear they are looking for volume and not talent.
Yes, and if they weren't retarded they would call themselves Walton Research, or perhaps be publicly traded with Walmart owning 80% of the stock and being the sole client?
IMHO they would need to acknowledge there is a problem and launch a public program to fix it. "Come fix the biggest consulting company in the world" could attract top talent. But they just can't do it, even if they wanted - how would customers react?
Wow! Given how awful it was on Oct 1 and that it was broken under the load of less than 10 req/s [1], it is outrageous how much it costed. It is probably standard for the government contracts everywhere, but when it is where I know how badly it is done and how much it costs, it's just doubly disgusting.
Interestingly enough, healthcare.gov itself is open source and uses Jekyll for the static site. It seems to be decently done. But the marketplace itself I guess is a whole different story.
"When we worked with the World Bank, they chose a plan from Rackspace for 16 servers," said Gundersen. "That added tens of thousands of dollars, with a huge hosting bill every month."
HHS had similar strategic plans for the new site, at least at first. "They were planning 32 servers, between staging, production and disaster recovery, with application servers for different environments," said Cole. "You're just talking about content. There just needs to be one server. We're going to have two, with one for backup. That's a deduction of 30 servers."
The government IT procurement process puts people that don't know what they are doing in charge of multi-million dollar budgets. The primary determinant of whether or not a company gets hired is its ability to navigate the procurement process - not their ability to actually do the job. In fact, many companies specialize in winning government contracts and then subcontract the work out to others.
It's a sad, broken system that will almost certainly never change - in part because we don't vote for the people in charge of these things. It really doesn't matter who is in the White House or in control of Congress. The fiefdoms within the government that control these issues have maintained a culture of mediocrity for decades, and it will likely remain that way.
I believe the exchanges are run by the states. Since there is overlap - a federal law being implemented at the state level - the procurement process may have had to follow federal guidelines, but I wouldn't be surprised if this were a case of 50 states with 50 different procurement processes.
Nearly all of the GOP states refused to play ball and are just defaulting to the federally made site.
I have a feeling one state's site will be the best and will perhaps be mirrored to other states within the coming decade. This is clearly not an overnight problem, but in the end, once each user gets their insurance they should never need to visit again unless they want to switch.
how do you propose reconfiguring the procurement process? Remember, it's got to be fair. You have to figure out how to charge a fair price. Everything has to be documented, so that it can be audited by the bureaucrat in a few months and given a second rubber stamp. Also we have to score the service providers, don't forget to ask them if they are minority-, veteran-, or disabled- owned, because each of those gives them +20 points.
The article mentioned only one side of the problem: the acquisition process. The government must follow the Federal Acquisition Regulations (FAR - http://www.acquisition.gov/far/) when contracting purchases of equipment and/or services. The FAR is not agile and is so strict that you CANNOT already know what you want, you must make the contract as generic as possible. Especially on large contracts (over $5M). For example, you can't say that you need laptops with Intel Core i5 processors because that would unfairly exclude other processor vendors from competing.
The other side of the problem, arguably the bigger side, is that most of the people in the acquisition process do not know how to write a contract that is flexible and really gets the government what it needs. For such a large project as the health care website, the people at Health and Human Services most certainly did not know how to articulate what they wanted. They could've hired a company to help them determine their requirements and write a proposal, but that's a significant (and long) undertaking itself.
In short, government systems will never be user friendly, scalable, or secure because we don't know how to acquire such a system.
I thought you can get around excluding vendors just by specifying generic requirements that only one vendor happens to fulfill. Like "must have inter-processor bandwidth of at least Xgbps" or whatever generic-yet-specific thing you can find.
Are you serious? Is there a perl implementation of python? Did I mention I am having to formally present to the executive steering committee to get permission to run perl?
Hah. Sounds terribly bureaucratic. But yes, I'm serious. It may not be trendy, but there's nothing Python can do that Perl can not. The differences are fairly cosmetic.
The key is to write good quality Perl, so it's fun:
1. Maybe they know of vulnerabilities that you don't. Remember the bug in Debian where cryptography was being seeded by the process id?
2. They might have put a lot of time into ensuring that one particular product was 100% secure and trust only that one. This is often the case for hardware, where vetting for security compliance can be difficult.
Are these rules in force on any host connected to the network? Do they have different networks with differing rules? How are the rules enforced? (That is, I can imagine switches and firewalls enforcing network rules, but "don't install python"?)
They do have different networks with different rules. When I was on the Michigan DOD network, we had the Microsoft Document Image Viewer. When I was deployed, the DOD network there did not have the Microsoft Document Image Viewer on the "approved" list and therefore could not be installed. It was such a pain working with TIFFs without it.
It should be noted that healthcare.gov might be the highest traffic website launch in the history of the internet. With millions of visitors on day one, I certainly can't think of one that would even compare.
They are dealing with scaling issues that are almost completely unprecedented. Every other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up.
Given this, I can't help but think that 90% of the scorn heaped upon healthcare.gov is politically motivated. When Twitter spent YEARS going down on a regular basis, or Simcity just completely shut down on launch day (with a fraction of the users of Obamacare), nobody demanded the companies that running them be shut down. Shit happens when you build a website that gets incredible amounts of traffic. It will work out just fine.
> It should be noted that healthcare.gov might be the highest traffic website launch in the history of the internet. With millions of visitors on day one, I certainly can't think of one that would even compare.
I can think of two that I've worked on personally:
- Yahoo!'s tribute site for the first anniversary of 9/11
- Ticketing for the 2008 Beijing Olympics
Yes, in both cases, the companies had experience & infrastructure for building high-scale website, but that should have been a requirement for the entities building healthcare.gov as well.
In both cases I've cited there were a lot of significant changes to the technical stack (e.g., it was first time we used PHP for a large scale project at Yahoo!).
Simcity is exchanging orders of magnitude more data between each client and the server with much stricter latency requirements. Same applies to Twitter (though the data density is less than in a game there are more connections and the volume of data is higher). I don't see how a website that serves static (in the sense they do not change over time, like twitter feeds or game sessions) pages is comparable to either.
Every other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up.
And how much those months or years of experience were leveraged in figuring out the touchy parts of the infrastructure? This shouldn't have been the first rodeo for anyone involved. There are numerous blog posts, books, videos, etc with ideas, what works, and what doesn't, for running websites of all sizes. Many of them are posted and debated on this very site.
More generally the problem is a lack of standards in technology management.
Malpractice is endemic in the IT space and nobody, either technologists or managers, are held accountable.
For instance, the issues @vinhboy talks about are "bad smells" that reek of unprofessionalism -- like the primary keys that aren't unique, the predictable race conditions, and all of the problems that recur again and again.
Apparently integrating with "legacy" systems was a big bottleneck (e.g. to verify your SSN). I don't understand why this verification had to be done "live" though; the obvious alternative would be to collect all the data, verify it asynchronously/overnight, and then flag up those people that had issues. You enter your birthday, SSN, etc and then the 1% of people that got their details wrong get an email the next day saying "we weren't able to verify your details, please log in and fix it".
Just saying there were like 20 articles about the all star obama campaign tech team that built a super scalable cloud infrastructure and ab-tested, email spammed, and narwhal'ed their way to victory. Couldn't get any of them for the actual governing?
1) Our government has a huge set of rules about procurement. Who's allowed to get their bid approved, the right kinds of projects, who's allowed to manage them, who's allowed to set project requirements (different than the former), who's allowed to pay them, etc. In theory this is all really smart and awesome: you don't get someone giving their brother-in-law a swell multi-million dollar contract, physically impaired people can access your site, etc.*
2) Governments and campaigns face different incentives. Campaigns don't have many cross cutting concerns: the goal is to get a candidate elected. Sure, some high powered consultants are simultaneously angling for a new job after, but IT and most staff are relatively immune from that. Your job is to get something done, and you want to get somethng done. Government, though, has no such incentives: you can do shitty work and not face consequences, and the mission is inherently less aspirational and more shoveling shit. Much of the time government bureaucracies also double as a social welfare program, or a private welfare program if a well-connected corporation has a well-placed surrogate legislator or bureaucrat.
*Reality, needless to say, can turn out a bit different.
"One might look at this and go: Why can't we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?" Johnson says. "The problem is that the way that federal contracting works is so burdensome that the only people who get contracts like this are experts at lobbying and experts at regulations that require you to get these sorts of contracts. And they're not experts at doing the job of building these websites."
No, quite literally not. The campaign team is a private organization, not subject to rules for hiring federal employees or spending tax money.
This is precisely the sort of shit that make people believe that the government is incompetent. It really is for certain undertakings that require some degree of operational flexibility.
Wait, isn't the government the one who makes the rules? So if they made the rules that makes them incapable of doing things that they are supposed to do, while the people, technologies and ability to do the same thing is readily available and the same thing can be and is done by others - isn't that a very definition of incompetence? Not doing what is perfectly doable because of self-inflicted inhibitions?
It doesn't "make people believe", it is incompetent. And, unfortunately, the very capable and competent people are working hard at making it worse by electing exactly the people who would perpetuate and expand the problem.
Ha, sounds a lot like most big companies. Why get and train competent staff to build a software development process when you can pay 10x as much to a vendor for software that doesn't really work.
Even if they could theoretically make a site that can handle every person in the country using it at exactly the same time, it'd still be bottlenecked by the much older systems it relies on at the IRS, SS, VA, DOD, DHS, etc. As the article alluded to, the Healthcare.gov site checks your family size, AGI, social security benefit status, citizenship status, incarceration status, veteran status and enrollment in other government health programs during the signup process to determine what you're eligible for.
This. The article sorta pines in the end for smaller "more agile devs", but I'm not sure a small shop could just come in and bust this out. There is SO MANY legacy systems that need to interface with this beast. I wouldn't put it past the designers to have been rate-limited at 50,000 whatevers by one of the vendors of the other systems that they have to talk to. Computers from the IRS have been hanging in there sometimes as far back as 40 years ago. I'm sure systems at many other of the lettered agencies are hobbling along just as well.
I totally agree that the government needs to become more lucrative to smaller more agile groups. I think it could ultimately get better products at lower prices if it opens up bidding. That said, the government has massive needs and massive legacy systems that many small shops may have a hard time reconciling. There's always ways to theorize about how the government could standardize and modernize its disparate stuff, but that requires strong leadership, a good direction, and money. None of these things appear to be in abundance right now. It's really unfortunate.
I tried to check prices on CA site on Oct 1. I never tried to sign up, so the system actually didn't have to go to IRS, DOD or DHS - I just wanted to see the plans offered. In fact, it had to do no more than look up a few tables and do a couple of basic calculations. Still, the site was as broken as one could be - about 75% of requests just timed out. The rest were presented in broken design and half-loaded pages. Navigation was completely broken. I spent couple of hours and many tries and retries until I could actually see how much an insurance plan would cost me. And LA times reports[1] the site has had 645K hits for the whole day. DOD, SS and VA is not the problem here - the site just had a crappy implementation.
Because it can be, and doing so provides a better experience. If you have the chance to design a system from scratch, you're going to want the best design you can afford, right? So why not create a marketplace where buying health coverage is as simple and straightforward as buying speakers on Amazon -- a list of products with prices and a buy button.
The asynchronous alternative is a menu of plans you may or may not qualify for, at prices that may change after you apply, based on inputs many people will get wrong (unintentionally and otherwise), with a true cost that won't be known until someone does your taxes up to a year later and computes what subsidies you qualified for.
Even if there's only a day between signing up and verifying your information, that turns an hour of "getting it done and over with" into a multi-day ordeal. Considering this is a mandated purchase for a lot of people, they want to devote multiple days to it about as much as they want to stand in a long line at their local DMV.
I don't think any of the questions being asked were particularly hard: certainly no harder than doing your taxes. The system could clearly say "based on the information you provided, here are your options".
I know it's nicer if it's instant, but I don't think anyone expected it to be instant; private health insurance is a multi-day process. Nobody is getting this health insurance until January. It sounds like the system is asynchronous anyway, in that it then goes off for further processing (no matter what the website promises).
In any case, we're talking about a delay of a week at most; I'm not suggesting that nothing be verified until tax season.
Because it's giving the user real-time access to which plans they are eligible for.
The only other two options I can think of would be:
-- Pre-verify all of the potential applicants, many of whom may already have coverage and never come to the exchange.
-- Ask users a bunch of questions that they may or may not know the answers to.
I really like the suggestion of doing the calculation for everyone in advance. We could even have sent a letter to everyone that would be better off if they enrolled.
Its a big database, but handling a db of that size isnt that hard of a problem (if you precomputed all results, and then threw the underlying details away because HOLY SHIT FIRE).
If you batch processed the delta from the other agencies every night (or as soon as they could if better), it could work!
Compare this with the medicaid "part D" government website launched in 2006. It was designed around an anticipated load of 20,000 simultaneous users so they scaled it to be able to serve 150k users. Anyone who builds websites in 2013 without understanding the spiky nature of traffic or the necessity to build in capacity overhead is incompetent, end of story.
The amount of money spent on building healthcare.gov (many millions) just makes that incompetence even more infuriating.
Being handed impossible constraints (serve 250k simultaneous users, verify them all with the IRS, but the IRS service you have to talk to can only handle 1000 simultaneous requests) isn't incompetence on the developer's part, and we don't know whether something like that isn't what's happening. The team that built this site (which has had no unintentional downtime, latency or server errors -- whatever code they wrote hasn't fallen over, it's just stranded most people on a "please wait" page when they try to register) has no power over systems at all the other agencies they have to work with.
Impossible? That's just a constraint. If you have a dependency on some other service that is highly rate limited then you work around that, using any one of a kajillion different techniques. You don't just drop everything on the floor, throw up your hands and blame everyone else.
I think 50,000 concurrent users is actually a fairly reasonable number, given that not all US households are in the market for health insurance on the exchange, and that not everyone will be using the site at the same time.
Concurrent, perhaps. But a thing like this? You're going to see spikes of many, many times more than that opening [insert time period]. See also: the Slashdot Effect, the Digg Effect, et cetera.
The real scary part is that we entrust these people with our most private information as well as decision making matrices that might very well determine what, when and if you receive the appropriate care.
To me these incidents, the broken procurement systems, the mediocrity (or worst) that permeates government and many other factors only serve to reinforce the idea that we need massively less government reach into our lives, not more. Frankly, the combination of healthcare and the IRS scares the living daylights out of me.
Blaming IT is an easy scapegoat. The notion that the Feds and various states should be operating marketplaces in the first place is absurd on its face. California and New York aren't subject to federal procurement guidelines, and their systems are basically non functional as well.
Government is successful with complex public facing IT in areas where the processes are well understood. The DMV, tax collection, and unemployment are good examples in most states. Medicare billing is a success story in the federal space. Most other things are not very successful unless they are limited in scope.
The whole system is a screwed up farce. "Obamacare" is a flawed system to begin with, to the point that nobody really understood what was needed for months. So the Feds and states were forced to roll with some highly generic requirements and specifics were made up as they went along. That scenario is a dream for Accenture (Cali) or CSC (ny) to churn hundreds of millions of dollars.
Can you elaborate on how selling an insurance policy is not "well understood" or how it's inherently more complicated than administering the tax code (!) or running medicare? I don't follow that at all. Insurance has been sold successfully for well over a century. Hell, governments have been running and regulating markets for all sorts of things (many of them much more complicated than a handful of insurance plan choices!) for the whole of modern history.
That doesn't mean that this site isn't a mess, but I don't see where the "screwed up farce" bit comes from. I rather suspect it's driven more by your choice of political tribe than any real intuition about what the government is capable of.
Hilarious considering that Medicare had far more serious problems throughout its inception and you consider the billing component to be successful.
Like all website launch problems in the months and years from now people are going to forget they ever happened. And instead will find having an independent marketplace to be incredibly useful. The states that have exchanges have already seen insurance premiums drop significantly.
I keep reading anecdotal stories on the Internet about premiums dropping significantly but I, some in my family and a friend have recently experienced the opposite.
I just compared premiums for essentially the same plans (3-person family, no subsidies, California) on http://www.ehealthinsurance.com/ (old insurance, 2013 plans)
and on http://coveredca.com/ (new ACA plans). The former are roughly 2x cheaper that the latter for comparable plans (cheapest one ~350 vs ~750). I also read other reports [1] of costs increasing. Could I see which reports of premiums dropping significantly - e.g. in CA - you are referring to?
Huh? Medicare billing is a successful program that works efficiently and well. Have their been issues with the Medicare program over the last 50 years? Yes. Most of the serious ones are policy problems.
I actually know someone who has worked for the VA in medical coding contracts. He still has Cobol programmers on his speed dial because some of the live code is still written in it.
Good luck changing the procurement process. The UK Gov has been trying to get more SMBs involved in UK Gov procurement[1] in hardware and software is failing pretty miserably at the moment.
> In order for the exchange marketplace software to check on which coverage and subsidies you're eligible for, it has to ping requests to the ... Homeland Security data
To what degree is the sheer complexity of the underlying government legislation responsible for cost, delays and complexity in the IT systems government pays to have built?
No wonder we are going broke. It's nothing but special cases, all the way down.
For those wanting deeper technical analysis around the problems and building off of some of the arguments Clay made about procurement, check out https://news.ycombinator.com/item?id=6518739
No. But the company that built the working sections of healthcare.gov - that is, everything except /marketplace* - and open-sourced it (https://github.com/CMSgov/healthcare.gov) is Development Seed. They are indeed the makers of MapBox.
As someone who has never worked on a huge site, I don't want to naively say that it could have been done better. But I feel like those are basic issues that could have easily saved them a lot of bandwidth.
Also, it seems to me that if they had so many issues on the front-end, it's a hint that maybe they also had many similar issues in the backend as well.