
IBM to pay more than $30m in compensation for Australian census fail - timv
http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-in-compensation-for-census-fail/8057240
======
stirlo
Before census day they were predicting two thirds of the 10 million Australian
households to complete the census online. They stated their site had been
tested with half a million concurrent users and was able to sustain double
that.

Unfortunately no one seems to have considered that the majority of users would
do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the
time frame and it was never going to work with a 1 million per hour capacity.

What is interesting is with IBM paying out they must have failed to deliver
the required capacity. I thought it more likely the government had not
tendered it properly and requested a lower capacity than needed but obviously
IBM just tried to cut costs and under resource the operation.

~~~
candiodari
I often wonder why companies and governments even trust large organisations
like IBM over individuals they've interviewed themselves. I mean, it's not
like IBM's consultant's competence is any secret. IBM is also trying to make
it worse, or at least, not making the competence factor any priority. Rather,
it's cost over everything:

[http://www.businessinsider.com.au/ibm-layoffs-1-month-
severa...](http://www.businessinsider.com.au/ibm-layoffs-1-month-
severance-2016-3)

I mean how do you get away with hiring IBM (or accenture, PwC, didata,
deloitte or, may God help us, Tata or something like that, where cost of
paycheck is literally the only factor in hiring).

~~~
ACow_Adonis
Such individuals are usually unable to detect real signal in the thing they're
hiring for, so interviewing in and of itself doesn't provide them any
qualitative information.

They also know the people judging them are in the same boat.

Thus they fall back to the only thing understandable to themselves and people
who will judge them: big brand and prestige/shininess. If the project
succeeds, you take credit. If it fails, you can't be blamed for "buying the
biggest and the best" (even if the biggest and the best are just a bunch of
stary-eyed ignorant grads with a salesman/project manager at the head).

/that's the good version. The bad version points out how many of these firms
gets business through networks, personal connections and "you scratch my back
I'll scratch yours" type dealings. //it's not a rant if it's true.

------
cyberferret
That's a pretty hefty fine, but I wonder what the original billings were for
developing the system in the first place?

I've heard reports that they spent >AUD$5Million on just the load testing
aspect, but I hope someone can verify that. And I trust that they have
improved comprehensive testing methodologies that actually work to improve
reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus
Notes??</sarcasm>)

I guess we can be thankful that is was 'merely' a census and not something
like electronic voting!

~~~
unsignedqword
In the US, Census data is used for redistricting purposes, which CAN influence
election outcomes in the future - so the implications of a census may be wider
than you think. I don't know exactly the way it works in Australia, but some
googling has led me to believe they use a system that is similar.

[https://en.wikipedia.org/wiki/Redistribution_(Australia)](https://en.wikipedia.org/wiki/Redistribution_\(Australia\))

~~~
Haydos585x2
Another major problem with the census was the Australian Bureau of Statistics
holding onto the data and changing their privacy policies which meant that in
an act of protest a ton of people entered false information or ommitted
details. A really poor census overall.

------
dghughes
Just wait until IBM's Phoenix Pay system and the scandal here in Canada comes
to a head. With 300,000 civil servants waiting six months to be paid and it's
still not fixed.

There must be a big penalty looming for such a massive level of incompetence.

Phoenix seems like a rather awkward name for an application.

[http://www.nytimes.com/2016/11/18/world/canada/government-
wo...](http://www.nytimes.com/2016/11/18/world/canada/government-workers-
pay.html)

>Though many government employees work in 9-to-5 office jobs, the public
payroll also includes a wide array of people like Mr. Ryan with complicated
work schedules and pay rules.

My father used to be in the Coast Guard he worked 28 days on and had 28 days
off. If you were injured that meant the days you were off were really two
days?? Try explain that to some bureaucrat in Ottawa who can thinks they are
speaking English (really French), they're 2,000 miles away and doesn't
understand the system either.

~~~
adambrenecki
Oh God, that sounds an awful lot like two more Australian systems: the
Queensland Health payroll debacle, where a bunch of health workers were paid
incorrectly or not at all. Or the National Disability Insurance Scheme
debacle, where healthcare providers haven't been paid, resulting in small
business owners having to take out huge amounts of personal debt just to pay
their employees, and sole traders unable to provide services their clients
need to live day to day while still keeping a roof over their own heads.
(They're both IBM projects too.)

~~~
shakna
Do you have a link for IBM's mishandling of the NDIS? Interested, but can't
find a link.

------
cyberferret
Availability and load balancing aside, did any other Australian citizen (who
actually managed to complete the census at all) notice that the UX was quite
ordinary? Reminded me of a web form circa 1990.

For instance, IIRC the census asked for my and my wife's country of birth and
nationality, but later on when filling in the info for our children, it asked
for the nationality of the both parents again. Simple logic checking back
through the data should have shown that our (the parent's) information was
already supplied, and this information for our children could be pre-filled or
skipped and auto filled in the database.

Just a few small things that would have made the filling out of the forms a
lot smoother, faster, and less complicated for lay people who were already
nervous about doing the data entry.

~~~
jestar_jokin
Except you may have step-children, adopted children, foster children, children
in the house not part of your family (friends of your child, children of your
friends), etc. Keep in mind that the census needs to include everyone who was
in the house at the time it was filled out, not everyone who is part of your
regular abode/family situation.

You might argue that it would work for the "most common scenario". In which
case why not pre-fill all the data to the statistically most common values?
(Hint: the census's purpose is to collect those statistics.)

Also, pre-filling might cause confusion as to which fields have already been
filled out by the user; if a value is present, they might wrongly assume no
action is required on their part.

(Having said all that, I haven't seen the online census.)

~~~
cyberferret
Sure. I did consider the scenario if the kids had a friend sleeping over that
night or something, or if the respondents had adoptive or foster children
etc., but I think when I traced back the line of questioning in my case there
seemed to be a logical thread link there that could/should have negated asking
for the same information again.

I am relying on my memory here from few months ago, so I could be wrong, but I
did feel at several points during the session that quite a few questions could
quite possibly be chained together in better fashion. Perhaps not so much a
failing of the actual site UX designer, but the census analysts who designed
the questions in the first place?

~~~
ACow_Adonis
Not responsible for this census, but I've done work with it in the past. Two
observations:

The questions are largely derivative and the same each year: cost to develop
and test while maintaining statistical validity and capture of all possible
scenarios, while still being interpreted correctly by the population, is
enormous.

Secondly, the number of permutations and combinations of answers is almost
universally underestimated by everyone I've ever met outside of this process.
Nationality is not the same as birthplace is not the same as parent is not the
same as adults in the household. The amount of diversity is staggering.

Additionally, although those two are the primary reasons for the question
designs (and I assure you, people will be doing deductions and logical
inferences on the back-end once the data is collected), being exhaustive at
the point of data entry rather than deductive allows for further meta analysis
on the data entry process and survey responding experience itself, whereas if
you decide to attempt to deduce, you miss out on a dimension of typo, misskey,
misinterpretation, and other analysis data that you would forgo in the former
case.

~~~
cyberferret
Thanks for your viewpoint from the 'sausage factory', so to speak. I am
beginning to understand the enormity of trying to combine ease of use, with
the effort to gather statistically valid data.

------
bitmapbrother
My only disappointment is that they didn't have to pay more for their
incompetence. IBM has been on a mission to replace their workforce with cheap
labour from India for quite a while. I'm glad the work they deliver to their
clients is finally being exposed for what it is - junk code with no quality
control.

------
justinsaccount
I wonder what the infrastructure provided by IBM consisted of.

Based on non-cloud setups like [http://nickcraver.com/blog/2016/02/17/stack-
overflow-the-arc...](http://nickcraver.com/blog/2016/02/17/stack-overflow-the-
architecture-2016-edition/)

A rack or two would have been able to handle the load.

~~~
will_hughes
You have to understand that what StackOverflow gets away with in terms of
hardware is not at all reflective of what other shops will be able to do.

SO can work on such a small amount of hardware because they've got people who
either know the entire infrastructure from ground up, or have the skillset and
interest to figure it out. Plus they have very tight coordination between the
folks doing the development, and running it day to day (possibly because it's
the same people).

Your average developer working on a government contract through
$BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset
or motivation to learn it. Even in the unlikely scenario that they did,
they're not going to be in a situation where they can act on it. It'll be a
case of developing to specification, throwing stuff over a wall to some sort
of QA, and then throwing the result of that over to someone to run it. The
whole thing will be run by project managers who have opex budgets to meet, on
infrastructure that was specified well before the thing was ever built.

------
beedogs
We requested paper forms long before census night, because we were pretty sure
this was going to happen. Didn't put our names or any other personally
identifying info on them, either. (Never received a fine.)

------
jacques_chester
The Prime Minister is correct in saying that it was a predictable debacle.

Because it's bloody hard to DDoS paper forms. Widely distributed atoms are
curiously disinterested in IP packets.

------
sqldba
Ummm the article isn't very clear.

It says the damage was estimated at 30 million. It says IBM paid something. It
didn't explicitly say IBM paid $30 million.

~~~
SyneRyder
First line of the article:

"Computer giant IBM will pay more than $30 million in compensation for its
role in the bungled census"

