
Public servants sit idle as Australian Tax Office's IT melts down - tankenmate
http://www.smh.com.au/national/public-service/thousands-of-public-servants-sit-idle-as-atos-it-melts-down-20161213-gt9xfd.html
======
sundvor
Ah, the beauty of mission critical IT outsourcing...

1) [http://www.intermedium.com.au/article/hp-enterprise-
services...](http://www.intermedium.com.au/article/hp-enterprise-services-
wins-final-ato-outsourcing-arrangement-contract)

2) [https://delimiter.com.au/2012/06/08/hp-starts-cutting-
aussie...](https://delimiter.com.au/2012/06/08/hp-starts-cutting-aussie-staff-
caspari-ascends/)

3) [http://www.theage.com.au/national/public-
service/thousands-o...](http://www.theage.com.au/national/public-
service/thousands-of-public-servants-sit-idle-as-atos-it-melts-
down-20161213-gt9xfd.html)

This might be a bit of a cheap shot, not knowing the exact failure, however my
bias is that they are related.

~~~
godmodus
nah, tbh i think you're spot on. outsourcing isn't "cheaper" it's also shitty
quality most of the time, which enables those cheap prices.

it's pretty irresponsible and downright dumb.

~~~
nickff
I think outsourcing was a very intelligent move; the bureaucrats kept their
jobs, got more money, and less responsibility.

------
technion
There are several examples of major 3PAR incidents in recent firmware:

[http://3parug.com/viewtopic.php?f=18&t=2309](http://3parug.com/viewtopic.php?f=18&t=2309)

[http://3parug.com/viewtopic.php?f=18&t=1975&p=9092](http://3parug.com/viewtopic.php?f=18&t=1975&p=9092)

Let me put this to you: If a "firmware update" involves copy pasting a several
KB bash script into a putty session, how surprised are you to see software
bugs?

------
tyingq
This old press release[1] suggests the issue might be with HP StorageWorks
P9500 storage arrays.[2]

[1] [http://www8.hp.com/us/en/hp-news/press-
release.html?id=94895...](http://www8.hp.com/us/en/hp-news/press-
release.html?id=948957#.WE_8iIgrJhE)

[2][http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr...](http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c02602824)

Edit: A newer press release (Nov/2016), showing the HPE 3PAR deployment. If
the comments below are correct, and it was 3PAR that failed, they sure weren't
in for long. [http://www.crn.com.au/news/ato-ripping-out-emc-storage-
and-m...](http://www.crn.com.au/news/ato-ripping-out-emc-storage-and-moving-
to-hpe-3par-441262)

 _" The new solution introduces many improvements, including faster
processing, reduced costs and significant simplification to the storage
environment."_

Ouch.

~~~
Namidairo
Other sources are suggesting that it was a HPE 3PAR that failed.[1][2]

HP are apparently calling it the first unit in the world to fail, but
apparently one in Kings College London kicked the bucket too. [3]

[1] [http://www.itnews.com.au/news/hpe-storage-crash-killed-
ato-o...](http://www.itnews.com.au/news/hpe-storage-crash-killed-ato-online-
services-444490) [2]
[https://www.hpe.com/au/en/storage/3par.html](https://www.hpe.com/au/en/storage/3par.html)
[3]
[http://www.theregister.co.uk/2016/10/19/kcls_strand_data_cen...](http://www.theregister.co.uk/2016/10/19/kcls_strand_data_centre_down/)

~~~
module0000
I've experienced a HPE 3PAR failure, but HP insisted we can't call it "data
loss"(or make a claim as such), since the data wasn't lost/damaged. Instead it
simply stopped exporting our LUNS, and the management interface became
unreachable. Ie..."your data isn't lost, you just can't get to it!" would
paraphrase their response to us.

Ended up returning to a large keepalived+iscsi+softraid linux storage cluster.

edit: typo

~~~
dekhn
Availability and durability are two distinct things. HPE maintains staff that
can recover the data through extreme measures, so they were technically
correct.

~~~
lb1lf
...'recover data through extreme measures' reminded me of this little nugget
from the comic strip User Friendly:

[http://www.userfriendly.org/cartoons/archives/97nov/uf17a199...](http://www.userfriendly.org/cartoons/archives/97nov/uf17a199.gif)

Sometime, somewhere, someone's had to resort to this recovery method, I am
sure.

~~~
mikestew
_Sometime, somewhere, someone 's had to resort to this recovery method, I am
sure._

Right before they went out of business. Years ago (like 15-20) I heard a stat
about companies that suffered a catastrophic data loss (lost the drives, no
usable backup). I don't recall the exact number, but something like 75% of
them were out of business within 12 months.

~~~
lb1lf
Yeah, think I saw the same statistic - and I believe it.

Tangentially related anecdote - years ago, the smallish company I worked for
had one - ONE - person booking our travels.

Seeing as we were basically living out of toolbags at the time, trying to cash
in as much as we could on a boom in a market in which we enjoyed a near
monopoly, the office was often manned by the travel booker, our IT guy and an
accountant.

All good, until the IT guy figures he can use his script-fu to automate
configuration of new computers.

He made some mistake in the line having the computer decide whether it was due
for a reinstall or not, leading EVERY computer checking in on the intranet to
proceed with formatting its /usr partition.

So - travel booker arrives at work, boots computer, loses data. And, as a
consequence, has no idea where anyone are anymore, much less any idea when she
ought to get us back home or onto the next job.

Anyone checking in via VPN found their laptop bricked in seconds.

She claims it was the most surreal experience of her life, getting concerned
phone calls from all over the world from field grunts asking what had happened
- oh, and could they please be allowed to go home?

I was one of the lucky ones; I was on Guam with an agreeable climate, sitting
on a crate of beer (parting gift from the vessel I'd just disembarked) and in
no hurry to get home.

A colleague who was stuck in Nuuk was somewhat less impressed.

By no means a catastrophic data loss, but still...

------
m-p-3
Let's all drink and hope for the best for these sysadmins hard at work on
this.

------
amarant
Never did like HP, this article sits well with my confirmation bias...

~~~
krona
I was expecting it to be about IBM. Imagine my surprise.

~~~
godmodus
right now all major hardware companies bar nvidia and AMD are in utter shite
and on a shit producing roll.

maybe they'll learn from this.

~~~
tyingq
I'm not sure it's their fault. Demand for "enterprisey stuff" fell
dramatically once the data centers started buying mostly commodity things.

High end storage doesn't have quite the margins that it used to.

------
cyberferret
A million gigabytes of data lost and NO taxpayer information was lost? I'll
call their bluff.

Still, impressive that they can restore that inside of two days (according to
their estimates).

~~~
shakna
If HP have actually followed the ATOs data retention policies, and if it was
just two SANs going down, then theoretically this could just be a hardware
failure where all the data is safe.

But I doubt HP did follow the policies to the letter.

I hope the ATO are incentivised by the loss of revenue to push for high
penalties, but I see them paying HP to fix something that they likely could
have anticipated: back pressure on the backup machine.

------
fencepost
"Lost" is probably headline overstatement. "Loses access to" would make much
more sense and would mesh with a single point of failure.

Since the article indicates that the failover system was also affected they
may have triggered a bug in the storage systems or it could be something
completely different like "someone let the magic smoke out of our bespoke
networking equipment and we didn't realize the closest spare was disassembled
in HP's network engineering lab in California."

~~~
manicdee
The data was corrupted on storage. It was lost. Then when switching over to
the backup hardware it got corrupted too.

No taxpayer's private data was compromised though, since third parties can not
compromise the data because it is corrupted.

------
dx034
At least losing in this context doesn't mean that someone else has the data
now.

------
nthcolumn
The headline makes it sound like they are doing nothing about it but in fact
it refers to the end-users.

------
unknown_apostle
Anybody any idea on how the ATO has managed to collect petabytes? Must include
audit logs and disk images etcetera, but even then...

~~~
spangry
I used to work there. The scanned returns probably make up a bit, but they
simply have lots of data (e.g. last 10 years of tax returns for every
taxpayer, bunch of data matching DBs, enormous maze-like website, maps to
buried treasure etc. etc.)

And you're correct, their audit programmes all produce shitloads of data, all
of which must be retained for legal record keeping (Archives Act).

EDIT: come to think of it, the biggest space eater would be scans of documents
sent in by folks who have been audited...

------
spangry
I just pray they don't outsource their voice signature database. Imagine if
someone got their hands on that...

------
warrenm
Got to "HP" and stopped reading

