
IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware - artsandsci
https://www.nextgov.com/it-modernization/2018/04/irs-60-year-old-it-system-failed-tax-day-due-new-hardware/147598/
======
raverbashing
The question for the industry remains: how do we deal with systems that should
be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as
they exist today 50 years ago.

How do we separate business logic in a self-contained way that can be
independent of hardware and (from a certain level) software?

~~~
romaniv
_> The question for the industry remains: how do we deal with systems that
should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have
computers as they exist today 50 years ago._

Is this a question the industry actually cares about? If anything, I'd say it
got much worse at creating long-living software. It sucks to run a mainframe-
based system from 50 years ago, but running a 10-year-old Web Forms
application can be borderline impossible. I'm scared to think of what will
happen with all the cloud-based stuff in the next 10 years.

 _> How do we separate business logic in a self-contained way that can be
independent of hardware and (from a certain level) software?_

I often say that there is no such thing as "business" logic. There is just
logic. The least maintainable systems I've worked with were the ones with the
most sophisticated attempts at abstraction.

In my opinion, the key to software longevity is making systems that allow for
local reasoning, i.e. analyzing parts without knowing the whole. This can be
achieved in a lot of different ways, but it needs to be a deliberate goal in
system design.

~~~
raverbashing
I agree it's a tough cookie. I actually don't think layers and layers of
abstraction is a good idea.

But something were you can describe something like "Read this value from this
field, do some math based on fields in another table, return the result"

That's why I raised the question, how do we do that over long periods of time
without rewriting it in several different ways? (ASM to flat files, COBOL and
JAVA to different SQL flavours, what else is next?)

And if you migrate, let's say from Oracle to MSSQL it will work for 90% of the
things, for the last 10% it will be a PITA.

------
cptskippy
I feel like the authors of this article are being very misleading by using
broader terms where they could be more specific, like using system where they
could instead say software. The IRS isn't running 60 year old hardware, and
the failure was in hardware that was just 18 months old. But because the IRS
is running software that is 60 years old, the authors can claim the "system"
dates back to the 1960s.

It's also clear the authors don't entirely understand what they're talking
about. In the second article linked to, which is written by an author of this
article, he states "[t]he Individual Master File, a massive application
written in the antiquated and low-level Assembly programming language" as if
to imply there is but one assembly language.

A GAO report I found chasing the last link in the article indicates that the
two oldest systems in the Government are the IMF and Business Master File
(BMF) which it says both run on IBM Hardware. My guess would be that the IRS
is running modern z/Architecture mainframes in System 360 Compatibility mode.
The System 360 dates back to the early 60s and IBM has maintained backward
compatibility with it all the way up through z/Architecture systems. This
would also explain why they're so far over budget and behind schedule.

~~~
chx
The oldest parts are not System 360, they run IBM 7074 assembly. Just FYI. The
caching system is much newer, they have some Java proxy, there were posts
about it the last year.

~~~
DonaldFisk
The manual for the 7070, which was similar, is here:
[http://www.bitsavers.org/pdf/ibm/7070/A22-7003-01_7070_Refer...](http://www.bitsavers.org/pdf/ibm/7070/A22-7003-01_7070_Reference_Jan60.pdf)

The instruction set documentation starts at page 42.

~~~
chx
The manual... and it's less than 300 pages... oh boy, I remember the System
3090 manual, it was several metres thick :D (in several volumes, of course)

------
otoburb
>> _Still, the crash forced the IRS to extend the tax filing deadline one day,
delaying some 14 million submissions._

For a system that is supposedly so brittle, they were able to get enough of it
back up and running to at least accept submissions _within one day_?

That's pretty amazing for 60 year old technology.

~~~
c22
You just have to get the stuck card unjammed, then repunch it and recollate...

~~~
thaumaturgy
...and it was the re-collation step that took a dedicated team of
professionals a full day.

~~~
varjag
To be fair, 140 million taxpayers account for a lot of punchcards.

~~~
Elv13
(assuming it is not /sarcasm)

The data is not stored in punch cards, the programs are/were. The data is/was
stored in tapes or in tape emulators.

Storing data on cards would make random access wayyyy too slow (the operator
would have to retrieve and load the card and keep them sorted manually, which
is impossible to do at scale)

On tape, however, you can predict the distance you have to rewind by computing
the physical length each records take if they have auto incremented IDs. A
tape can contain a _lot_ of records. Especially if it's a modern one or an
array backed emulated one. At least that's what some people I went to school
with told me it's how they did it.

~~~
rbanffy
I did a bit of stuff with data coming in on punchcards. You get a batch of
punchcards, run it through a program (which can load data from tape or other
punchcards) and a new batch of new punchcards with updated data come out on
the other side, along with an impressive pile of printed invoices.

Wide adoption of tape was a lifesaver.

------
sehugg
_A delivery date for CADE 2, the IRS’ subsequent modernization effort, has
slipped several years even as contractors working on the project have earned
as much as $290 million._

They call them "Beltway Bandits" for a reason.

~~~
empath75
As someone working on a modernization contract, the problems are more
complicated than just contractors standing on shovels.

Building an app to a complex and ever shifting spec is a much different
process than the way that startups work. You can’t just release a half done
app to friends and family. You need to do the whole thing at once, with 100%
if the functionality and it has to be ready for everyone in the country to use
it in day 1. And the federal government has fairly onerous requirements for
getting stuff through security— you can’t just google for random shit from
GitHub and expect to put it straight into production.

I can only speak to my particular project, but everyone on it is working hard
at getting it done on time and on budget, using modern deployment and
development practices, but there’s limits to how much you’re allowed to just
move fast and break things.

------
favorited
Fun (scary) fact – significant portions of the IRS tax system is still written
in assembly. It's basically the original codebase plus 40 years of additions
and patches as the tax code changed.

They were working on projects to "elevate" it to Java, because it's pretty
difficult to maintain it as-is.

~~~
_RPM
Should it be written in Go, or Rust?

~~~
bsder
Because that always works so well ...

See the Chrysler Comprehensive Compensation fiasco:
[https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...](https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System)

These kinds of systems are all about the _exceptions_ and how you enable
humans to deal with those, not the rules and how the computer deals with
those.

The problem is that the system first needs a comprehensive test suite written.
_THEN_ you could actually think about rewriting pieces.

------
johnklos
Something that the kids of IT today will never learn:

Never replace anything unless the thing doing the replacing is better than the
thing being replaced.

The IRS' IT "upgrade" is just a huge money grab. Nobody has the slightest
interest or desire to do anything properly - they simply want in on that
lucrative IRS money.

It's like the saying: "Anyone who is capable of getting themselves made
President should on no account be allowed to do the job." Likewise, the kind
of people who are bidding on this project are certainly not the kind of people
that should ever be allowed anywhere near the IRS' computer systems.

~~~
mbell
Or you can continuously keep your code / systems up to date and never get
wedged into a nearly impossible situation of having to rewrite 60 year old
code (which likely has nothing resembling a test suite nor even a spare piece
of compatible hardware to test on).

~~~
reaperducer
> Or you can continuously keep your code / systems up to date

Easy to say as a grunt, but businesses (and governments) are run by managers
and accountants.

What you describe is upgrading for upgrading's sake. If a system isn't broken,
there's no point or financial reason to replace it.

Imagine trying to upgrade a system with incredibly complex data on a BILLION
users that has to work 24/7, then deal with massive data influx four times a
year plus catastrophic data ingestion one time a year.

And not only take in and sort those records, but validate the data and perform
calculations based on rules that change almost monthly, and that sometimes
must be applied to years-old data and sometimes not.

Plus, you're dealing with people's money. You don't fuck around with people's
money.

It would be easier to merge Facebook and Google's databases.

~~~
mbell
> Imagine trying to upgrade a system with incredibly complex data on a BILLION
> users that has to work 24/7, then deal with massive data influx four times a
> year plus catastrophic data ingestion one time a year.

I think you're over-estimating the amount of data involved. There were about
150M individual tax returns and ~5.8M corporate returns in 2015. Even if we
assume 50 years of storage in the 'active' dataset that is < 8B returns. Also,
most returns don't contain a particularly large amount of data in them. Pretty
sure we're talking about an in-memory scale dataset here, certainly for the
active set in any given year. It actually would not surprise me if we (current
co) ingest more data in a week than the entirety of the US tax return history.

> And not only take in and sort those records, but validate the data and
> perform calculations based on rules that change almost monthly, and that
> sometimes must be applied to years-old data and sometimes not.

I might believe that is an issue if the IRS actually calculated everyones
taxes for them and sent out a 'do you agree' form, but they don't. It's more
like they randomly sample some returns based on some hand picked 'warning'
flags and have someone manually check them, maybe with the help of some excel
formulae.

~~~
reaperducer
>I think you're over-estimating the amount of data involved.

The one billion figure was from the article.

------
BrandoElFollito
What part is complicated when transitioning such a system?

Legacy data which must be transformed?

Old business logic which must be maintained? I do not know if there is a need
to replay old rules vs. just kerping a snapshot

Continuity? Maybe there are so many transactions happening, linked to existing
data that an interruption is complicated?

Risk of accessing old data which is better left undisturbed?

------
olivermarks
'The Individual Master File contains data from 1 billion taxpayer accounts
dating back several decades and is the chief IRS application responsible for
receiving 100 million Americans’ individual taxpayer data and dispensing
refunds. '

I hope they bought a new hard drive to back it up...

~~~
mark212
don't know about you, but I'm hoping they _didn 't_.

------
js8
What kind of hardware was it running on? The article is not very specific
about that.

(At first I thought that they upgraded IBM mainframe to z14, and that caused
the crash, but it's not yet 60 years old.)

~~~
cmiles74
While the code is 60 years old, the hardware is modern. According to this
article, the component that failed is about 18 months old. When they talk
about the software and hardware "no longer being supported", I think they
really mean that the original target platforms are only present in the form of
emulated software. IBM still supports execution of System/360 software on the
System z. [1]

"Overall IRS maintains over 20 million lines of assembly code. These millions
of lines of archaic software, and hardware, that is no longer supported become
more difficult and costly to maintain each year, and poses significant
cybersecurity risks. To IRS’ credit it keeps these old systems running during
filing season, but relying on these antiquated systems for our nation’s
primary source of revenue is highly risky, meaning that the chance of having a
failure during the filing season is continually increasing." \- David Powner,
director of IT management issues for the Government Accountability Office [0]

"[IRS Chief Information Officer Gina Garza] explained the code for the IMF was
written in the 1960s, but the hardware it runs on is modern." [0]

[0] [https://federalnewsradio.com/technology-
main/2017/10/a-sense...](https://federalnewsradio.com/technology-
main/2017/10/a-sense-of-urgency-as-irs-legacy-it-systems-grow-increasingly-
older/)

[1] [https://arstechnica.com/information-
technology/2014/04/50-ye...](https://arstechnica.com/information-
technology/2014/04/50-years-ago-ibm-created-mainframe-that-helped-bring-men-
to-the-moon/)

------
PinkMilkshake
So they need something proven, scalable, highly concurrent, fault tolerant,
and runs in a VM so it's not tied to hardware.

Sounds familiar.

------
eecsninja
Be thankful we're not getting all the government we're paying for. --Will
Rogers

Source:
[https://www.brainyquote.com/quotes/will_rogers_128073](https://www.brainyquote.com/quotes/will_rogers_128073)

------
jgalt212
Can the USDS assist here?

~~~
normaljoe
Last time I talked to somebody at USDS two years ago maybe, they where already
helping. I think the biggest task was trying to figure out getting off tape
and moving normal hard drive storage. Apparently this is hard because the
really old programs are the OS and file system. Without the details it sounds
to me this is what broke or something around that.

------
tomohawk
It's probably easier to just scrap the income tax than to fix the system.
Maybe go to a land tax instead.

------
interfixus
> _[...] date back to 1960, when John F. Kennedy was president_

No he wasn't, so my confidence starts flagging.

------
49bc
When they can write AI to understand and implement the entire US tax system,
then I’ll start worrying about the incoming AI apocalypse they’re always
taking about.

~~~
noetic_techy
More than likely, it will shoot back a question: "Why the hell did you make
this so complex?"

~~~
colemannugent
"A strange game. The only winning move is not to file"

~~~
Nelson69
If only it were that simple...

Not filing or, worse, making an error in your filing makes it a really really
nasty game. No recommended.

I'll even break my own rule: "Not filing considered harmful"

------
sunstone
Next time they do this they might want to trying testing out the new system
before putting it into production.

------
ams6110
> Since Republicans gained control of the House of Representatives in 2010,
> their partisan attacks have left the IRS with nearly 10,000 fewer customer
> service representatives to assist taxpayers and a patchwork of IT systems,
> some dating back to the Kennedy Administration, which is ultimately harming
> all taxpayers,” Rep. Gerry Connolly, D-Va., told Nextgov.

So the patchwork 1960s-era systems are the fault of Republicans in the current
decade?

Maybe "starve the beast" really is a better approach. Nothing else has worked.

~~~
briandear
The real question is why we have a tax system that would need 10,000
additional customer support reps. The problem isn’t a lack of reps, it’s a
system that requires that many reps.

The flat tax, file-on-a-postcard system proposed by 2016 presidential
candidate Ted Cruz was mocked by the establishment because it seemed tha
politicians on both sides have a vested interest in a complex tax code — it is
the means by which politicians derive their power. See Milton Freidman on the
subject:

[https://youtu.be/TruCIPy79w8](https://youtu.be/TruCIPy79w8)

~~~
monocasa
You can stay away from regressive schemes like a flat tax, _and_ have a tax
system with a rubberstamp level of UX. That's how it works in most of Europe
and could work here. The IRS already has all of the information needed outside
of gambling and stocks, and could simply send you forms already filled out for
you to sign. H&R Block and TurboTax simply heavily lobby to keep this from
happening to keep their business model around.

~~~
conanbatt
Whats interesting about the flat taxes is that it makes it hard to avoid at
the highest levels, where complex tax structures end up having a very variant
level of taxation. Much like corporate taxation is 35%, but google or facebook
pay only 11%.

Milton Friedman used to say that a flat tax of 15% would be enough to replace
the entire scheme of income tax. There is also one major boon: lots of capital
is spent on taxes, from accountants to lawyers to gubernamental organizations,
etc etc.

An effective 15% rate without exemptions would fly very well with the right
(simple flat taxes) an the left (no exemptions) but as MF said, the political
stances on this topic dont trust each other and they are both right: even if
we moved to such a system, the right would still advocate for exemptions and
the left for increases in taxation.

Its a noticeable deadlock.

I also want to mention that flat taxes are not regressive by definition. The
progressiveness or regressiveness in that case requires to see where the taxes
are spent.

~~~
monocasa
First off, the difference between regressive and progressive says nothing
about how the taxes are spent. A flat tax _is_ regressive by definition.

Secondly, I'm not a big fan of how you (and quite a few economists) conflate
personal and corporate income taxes. It really muddies the water as even
questions like "what even is income" is different between the two.

Additionally, flat tax doesn't go well with the left. On the corporate end,
exemptions in a lot of ways are the carrot to the sticks of fines and
penalties that we use to regulate business (encouraging positive behavior). On
the personal end, you greatly increase the tax burden on those least able to
bear it, with an increase in benefits being totally orthogonal (most flat tax
proposals intend to be revenue neutral). Someone making $25k nearly doubles
their tax burden with a 15% flat tax.

~~~
RightMillennial
Technically, a flat tax by definition is neither regressive nor progressive,
it is proportional. Federal income taxes and I presume corporate taxes are
currently progressive (with a million and one deductions). Moving to a flat
tax for either of those would be a move in the regressive direction, but it's
still not technically regressive.

Personally, I'm fine with taxes having progressive rates but not the countless
deductions.

~~~
monocasa
_Technically_ it is regressive. The flat amount comes out of rent and food on
the low end of your tax base, and luxury on the high end.

~~~
conanbatt
Thats sales taxes, but flat income taxes would not be like that.

~~~
monocasa
Uhhh, yes it's exactly like that, regardless of what is getting taxed.

Let's say today I make $24,900/yr. That puts me at the edge of the 20th
percentile for 2017. The average effective tax rate for people of my income
was -4.7% (or a tax of -$1170)[0]. You're suggesting increasing my taxes to
15%, taking $4905 out of my pocket as compared to today. Now, since I make
next to nothing, pretty much everything I make is going to rent, food, and
just generally getting by, and I have nearly $5k less real money (or ~20% of
my income!!!) in my bank account to make that happen.

[0] - [http://www.taxpolicycenter.org/model-estimates/baseline-
aver...](http://www.taxpolicycenter.org/model-estimates/baseline-average-
effective-tax-rates-march-2017/t17-0042-average-effective-federal)

~~~
conanbatt
Hah. You are making the case that a flat tax is regressive because current
income taxes are progressive. So moving from progressive to flat is
regressive.

Its a tautological affirmation on progressive taxes being progressive. A flat
tax would make a $24,900 pay $3,735, and if you made $249,000 you would pay
$3735, your taxes increase just as proportional to your income. Its
just..flat.

~~~
monocasa
No I'm making the argument that a flat tax is regressive because the
utilitarian value of a dollar isn't anywhere near linear with total income.

With a flat tax:

* On the high end you're taking from ability to have luxuries.

* On the low end you're taking from the ability to pay rent and put enough food on your table for your children.

~~~
conanbatt
After criticizing my argument that looking at spending of taxes is necessary
to argue if its progressive or regressive, you now propose an argument that
would make progressive taxes also regressive unless they are enough to
compensate the utilitarian value of how the poorer and the richer spend their
own wealth.

The utilitarian argument is for me one of the most practical and also one of
the most scary. Are you ready to exploit the poor if it were proven that it
was economically profitable to do so? I digress. My point is that seeing flat
taxes as regressive is either false or inconclusive, regardless of the desire
of being flat, regressive or progressive in taxation.

