
America’s Cities Are Running on Software from the ’80s - eplanit
https://www.bloombergquint.com/businessweek/america-s-cities-are-running-on-software-from-the-80s
======
bigpicture
A few years ago I had to update some FORTRAN code written in the 1980s that
did bond valuations and some other financial stuff. It hadn't been touched
since it was written and was now running out of space because the number of
bonds keeps increasing year after year.

I'd never done anything with FORTRAN before, so I spent a couple days reading
about it and looking into how it works, fearful that I'd muck everything up
trying to fix it.

I open the code and find that it was the most elegantly written software you
could wish for, with fantastic comments and structure. I changed two numbers
and was done, buying us another 30 years or so of rock-solid service.

~~~
maxxxxx
A lot of young people think that the people who wrote software 30 years ago
were just plain stupid and all wrote crappy code. In reality there were people
back then who did a good job and there were people who didn’t. They just used
different tools. I can’t wait for the current systems to become 30 years old.
Making changes to micro services architectures written in different languages
over multiple servers will be a lot of fun.

~~~
roymurdock
Necessity is the mother of invention and memory/resource scarcity is the
mother of quality code

A lot of junk and spaghetti code gets written (or copied/pasted together)
today because there are few hardware-based constraints in much of modern app
and system dev

I work with a lot of embedded engineers and have the utmost respect for those
that work on safety-critical, legacy systems

~~~
carrigan
I don't believe that hardware constraints have anything to do with code
quality. Running your code on a microcontroller will require your algorithms
to run in a smaller RAM footprint and potentially come with timing
requirements, but do not dictate that you write quality code to do so. You can
solve those problems with thousands of lines of uncommented assembly
efficiently, but that does not mean it is quality code.

I have worked for a significant time in my career as both an embedded engineer
as well as a backend engineer, and in general I find that the backend code is
way easier to read, maintain, and extend. Embedded code is seldom properly
tested and most of it is written in C where people can abuse a library's
contracts or the preprocessor. It is not uncommon to find functions that are
over a thousand lines in the embedded world. Compare this to Rails where there
are a ton of standards, short and succinct functions, and good support for
testing.

I guess it depends on your definition of "quality code". If you mean code that
is dependable and will do one thing well on one platform for the rest of time,
then embedded code could be considered high quality. I would debate that these
items have more to do with the binary than the code though. If you mean code
that conveys how a program works well to other programmers, is under test, and
follows some standard structure, I would pick modern app development as
typically being much higher quality.

~~~
pjc50
I don't know where this got a downvote from because it's correct - it's quite
hard to build sensible automatic integration tests for embedded code unless
you have the luxury of a full system emulation.

The only thing that resource constraint forces is _less_ code, especially less
dependencies, because you run out of space.

The Toyota "unintended acceleration" court case was a flagship example of bad
embedded code, that we rarely get to see.

~~~
freedomben
Interesting, I thought the "unintended acceleration" was actually just floor
mats creeping up and holding the accelerator pedal. Gonna have to google some
stuff now :-)

~~~
pjc50
[https://news.ycombinator.com/item?id=9643204](https://news.ycombinator.com/item?id=9643204)

It's one of those Rashomon situations where I'm not sure we can ever be
entirely sure but it seems to have stopped.

~~~
jackalo
Thank you for linking that. The comments in it share some terrifying stories.

------
inetknght
> _Assessors are prone to make mistakes when using the vintage software
> because it can’t display all the basic information for a given property on
> one screen._

Well they're in for a surprise if they think modern user experience designers
would fix that.

~~~
Spooky23
I used to support a few assessors offices.

The software is pretty simple, and mistakes were almost always process
problems associated with complex exemptions and classification.

The other set of problems were related to small market sample size for some
corner cases. You can see this at play with Zillow.

Software is used as a boogeyman for leadership and process failure. A clipper
system from 1985 is a great solution. The problem with it is that it’s highly
likely that the needs of the business have changed over 30 years, and the
software has not. That just reflects the lack of capital investment and
process improvement typical in municipal government.

It’s easier to say “OMG, Clipper from 1980!” Because that is better than “we
are a bunch of idiots”.

In the town that I grew up in, the process prior to 1980s computerization was
a 100 year old paper process that was essentially the same as the computerized
process — maybe better as the internal controls were better. Some of the
properties in that town had deeds dating back to the Dutch colonial period,
and in many ways the 1650 process was similar to the 2019 process, except they
used percentage of rents and a levy of crops based on market value.

~~~
nobleach
CA Clipper, FoxPro.... those systems were awesome.

~~~
Roboprog
My first part time job while back in college (1985) was doing dBase II work.
Then dBase III+, Clipper 86, Clipper 87, ...

It sucked not having transactions and referential integrity, but I liked not
having to second guess a query planner :-)

Systems didn’t scale past small workgroup LANs very well, though.

------
chobytes
Im not sure that I think software is always bad just because its old.

I recall working as a technician years ago we used text user interface ticket
system from the early 90s. While it wasnt the prettiest system it did its job
and one could navigate it quickly once they has commited the keystrokes to
muscle memory.

We thought about replacing it with more modern software but after evaluating a
few of the options found none of them to be particularly compelling. If
anything, we appreciated what our “old” system could do even more.

~~~
kalleboo
Where I lived before their old train ticket machines had an array of
buttons[0] where you keyed in the zone code you were going to,
adult/child/senior, hit OK, and stuck your magstripe card in. Once you had
ridden a couple of times and knew your home zone code, you could buy a ticket
in seconds. Of course, for first time users (especially tourists) they were
more confusing.

They then replaced them with Windows touchscreen-based machines, where just
paging through the stations to find your destination by name took as long as
it took to buy your ticket with the old machine. Add to that all your typical
touchscreen issues (calibration drift etc)

I've been to around 30 countries worldwide and never anywhere seen machines as
fast as use as the old ones for repeat users.

[0]
[https://images.hdsydsvenskan.se/980x588/KoHYjzCi_hdzfPAF4xY_...](https://images.hdsydsvenskan.se/980x588/KoHYjzCi_hdzfPAF4xY_j0PR9ls.jpg)

~~~
magduf
This is a good illustration of how differently a UI can be designed for
different groups of users, who have different needs.

Your old train ticket machines were apparently great for locals who were
familiar with them, and had memorized information like the zone code, but they
were surely horrible for anyone else. By contrast, on my last trip to Germany,
the Deutsche Bahn ticket machines weren't the fastest things in the world to
use, but for tourists from other countries, they worked pretty well, since
they had dozens of different languages you could choose to use the machine in,
and had the ability to look up destinations, which is surely a lot easier than
trying to figure out some arcade zone code.

In short, an interface that's highly efficient for an extremely experienced
user probably isn't going to work well for someone who's never used the system
before, and vice-versa. For a train ticket machine, it seems to me you want
something that's easy for newcomers; you can't assume everyone is a regular
user of the system. However, a lot of these comments are talking about things
like warehouse inventory systems; these are things that only a relatively
small number of employees will be using, and they're going to be experts or
working towards that, and efficiency is the goal, not newbie-friendliness.

~~~
CharlesColeman
> In short, an interface that's highly efficient for an extremely experienced
> user probably isn't going to work well for someone who's never used the
> system before, and vice-versa. For a train ticket machine, it seems to me
> you want something that's easy for newcomers; you can't assume everyone is a
> regular user of the system.

It's not an either/or. You need to support both sets of users well.

Sure, if 90% if _interactions_ are going to be by newbie users, design the
system for them. But for something like a train ticket system, maybe 50% of
_users_ may be newbie visitors, but only 10% of _interactions_ are by them. It
doesn't seem like the right call to cater to newbies if it means making the
remaining 90% of non-newbie interactions noticeably worse.

~~~
magduf
Actually, it might. If half the users can't use your highly-efficient system
and then have to go bother your workers to figure out how to buy a ticket,
that's probably worse than making things slightly easier for your regular
visitors.

~~~
CharlesColeman
> Actually, it might. If half the users can't use your highly-efficient system
> and then have to go bother your workers to figure out how to buy a ticket,

That's the kind of user-hostile decision-making that annoys and infuriates
people. It's basically saying it's better to bother a hundred other people
than have one person bother you.

In my comment I said that you need to support _both_ sets of public transit
users well, and that it's not either/or. In many cases it's a mistake to make
the dumbed-down newbie interface the only interface, because that could make
your system noticeably _worse_ for most users.

Personally, in cases like this, I think some kind of touch-screen guide to the
efficient expert interface is probably the right balance. Make it easier for
newbies to scrape by (because that's the best they'll ever be able to do), but
give frequent users a path a better experience rather than holding them back.

> that's probably worse than making things slightly easier for your regular
> visitors.

I think we're talking about _residents_ , not visitors, in this case.

~~~
magduf
No, I agree: ideally your UI will support both classes of users.
Unfortunately, it usually seems we can only have one or the other, so I'd
argue that for train tickets you want to lean towards supporting newbies more.
For warehouse inventory, I'd argue that newbies should be utterly disregarded.

>That's the kind of user-hostile decision-making that annoys and infuriates
people. It's basically saying it's better to bother a hundred other people
than have one person bother you.

I don't understand this comment. If someone wants to buy a train ticket and
the system doesn't even support their language, and they can't understand it,
literally the only option they have is to seek help. I don't have to be able
to read German to use the Deutsche Bahn train-ticket systems; they have
options for English, French, Spanish, and even many other lesser-used
languages like Polish, Czech, etc. If your city has a lot of international
visitors, you either need to have a system that they can use easier (by
supporting their language), or you need to hire a bunch of multi-lingual
agents to work full-time at all the stations, or have some kind of translating
service available for your employees to call to help these people. Guess which
one is cheaper?

~~~
CharlesColeman
> I don't understand this comment. If someone wants to buy a train ticket and
> the system doesn't even support their language

I think we're talking about different things there. I was focusing more on the
language-agnostic interface design (e.g. hand-holding but slow vs fast after a
learning curve), but you seem to be focusing more on internationalization and
language issues.

------
tyre
This is not a software problem. It's a purchasing problem.

We built better software for cities and ran into this over and over and over
again. They're not incentivized to change.

Citizens don't care or know enough to raise hell about city operations. The
employees who do the work can't advocate for themselves and/or feel no reason
to risk their jobs to make their own lives better (there's no reason to be
more efficient.) Leadership cares mostly about press releases—blockchain or AI
or whatever buzzword sounds better to them than the incremental changes that
will actually contribute to fixing cities.

It's frustrating, bleak, and the inverse of inspiring.

I tried for 2.5 years before losing hope. My co-founder carried on another
year before running into the same. The best way to get people what they need
is to bribe them—e.g. work through an intermediary who takes a cut and
delivers the rest to a political campaign—which we refused to do.

Local governments affect most Americans' day-to-day far more than the national
political shitshow and this only becomes more true as populations further
concentrate in cities. And yet, they are utterly soul-sucking morasses of
internal and external politics, misaligned incentives, and bureaucracy. They
drive out the people who can and genuinely want to effect change.

It's a damn shame.

~~~
pergadad
For public sector it's inherently risky to buy software and even devices. They
often lack the capacity to properly outline what they need, to understand
what's possible, to know the real cost/price they'd need to offer to get good
results for tenders, and most importantly lack the ability to write for
technical tenders to get what they need.

The effect of that if something is running well often the most rational
solution is to leave it running. Yes it might not be the most efficient
solution, but it might still be more cost effective than losing 3 months of
work due to conversion to a new system, in addition to all the extra cost,
training, downtime, ...

In the current ecosystem more and more also seems to be license based. Do you
really want to change a decently running system with just occasional
maintenance cost for an untested system that will change all existing
processes and in addition ups your annual cost by quite a bit? Any time you
change a system you probably have to change some additional systems too as
suddenly things don't work together anymore....

Cities are often stretched for resources ranging from (good) personnel to
financial to physical space to ... It's probably in many cases the rational
decision to stick with how it is if the system mostly works.

~~~
pergadad
Can't edit, so just to add that public admin also has to be conservative.
Seems like you were running a startup with a great new product. That means
they have zero idea whether you'll still be around in a year's time! Really
dangerous to commit to a system where you don't know the long term
perspective- especially if you're a public admin which means your switching
costs are high, timeline to make decisions long, and you have a legal or moral
obligation to fulfil your tasks...

To give one other concrete example, i know of a country that chose a great new
software solution to provide information exchange between medical providers.
The tool runs well, they got long term commitment and a safe provider. BUT
they then found that there's literally no one except the provider itself who
can train all the users as the UI, process and exisiting training materials
are all copyrighted (or otherwise protected, maybe through contractual
clauses) by the provider. Means they found s gigantic additional cost even
when they thought they'd done everything right in the choice of software.

Again, risk can be bigger than the reward.

------
Johnny555
I think this is part of the problem:

"We’re dealing with an irrational public who wants greater and greater service
delivery at the same time they want their taxes to be lower,"

There's noting irrational about wanting government to take advantage of the
same technological advancements that businesses use to improve service while
cutting costs.

When I order from Amazon, I see real-time inventory, get immediate order
confirmation, near real-time tracking as the package is handled (and actual
real-time tracking for some orders were I can see where the delivery vehicle
is), and delivery to my door step is 2 days (or sometimes the same day).

In contrast, when I ordered a dog license from my county, I had to _mail_ (FAX
was also accepted) a paper form, and then 6 weeks later I received a metal
plate in the mail (just a numeric ID on it, no personalization). The only way
I knew they even received my order is because my _check_ cleared the bank 2
weeks after I mailed it (no credit cards allowed).

I don't expect my county animal services department to build an order an
fulfillment system that rivals Amazon's, but there are thousands of this
offices across the country, they could all cooperate on one system and save
money dealing wit these paper forms.

~~~
flatfilefan
"There's noting irrational about wanting government to take advantage of the
same technological advancements that businesses use to improve service while
cutting costs." \- the irrational thing is to expect this in absence of
competition. How about having two or more local governments acting at the same
time to enable the true competition?

~~~
Johnny555
It may not be realistic, but I don't think it's irrational for citizens to
expect good customer service from government.

Competition is no guarantee of good customer service -- cellular companies
have some of the worst customer service, but almost everyone has 2 - 4 cell
companies to chose from.

~~~
flatfilefan
Well, those are at best oligopoly. The competition is severely limited through
the licensed access to the radio bands.

------
AdmiralAsshat
I work on several pieces of business software at my job. One was built for
UNIX mainframes, almost entirely in C, with the end-users still capable of
running on dumb terminals (or these days a PuTTY window). The "successor" runs
on Windows servers with J2EE, a GUI, and all the modern niceties.

The most common complaint I hear from people who've switched from the legacy
software to the modern one is that their end-users' productivity is greatly
reduced by the point-and-click interface. The older software's TUI had a very
steep learning curve, but almost like learning Vim, once they developed a
muscle memory for the necessary keystrokes, they could access and update
information much more quickly.

~~~
forinti
I once saw an attempt at modernising a mainframe application that had a single
input line where the user would type a sequence of numbers. The new GUI had
all the fields organised and labeled.

The user then requested and got a single input box where he would type in all
the numbers just the way he did before.

------
lqet
So? I remember a large bookshop in my hometown in which each section of the
store had a terminal with DOS-like interface to their internal database.
Employees were able to use it blazingly fast, and they still used it a few
years ago. Except for security concerns, why should newer software
automatically be better?

~~~
rightbyte
Old crud apps ware probably the prime of usability. These hotel booking
systems with green text terminals and hot keys for everything.

The video store where I grew up had a similar text based interface to some
floppy located database for rentals. Amazingly fast and some override button
to force invalid entries when someone else had messed up or something was
registered wrong.

Binaries run straight from a floppy with no fancy OS to interrupt the very
simple yet manually tedious things it does. Many apps today are way over
cooked for what they do.

------
mc32
It seems they this kind of system would benefit from a more user friendly
interface; however, I don’t think all decades old software necessarily needs
to be uprooted and replaced because it’s old, so long as it’s functional.

BaRT, I wonder if they’ve upgraded their system software to more modern
software? They certainly could add more intelligence (system health, wear,
etc.) and begin automating more aspects, so it could make sense to upgrade
here, but I’m sure there are lots of places where upgrades are not necessary
and the old software is good enough, so long as it’s still supported by a
vendor.

~~~
jdavis703
Which old software are we talking about? Is it the Clipper Card system that
has tens of thousands of business rules that prevent transit agendcies from
rolling out new passes and pricing strategies? Or are we taking about the
software that runs the trains and currently causes a bottle neck through the
transbay tube?

Not everything needs to be rewritten in React using Google’s Material UI
design standards, but my god, our public software really sucks.

Even as the article talked about the tax assessors office... I know someone
who bought a condo, but SF can’t collect property tax because their office is
so backlogged. So instead they told the owner to make sure they save the money
because in a couple of years they’re going to come asking for it.

~~~
derwiki
> SF can’t collect property tax because their office is so backlogged

IME they're able to collect property tax, no problem.

~~~
gph
Clearly the truth lies somewhere in between both these statements. Something
like:

"SF can't collect the full and accurate amount of property tax in a timely
manner from properties that were recently reassessed because their local
assessor office is backlogged due to use of old and inefficient software."

Which is basically what this whole article is saying.

------
peatmoss
Thought exercise: if you wanted to write code to support a business or
nonprofit, and you knew that the software would be used for many years, and
probably never see an update, and wouldn’t have anyone to tend network
services, how would you build it?

Me, I'd probably go with a static executable with a terminal UI and a flat
file sqlite database that could be copied by mortals, and hope that whatever I
set up as a backup regime would survive future migrations to new hardware.

~~~
smacktoward
A fascinating idea -- kind of a software analogue to the 10,000-year clock
([https://en.wikipedia.org/wiki/Clock_of_the_Long_Now](https://en.wikipedia.org/wiki/Clock_of_the_Long_Now)).

------
mdesq
A few years ago, I met guy in my travels who seemed quite carefree and just
enjoying frequent international travel with his spouse, funding various art
initiatives, etc. Turns out he wrote some utility-related software for a large
American city back in the 80s and has nice a recurring revenue stream from
that. Other than some now very occasional support requirements, it's
essentially just a source of passive income. He said switching costs would be
too great, and there's no need to change something that's working.

------
JoeAltmaier
America's industry is running on software from the '80s. You can't imagine the
horrors on the shop floor. Never mind XP or Windows ME - DOS with green-screen
terminals; hand-coded apps booting on home-brew hardware; IBM dinosaurs that
should be extinct but lumber on in the back office.

And there's nobody that can help them fix anything. So upgrade you say? Then
they'd have to start from scratch. And they long ago laid off the accounting
experts, tax lawyers and business planners. They have button-pushers and data
entry folks. Nobody left who knows were the data comes from, where it goes, or
who all uses it.

~~~
reaperducer
_You can 't imagine the horrors on the shop floor_

OK...

 _DOS with green-screen terminals_

Not a horror. If it works, why mess with it?

 _hand-coded apps booting on home-brew hardware_

Also not a horror. Bespoke hardware is everywhere, as are hand-coded apps. AI
does not yet build software.

 _IBM dinosaurs_

Still not a horror. Perhaps you just don't have a lot of experience with
legacy systems.

~~~
linuxftw
> Not a horror. If it works, why mess with it?

Can you still get parts for said machine? Do the disk have bit errors? Is the
RAM faulty?

Things fail, sometimes in non-obvious ways. It's important to be able to fix
your existing system. Often times that means having a plan to upgrade to a new
system.

~~~
reaperducer
_Can you still get parts for said machine?_

Sure, why not?

Worst case scenario for a DOS machine is that you run it in DOSBOX or a
similar VM.

------
redsavagefiero
Good. We don't have to worry about node.js, apache whatsit and Google go-away
v1.0 being our goto technologies in a critical space. That's progress. Stick a
rest/rpc interface on the outputs and ctl and attach a wireless dongle to a
middlebox: voila modernized! If you really want to get fancy attach ELK to the
'stack' and put what data you want in it to represent something or another.
Mission accomplished!

~~~
derwiki
This is sorta what my university did when they launched online registration.
And that was why only 10 students at a time could use the online registration
system-- that was the number of virtual terminals the mainframe had. And it
was a Java applet :weary:

------
oliwarner
Things like this obviously attract headlines, but the real problem comes from
the millions of other machines, decades newer than the ones in TFA, that are
still irredeemably EOL.

I contract for a few companies that supply local governments in the UK. I
regularly have to fill out stringent checklists about IT security provisions.
PCI-DSS and beyond.

So I always chuckle when we meet them on-site and they're running an XP
install with IE8, plastered with toolbars. These are machines I would sooner
throw into the abyss than try to rescue. They'll make me jump through hoops
but there they are inputting and manipulating citizen data on machines that
are almost certainly part of a botnet.

Tablets are also increasingly a problem. As a b2b-service webdeb, I still see
a _lot_ of first gen iPad Minis in active service in enterprise. Last patched
mid-2016. Oh well.

The next big worm attack is going to be devastating to organisations like
this.

~~~
at-fates-hands
> Tablets are also increasingly a problem. As a b2b-service webdeb, I still
> see a lot of first gen iPad Minis in active service in enterprise. Last
> patched mid-2016.

I used to work for a large oil company. Their IT guys were pretty adamant
about security. Right before I left the company, they purchased another
smaller distributor. All their guys had ipad minis and same thing, were last
patched in 2016. The IT guys right away were telling the execs, "Yeah, they're
all getting Win10 laptops, fully patched and locked down to use only our
software. No way we're going to keep using these."

------
deanalevitt
> _Minnesota spent about a decade and $100 million to replace its ancient
> vehicle-licensing and registration software, but the new version arrived
> with so many glitches in 2017 that Governor Tim Walz has asked for an
> additional $16 million to fix it._

This raises so many questions.

~~~
kiba
Raise what questions? It seems par the course for many software projects.

~~~
imtringued
I don't see how cost overruns by more than an order of magnitude are par for
the course.

~~~
growtofill
That’s 16%, not an order of magnitude.

------
wolfspider
This is essentially what I have been doing the past few years among other
tasks. The difference is we do a lot of things in-house and even receive
security audits from companies like IBM and insurance-based development
process audits as well (do you have backups of source code, CI, integration
testing, etc..). Getting off an AS/400 is not easy as it has a completely
different architecture and OS. When it comes to data like this legal
assurances have to be in place and voted on before real change happens. We
have a financial package which replaces this system and some of it written by
our team while the backend and most of the migration handled by an outsourced
company. The challenges we faced were handling lots of data going back decades
to migrate and doing audits for different departments as the migration was
ongoing (parallels). All staff have to be trained up in the new software. All
of our new stuff has to be PCI compliant. Many processes based on paperwork
have to be turned into an e-filing system- this means heavy OCR workloads to
get it into a newer system. A lot of financials end up becoming transparent
and visible to the public as a result. If the government were to adopt better
practices where technology is concerned it will take interest from the
citizens themselves and many additional high level positions focused solely on
technology. In this article when it says it took them 4 years to just acquire
the funds to start on the project I believe it.

------
missblit
> the shift from old-school servers to rented cloud storage has made it
> tougher than ever to fund upgrades

They do know that it's still possible to run old-school servers, right? There
must be some upside or need to switch to cloud computing, either that or
people just get blinded by buzzwords.

Hard to say from the article. Anyone know why governments may switch to cloud
despite the issues in the article?

~~~
gambler
I don't know about governments, by at my last job the transition to cloud was
driven mostly by:

1\. The desire of some people to have AWS stuff on their resumes.

2\. The optimistic belief that the same people who couldn't figure out how to
manage normal servers in 10+ years will figure out how to design good
serverless apps in a few month.

------
aeturnum
_sigh_

The buildings were built in the '80s too (sometimes even earlier!) and it's
fine. Software is no longer new. If there are problems, they don't come from
the age of the constructions.

~~~
sambf
The vast majority of Paris buildings were build between 1851 and 1914 [1]. I
myself live in a 1903 building.

[1]
[https://www.comeetie.fr/galerie/BatiParis/#12/48.8589/2.3491](https://www.comeetie.fr/galerie/BatiParis/#12/48.8589/2.3491)

------
taf2
AM I the only one that thinks this is totally awesome. If the code I write
today is still in use in 40 years that would feel good.

~~~
flatfilefan
Especially when it is so good that it looks like a black box to the word:
correct input -> correct output; and nothing more.

------
gambler
It's a mistake to think that just because something is written in 2019, it
automatically benefits from the extra 40 years of collective experience. The
collective has Korsakoff syndrome and aversion to history.

To be fair, there are two general kinds of legacy systems. Ones that work so
well it makes no economic sense to replace them and ones that are such a
giant, horrible clusterfuck that they warped the entire organization around
their own idiosyncrasies. In the second case, replacing the system requires
replacing most business processes as well, so it's very hard.

------
gdulli
Meanwhile, my modern engineering organization has projects less than three
years old that are already obsolete or orphaned because they were built on
someone's favorite JS framework or CMS that isn't relevant or growing anymore
and no one wants to touch them now.

------
kilon
If the sofware is from the 80s that is very good news, it means it runs on
simpler hardware and the code will be simpler and easier to fix. Which is what
made it viable for us to code in Assembly back in the 80s and early 90s with
the same ease someone codes in c# nowadays. Coding in Assembly now is pure
insanity.

Sure you could still write super ugly code but super ugly simple code is far
better than super ugly complex code. Even if it comes with a forest of gotos.

------
sixdimensional
I have one phrase to add to this discussion - "sustainability"[1]. Systems
have to be engineered differently with that in mind.

I have real concerns that most of what we are developing in high level
languages/systems (with dependency management dependent on external sources)
will not last.

[1]
[https://simple.m.wikipedia.org/wiki/Sustainability](https://simple.m.wikipedia.org/wiki/Sustainability)

------
failrate
Let's be open: software hasn't become fundamentally better even with better
tools and languages. Quality software is more a product of the person or team
that created it than the available bells and whistles.

~~~
frosted-flakes
Maybe not higher quality, but certainly more capabilities.

~~~
failrate
I yield that my potential high point in terms of scale and complexity are
objectively higher, but I feel like we are solving the same problems over and
again with various permutations of bandaids and baling wire.

------
blunte
This is like everything else in life: there is the initial cost, there are
maintenance and support costs, and there are replacement costs.

With roads, it's fairly obvious when maintenance or replacement is required,
so people responsible for financial expenditures cannot ignore them. But with
software, the "invisible" nature of it seems to allow those responsible to
ignore reality and kick the problem down the road until hopefully it is
someone else's problem. (Incidentally, this is how a lot of modern public
companies with with respect to 5-year CEOs and quarterly earnings and actual
responsibility to shareholders, employees, and public).

Anyone who has technical experience knows that it's a tough sell to convince
upper management to spend money on problems that they can't even see. The big
question is, when is the technical debt (or outright catastrophic failure of a
decayed system) worth more loss in revenue than its replacement would cost?

------
rmason
I've publicly advocated a new development model to both the state and cities
here in Michigan.

Share the cost of development and make the software open source. Unique
differences such as laws and tax rates should be in proprietary plugins.

Makes zero sense to me that say fifty states pay a vendor to create software
for a new federal law. The state's pay for one off software but the vendor
writes a single program that's either database driven or uses custom functions
to reflect stuff unique to the state. It's a colossal waste of money.

I can't get the government people to tell me why I am wrong. They just seemed
locked in to doing things the way they've always done them.

Hey, where's that 8 inch floppy again, I've got work to do.

~~~
robaato
Entrenched business models/relationships/incentives no doubt at work...

Rather like ancient guilds and raising the bar of entry to newcomers who might
otherwise undercut them.

------
intrasight
My first job in the late 80s was programming nuclear power plants. It was a
good learning experience. I bet that software is still in place too ;)

~~~
southern_cross
IIRC, the original software pretty much has to run as-is until the day the
nuclear power plant gets retired - typically 40 years, at least.

~~~
flatfilefan
What about the bit rot, decay, half life and all that?

~~~
southern_cross
As I understand it, the software itself never really gets changed, because if
it did then the whole plant might have to undergo testing and regulatory
approval again. But the reactors get shut down periodically (every three
years, maybe) for refueling, so the control hardware undergoes routine
maintenance at that time. Or something like that.

------
bungie4
Nothing is easy anymore.

The stacks are so deep with millions of lines of code. When it works, its
amazing. But it seems not much just 'works' anymore.

Compared to the 80-90's. You were very close to the metal. Comparatively
speaking.

------
AngeloAnolin
I think this stems from the fact that IT in government is still treated as a
cost rather than an integral part of the business. Generally, software
projects will require hundred of hours of vetting, subject matter expert
analysis, political will and pressure before they even get past the approval
stage. And by the time the project is approved, the process which the project
was intended to modernize may have already adopted a different set of methods
thereby setting the project for failure (by not meeting what was needed).

------
tomatotomato37
And? I'd rather civic data being handled on some ancient console that's
relatively bugfree than some modern webapp nonsense that loses data half the
time but has pretty hamburger menus.

------
buboard
Oh well, good for them, have you seen the software they make nowadays?

------
andrevergamito
I work for a billion-dollar-company, in the semiconductor industry and yet we
use a 1986 asset tracking software we access through telnet and despite the 7M
USD effort in three year for the development of newer software we ended up
reverting to the old software. Non to mention we still use Lotus Notes for
email and windows NT on some production machines. The same machines that
validates a good 30percent of electrical components on electric devices and
cars...

------
robaato
Slightly off topic, my first significant job in '80s was programming a Stratus
in their PL/1 implementation. As I recall it was really quite a nice mix of
some COBOL record structures with more modern language options. Shame it
didn't last! The Stratus itself was well engineered but missed their
opportunity in the market subsequently.

Moved on to writing C for an RS/6000 and doing graphics work (few low level
libraries around then). Interesting times...

------
ahallock
Is software from the 80s necessarily bad? I think it's a fallacy to assume age
equates to poor quality, but you see this scare tactic used in media quite a
bit.

~~~
coldacid
It's not bad, but requirements and usage methods have changed, and the
software hasn't. It's fair to update or replace software in such a situation,
so long as the new/updated system meets the actual needs of its users as they
are today.

------
griffinkelly
Reminds me of a great story on 60 minutes about the MTA and how they're using
signaling systems built in the early 1900s: [https://www.cbsnews.com/news/mta-
why-has-the-nyc-subway-gone...](https://www.cbsnews.com/news/mta-why-has-the-
nyc-subway-gone-off-the-rails-60-minutes/)

------
WhompingWindows
I work in the federal government and the performance of the tech is abysmal.
If all of the ridiculously talented developers in SV who are using algorithms
to capture our attention via psychological hooks, instead used their talents
to improve our government tech infrastructure, the world would be a far better
place.

~~~
magduf
The problem there is that the government doesn't want to hire those
ridiculously talented developers in SV to do that improvement. The government
process for procuring software is totally broken and hopelessly inefficient,
and when the government does try to do it on its own, they can't hire decent
developers because they absolutely refuse to pay industry-scale wages. Somehow
they think that the "benefits" of a Federal job are competitive with what
industry is paying, but I guess they haven't been watching the news where
Federal workers have to constantly worry about getting furloughed due to
government shutdowns.

~~~
aaya
Agreed. It seems that these institutions are unwilling or unable to pay for
talent and that they don't have the capacity--in terms of processes and
culture--to leverage that talent even if they were able to acquire it.

Here's a relevant comment thread from a post a few weeks ago about Wells Fargo
going offline:

[https://news.ycombinator.com/item?id=19110672](https://news.ycombinator.com/item?id=19110672)

------
snarfy
> Issue: Control systems for most of the city’s 3,000 traffic signals date to
> the 1960s and make it harder for the department to manage congestion. Cost
> (per intersection): $175,000 to $735,000

That cost seems outrageous. Are traffic signal control boxes really that
expensive?

~~~
gerbilly
Yes, they are.

Just put in roundabouts.[1] No control system costs anymore!

[1] Why are there hardly any roundabouts in north america? Because there's no
money to be made from them.

~~~
reaperducer
_Why are there hardly any roundabouts in north america? Because there 's no
money to be made from them._

I'm not sure why people think this. Maybe people don't get around enough? I
see roundabouts all the time from Seattle to Nevada to Utah, and beyond.

The problem with roundabouts is that they don't scale. In the 60's and 70's,
New Jersey had three massive roundabouts on Route 23 where it now intersects
with I-287. They had to be removed in the 80's because they simply couldn't
handle rush hour traffic.

------
evadne
I would encourage thinking about this through the lens of survivor bias. If
said pieces of software were so bad, they would have been replaced. So, the
business utility delivered by these pieces of software surely exceed the cost
of maintenance and upkeep.

------
dev_dull
> _Minnesota spent about a decade and $100 million to replace its ancient
> vehicle-licensing and registration software, but the new version arrived
> with so many glitches in 2017 that Governor Tim Walz has asked for an
> additional $16 million to fix it._

This is exactly why cities don’t replace these old pieces of software. It’s
not that a intern couldn’t write something better. It’s that the physical act
of “replacing” the old with the new is often very expensive.

This is a story that’s unfortunately all too common in government. Other
people’s money on other people and whatnot.

------
stcredzero
_For local officials throughout the country, the shift from old-school servers
to rented cloud storage has made it tougher than ever to fund upgrades. They
can budget physical equipment as capital expenses, meaning they could issue
bonds to pay for them. But cloud computing is a service, as the people selling
it love to say, which means officials have to pay for it with operating funds_

Could a company structure access to cloud computing over a fixed period of
time as a capital expense? What if there was some sort of on-site access
device involved?

------
gtycomb
You are looking at one of the great lost opportunites for hacker
enterprenuers. Cities/state agencies depend on cio-purchasing managers to help
maintain their IT, and this ends up in third-party handshakes i.e jobs run by
very specialised outsourcing outfits who know how to market here. It takes a
different type of management mindset to turn this around but things are
entrenched firmly for change. The agencies will keep getting the '80s
technology with no end in sight.

------
tomc1985
Dear people, can we please stop freaking out about old software?

------
ArtWomb
Just a heads up. Sat Mar 2, there is a kickoff event for open data week and
civic tech in NYC

[https://beta.nyc/](https://beta.nyc/)

------
fmela
Oh, that's not so bad. American nuclear missiles are running software from the
70s, loaded via 8-inch floppy disks.

[https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-
nucl...](https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-
nuclear/index.html)

------
clavalle
Legacy code is everywhere.

What slows down adoption of new software in these areas isn't generally the
system being replaced itself, but the requirement that the new software
interface with some other 'critical' legacy system(s). It is an ossified web
of business processes and tools and their dependencies.

~~~
nobleach
The biggest problem I ran into (while supporting these old systems based on
DOS and Netware 3.1) was keeping them inline with newer tech. I had a client
(an accountant) with several old DOS based apps. Getting their network of
printers to work reliably was a crapshoot. I'd get calls all the time that
some program just stopped printing. At least Netware stayed reliable.

------
rc_hadoken
Things that last a long time tend to keep lasting a long time. I'd bet the
code is stellar. New things have a higher risk of what I call 'non-survival'.
A concept I've learned from Taleb.

------
boznz
Always look at your dependencies and in that I include the programming
language, the os and the hardware. I have seen good systems fail for
supporting the wrong label printer.

------
tsmarsh
Says the company that runs on the Bloomberg Terminal.

~~~
smhenderson
I've never seen one but I've read that they're actually quite user friendly
and provide a wealth of information. Was what I read over rated? Any examples
of why they're not so great?

~~~
tsmarsh
They are, but they're basically 80s/90s technology.

I work in emacs with lisps (mostly clojure) on x86, usually via a terminal
emulator. Old tech is stable, fast and cheap. I know that, you know that,
Bloomberg make their billions off of it... but they're calling out Cities for
it. Just weird.

------
turc1656
This is the problem right here - _" It took her office almost four years to
secure $36 million for updated assessors’ hardware and software that can,
among other things, give priority to cases in which delays may prove costly."_

Why the hell does it cost $36 million for that? It's typical
bloat/waste/padding as far as I'm concerned. Just another municipality that
pisses taxpayer money away. I guarantee you that a private business could have
made that for a fraction of the price.

------
djohnston
that probably means it's well written code.

------
quotha
if it ain't broke..

------
gambler
Almost every week, I wish the framework I am working right now was written in
Smalltalk 80 rather than Java.

------
microcolonel
If it ain't broke, don't fix it. If it is broke, do fix it.

------
moorhosj
Entire country running on Constitution from 1700s.

~~~
dragonwriter
Actually, the last update was in 1992, which is pretty far from the 1700s.

(Admittedly, the update that was merged in 1992 was submitted in 1789.)

~~~
Aloha
Talk about protracted development ;-)

------
llacb47
I'm glad that my county has modernized.

------
discreditable
This article makes me wonder what nightmares await those who have to maintain
today's software at 30 years old.

~~~
AnIdiotOnTheNet
It'd have to be valuable enough to still be used in 30 years.

------
thisisweirdok
There are multi-billion dollar companies that still need IE11 because they
don't want to build new internal applications. Contrasted to the funding that
some cities get, it's amazing they have computers at all.

There's this huge fervor over disruption and startup dollars, but just like
the rest of the country... municipal software infrastructure is just sitting
around full of holes.

------
StreamBright
Ain't broken, don't fix it.

------
ganonm
Maths is running on 'software' from 500BC. Don't see anyone complaining.

