
The most obsolete infrastructure money could buy – my worst job ever - kryptiskt
https://www.snellman.net/blog/archive/2015-09-01-the-most-obsolete-infrastructure-money-could-buy/
======
russell
As soon as I saw the title, I thought of the furor when Gingrich and Clinton
discovered that the FAA was using vacuum tube computers and radar in the air
traffic control system.
[http://govinfo.library.unt.edu/npr/library/clinton.htm](http://govinfo.library.unt.edu/npr/library/clinton.htm)
The foreign (sole) vacuum tube supplier that Clinton mentioned was Russia. Oh,
the embarrassment!

I knew that there were some delays in rolling out the the replacement system,
so I did some googling to see when it actually occurred. My god, it's now. We
are in the middle of it. The upgrade was responsible for the recent air
traffic meltdown. Please somebody, point out my flawed research skills. I cant
believe this.

~~~
warfangle
A similarly embarrassing state of affairs is the train traffic control system
used by the NYC subway system:
[https://www.youtube.com/watch?v=Mjx3S3UjmnA](https://www.youtube.com/watch?v=Mjx3S3UjmnA)

~~~
jholman
Wow. Their end-goal upgrade, the one that they've installed on one line, and
are installing on everything else looks so antiquated.

My city's subway/elevated train system will be 30 years old this December.
Train control is _fully automated_ ; there are no drivers on trains, and in
normal operation there is no remote (human) operator either. It's been
automated for 30 years.
[https://en.wikipedia.org/wiki/SkyTrain_(Vancouver)](https://en.wikipedia.org/wiki/SkyTrain_\(Vancouver\))

(Side note: during the BART strike, some HN commenters suggested computers to
replace drivers, and other commenters said that wouldn't be safe. Such
hilarious technophobia on HN.)

I don't want to trivialize NYC's challenges and accomplishments: for one thing
their system is far, far more complicated (468 stations on 34 lines, vs 47
stations on 3 lines), and for another we all know it's very hard to upgrade
legacy systems.

But I'm astonished that they would work so hard to modernize their 1930s
technology to 1970s standards in the 2010s.

~~~
bdonlan
The problem with some of these old lines is that they might have level
crossings in which pedestrians or cars might occupy the tracks - so someone
needs to be there to stop the train if that happens. Fully automated systems
like the SkyTrain or Japan's Yurikamome line have grade-separated tracks (and
in some cases platform gates as well) to ensure that sort of incursion doesn't
happen.

~~~
warfangle
The NYC subway system has no level crossings where pedestrians can occupy the
tracks.

The only level crossing where another train might occupy the tracks is Queens-
bound from the JMZ stop at Myrtle Ave & Broadway in Brooklyn. The M crosses at
grade over the J/Z track.

------
mekael
Ah legacy enterprise software/hardware, it keeps me employed and insane.

Where i currently work we have a system which was originally designed in
'69(coding started in 75) running on an ibm iSeries. It's old old tech and
most of the system is cludged together so poorly that instead of being able to
write patches we just write wrappers around the applications(micro services
really) that modify the program as its running. Yeahp.

We have a system written in something called "Workstation Basic" which was
designed/written in '79( a decade before I was born). The original author no
longer works for us and can only be contacted certain times of the year
because he's a snowbird.

Millions of dollars of business is done in access databases and custom made
excel workbooks.

It's a fun time.

~~~
prawn
I know of someone who is in the position of your snowbird, though still
employed. He almost delights in the fact that there's been no effort to
upgrade from something completely reliant on him, even though he knows the
situation is crazy. And he has no personal interest in writing himself out of
a job. I can't approve of that attitude.

~~~
angersock
_I can 't approve of that attitude._

Why not?

Most companies that end up with that kind of hardcore legacy code a) refuse to
upgrade on a reasonable basis and b) refuse to compensate their developers in
proportion to the value they ultimately deliver.

So, fuck'em. Might as well enjoy the pseudo pension.

~~~
nostalgiac
I agree.

If the company has pidgeon holed themselves into the situation, they clearly
have the 2 options you stated, and that should not reflect poorly on the
original author.

------
crystalmeph
Not for technical reasons, but my worst job in software was also my first, a
part time gig in college back in the late 90s.

It was a software company owned by two people that didn't write code or do
anything on a computer other than send email. They had somehow managed to hire
somebody to make a moderately successful version 1.0 of their product, but now
none of the original programmers were there and they were trying to come up
with a brand new release.

They mentioned during the interview that they had been "burned" by every
programmer they'd ever hired. In retrospect, this was a red flag that the
business owners were the only common denominator there, but again, this was my
first ever software job, I had no idea what to expect.

They stopped paying me after 1 month, and kept promising that I'd get paid as
soon as the big contract they were pursuing landed. I quit shortly after I
took a phone call from one of their creditors demanding that they fax over a
check.

~~~
mey
If this was in the US, you should seek back wages through your state.

~~~
crystalmeph
I got another job quickly, and it didn't hurt me too much financially (I was
still living at home with my parents at the time). But you're right, I should
have, if only to force them them out of business and out of the next guy's
misery.

~~~
danielweber
It still may not be too late. Plus other employees who were screwed but
unaware of their rights may automatically get checks.

~~~
vacri
You're suggesting legally chasing a few weeks salary after nearly 20 years?
That's quite a grudge you're encouraging, and likely to cost much more than it
would gain, even if it could be proven.

~~~
RyJones
In Washington State, the state will pursue them with glee and vigor free of
charge (to you). I think 20 years is a little outside how old a debt they
would track, though.

~~~
danielweber
Most states will do the fight. 20 years may well be too late, but anyone in a
similar position should just call and find out.

------
vonmoltke
Not nearly as bad, but this reminds me of some of the ridiculous requirements
at the last program I worked at my previous employer.

One of those was that our code had to compile on a 32-bit SPARC pizza box
running Solaris 2.6. This was in 2009. It was all about some stupid metric:
our code portability was measured by how many different platforms we could
compile and run on. It didn't matter how obsolete or inappropriate those
platforms were; if we could compile and run it counted. Having that number go
down was politically unacceptable.

~~~
nailer
There's something strange about Solaris. Around the same time, an investment
bank (where the Solaris team constantly justified their existence with
'dtrace!' 'zfs!' 'zones!') was paying 50K a year per server multiplied by
hundreds of servers to run Solaris 8 extended support - which didn't have any
of those things, which is good, because the staff didn't know how to use them
other than say they made Solaris great.

~~~
trhway
>where the Solaris team constantly justified their existence with 'dtrace!'
'zfs!' 'zones!'

they did that inside Sun as well. Most noise was about DTrace, the most
useless thing for the Sun/Solaris at that moment.

~~~
mattzito
Dtrace was, and still is, awesome. Linux has come a long way in this area in
the last few years, but DTrace was one of the hottest systems management
capabilities I'd seen in forever, particularly on the very large SMP systems
that had become Solaris's sweet spot in those waning years.

~~~
gaius
Quite. Systemtap is a joke compared to dtrace.

~~~
nailer
There are reasonable criticisms of Systemtap, but a lot of Sun marketing was
just bizarre. They set it up on Ubuntu (which doesn't include it, and which
isn't their competitor: Red Hat is) as an example of what the setup is like.
They kept complaining it was unsafe, when it was only unsafe if you wanted to
use raw C for tapsets. Sun had some great tech, but they were also a sales and
marketing machine that had no problems spouting garbage to make a sale.

------
dan_m2k
Not as vintage as this, but back in '10, I, along with a couple of other
freelancers that I'd worked with in the past got asked to work in a <legacy
clothing manufacturer> in a big, old mill in Yorkshire.

We were immediately isolated from the modern-ish IT department and placed in a
room at the end of the mill that contained a couple of dozen vacuum drive tape
machines. So loud that they silenced the room when they were warming up.

Our task was to "check for security problems" in their ecommerce platform
because there was some serious financial motivation from the men upstairs.

Their 'platform' was a decade-ish old install of osCommerce that had been
hacked to death to make it support multi-side and multi language, together
with some customisations to it's templating engine to provide asset reuse
across sites.

It was immediately apparent that all the SQL injection and most other
vulnerabilities present years ago in that old version of OSc were still there.
They were alongside the many, many vulns that were introduced with the hacks.
Oh, and most of the hacks were done by a mixture of programmers on the
continent, so variable names and comments were easy to understand.

They didn't want to even consider upgrading, just patch it up and get outta
there.

Out of professionalism it was necessary to document the many times management
passed on our recommendations to fix issues we found. Looking through that
file, most of those issues seem to be present even today.

------
notacoward
How ancient this all seems . . . until you realize people are doing the _exact
same thing_ by shipping "golden" VM/container images around instead of doing
proper package and configuration management.

~~~
1971genocide
As a engineer I do not have time to get into this new VM/container thing. I
fail to see how it leads to anything productive ( no offence ).

Call me a simpleton but I am happy with better package management and config
systems.

~~~
acdha
Think of both as ways to get a consistent baseline system which your
configuration will be applied to. It's faster to spin up a VM or container
than it is to do a fresh install on bare metal so you can realistically test
that you really could do a clean install from the documented starting point.

Containers add even faster startup time and the option of isolation so you can
have multiple containers which think they're writing to the same location but
actually aren't. That's not terribly important in a one-app-per-system model
but it's awesome for testing or running multiple instances of an
insufficiently-configurable app on the same system.

------
PaulHoule
The only way to stop this madness is to go on strike against it, as you did.

I work very hard to never mention online that I've had anything to do with
SharePoint or ColdFusion because it will bring recruiters with a job that will
kill you.

~~~
mring33621
Dear %prospect_name%;

I was in our recruiting DB the other day, and came across an old version of
your résumé. I am very impressed with your background and think you might be a
perfect fit for an opening that we have for a Senior ColdFusion Developer.
Please get back to me at your earliest convenience regarding this exciting
opportunity. Thank you.

Sincerely,

%recruiter_sig%

~~~
Kalium
You're going to give me nightmares. I still get emails like this from a six-
year-old version of my resume.

~~~
ErikRogneby
Where is the national do-not-call list for clueless tech recruiters?

~~~
rogerbinns
I maintain one at
[http://www.rogerbinns.com/blocked_recruiters.txt](http://www.rogerbinns.com/blocked_recruiters.txt)

My resume has a "note to recruiters" attached. It is very clear if a recruiter
has made any effort to read any of that. The clueless ones end up in the list
above.
[http://www.rogerbinns.com/recruiters.html](http://www.rogerbinns.com/recruiters.html)

~~~
pronoiac
Oof, I recognize one of those, from an email today. I'd thought I was talking
to an internal recruiter... Incidentally, you have some duplicates.

~~~
rogerbinns
Sometimes they are so egregious I have to add them twice!

I'm guessing it occurred due to doing a batch update, and having multiple
emails from the same spammer.

------
userbinator
Aside from the bureaucracy, that actually sounds like a _fun_ job to me,
solving obscure problems and working with things almost no one has heard of,
instead of churning out and maintaining boring "cookie-cutter" applications.
Then again, I do like retrocomputing, reverse-engineering, and the
demoscene...

 _I imagined myself in a job interview in 2010, trying to explain how useful
my extensive knowledge of Xenix, PL /M build systems, and VMS would be to my
prospective new employer._

You can phrase it as having gained that knowledge through your problem-solving
skills. Working on obscure things, that you can't just Google the answer to
when you get stuck, really helps build those skills.

~~~
stdbrouw
Wouldn't the sheer pointlessness of it all bother you after a while? If
someone wanted to pay you to dig a ditch and fill it up again, for the right
price, would you do it? I mean, I'm not trying to claim that working for that
hot new startup which is going to revolutionize ___ or endlessly tweaking some
megacorp's ERP system is necessarily such a boon to humanity, but still.

~~~
saalweachter
Most things we do are a little pointless in the end. In the long run, the
universe will cool to uniformly distributed inert matter, and all of our
efforts will scarcely affect the distribution.

I currently work at big company X doing cutting edge work on the latest
rewrite of big system Y, used by millions of people each day. In five years,
the next rewrite will begin, and probably within ten years every last line of
code I've written will be deprecated and deleted.

------
Theodores
I supported booking systems that could have bookings stretching far into the
future, this made switching systems tricky as a lot of quite complex bookings
would need recreating with each taking proper time to do. The scope for
automation was there but that wasn't possible.

With the software we supported there was no expertise in the company as the
original developers left a long time ago. Therefore there was no way to
understand why something broke or to write a patch. Instead everything was
fixed by rote, if this went wrong then you did this. If that went wrong then
you did whatever was needed for that. All of these fixes were handed down in
the oral tradition and you took your own notes - there was no manual.

We further out sourced to India so there was no trace of the people
responsible for the code, just us clueless idiots.

Some of these systems were MS DOS based and this was about a decade ago. What
I did manage though was a means of getting on to any computer anywhere in the
world even if it was totally air gapped and owned by someone that did not
speak English. Even better, in most cases this would take seconds from a
standing start. If I needed to access a computer that was on a remote network
but turned off, I could usually Magic Packet it on and be fixing it as soon as
it booted. This was a golden era for my remote access skills, nowadays I have
difficulties getting my own computer online!

There is another way of looking at this, from the perspective of what will be
out there for you to do in 25 years time, I am optimistic that there will be
legacy work for us if we cannot keep up with the new stuff.

~~~
rwallace
> What I did manage though was a means of getting on to any computer anywhere
> in the world even if it was totally air gapped and owned by someone that did
> not speak English. Even better, in most cases this would take seconds from a
> standing start.

I'm curious - how did you do this?

------
knodi123
My first job in software was also my worst. I was learning a lot by building
and enhancing a SaaS applicant tracking system for a small firm in Denton. But
it was run by really shady people; our CEO would hire grossly unqualified
people that he was friends with back in his home country... Our director of QA
had previously been a high school history teacher, and had never used a
computer before... it was nuts.

We were also actively screwing one of our business partners. They sued us, and
my boss considered me a very dangerous witness, so he sent me on a paid
vacation so I couldn't be subpoenaed. Crazy place.

------
bigger_cheese
My work is pretty bad.

We have a box running windows 3.1 hooked up to an instrument on our plant. The
answer I got when I asked about it was it has a proprietary ISA DAQ card in it
with 'flaky' drivers' that prevents us upgrading it.

It's a non critical instrument but is providing 'useful' data I think its some
type of microwave radar measurement instrument - very specialised hardware.
It's been working well since the 90's so not worth spending money until it
fails.

"Proprietary hardware cards" are notorious in industrial control system world
they inevitably lock you into some ancient infrastruture when modern hardware
and O/S's stop supporting ISA slots and things like that.

~~~
cptnbob
One of our clients still has a DOS box connected to a 14.4k modem that we have
to drag EDI interchanges off. You wouldn't believe how much it cost to get a
modem and POTS line installed in our DC so we could talk to it. There's a
windows service written in .net that talks to it via a serial port and POSTs
it to a REST endpoint.

~~~
cnvogel
Of course you could put a small internet connected embedded box to your
clients place... (maybe even a commercial terminal server, if you trust the
security of it) and directly emulate the whole modem+POTS line.

~~~
cptnbob
They paid for our modem setup so meh ;-)

------
nasalgoat
Although nothing compared to what was in this story, you should see the level
of obsolete crap in place at adult website providers. Systems that haven't
changed since the 90's that still generate revenue, so they'll never be shut
off until there's a flood or a 40MB hard drive finally dies.

~~~
kenko
Have to admit, that approach makes a lot of sense to me. It's still chugging
along and bringing in $, so why not just have a policy of benevolent neglect?

~~~
nasalgoat
That Pentium-90 is probably sucking a ton of power at this point, when it
could be virtualized and you could run 50 of those boxes on an $800 Dell pizza
box. I mean, they are literally running the same hardware from the 90s.

~~~
GFK_of_xmaspast
OK you're gonna save $X / year, how many person-hours is it going to cost you
to do the move?

~~~
nasalgoat
I don't know, a couple of 8-hour days?

Just to make a backup alone is worth it. Did I forget to mention none of it is
documented or backed up properly?

~~~
dexterdog
But touching it could break it. Sometimes it's best to let it run its course.

~~~
Killswitch
Until it dies and you start over from scratch.

I am from the adult industry. They are notorious for having that "welp, server
harddrive failed, need to rebuild from scratch" mentality.

~~~
justinlardinois
Never quite expected these words to be typed on my keyboard, but the porn
industry is something I'd like to read a tech article about.

~~~
hga
I've read in passing that in times past they were web technology pioneers.
Seem to remember some were using Freebsd when the canonical approach was Sun
etc. big iron and UNIX(TM), and Linux was perhaps not quite yet mature enough.

~~~
justinlardinois
From what I've read, the adult industry (or, more generally, the human desire
to have porn) has pioneered technology in general. I can't recall the source,
but I recall reading something that said that most, if not all visual media
were used for porn almost immediately after their inception.

I think porn is also something where your architecture only needs to _just
work_ , so that encourages cutting corners and occasionally creative
solutions.

~~~
trhway
>From what I've read, the adult industry (or, more generally, the human desire
to have porn) has pioneered technology in general. I can't recall the source,
but I recall reading something that said that most, if not all visual media
were used for porn almost immediately after their inception.

no kidding.
[https://en.wikipedia.org/wiki/Venus_figurines](https://en.wikipedia.org/wiki/Venus_figurines)

" Various figurines exaggerate the abdomen, hips, breasts, thighs, or vulva.
In contrast, arms and feet are often absent, and the head is usually small and
faceless.

The original cultural meaning and purpose of these artifacts is not known. It
has frequently been suggested that they may have served a ritual or symbolic
function. "

yep :) Imagine an archaeologist in the year 3000 looking at backups of our
days Internet and wondering what "ritual or symbolic function" all those
images and videos played...

------
hinkley
Jesus, man. Company X? Company Y? Product Z?

Variable naming is a very important skill.

~~~
kevin_thibedeau
On legacy systems you may only have six significant characters for
identifiers. Gotta scrounge for every last character.

~~~
me2i81
Six characters! Luxury. I worked on a system where we only had five.

~~~
rwallace
Five characters! Luxury. I learned to program on a system where we only had
two. (Not joking: on the Commodore eight bit machines, only the first two
characters of variable names were significant.)

~~~
cecilpl
Applesoft BASIC (for the Apple ][e) was like this too.

~~~
StillBored
Yes, because both apple and commodore license microsoft basic as the core of
their basic implementations.

------
helmut_hed
I do love war stories, but I think the OP has missed something:

 _After the meeting one of the managers told me that it was really our job to
come up with projects that the customer wanted to buy, not the other way
around. And it usually couldn 't just be a general project for minor
improvements, it'd need clear and ideally measurable goals_

I think the OP has misunderstood the nature of consulting. This is _exactly_
what needs to be done to convince X to spend some of their cash on having Y do
work for them. You need to make a business case, with concrete duration, cost,
goals, and benefits. That this system is a mess is really not in dispute by
anyone... but no one will pay to improve it "just because".

------
SeanLuke
Mine. Pick Basic, with only an ed-like line editor, in all caps, on Pick OS
running inside an emulator called VMark UniVerse on HP/UX. I lasted a year.

~~~
csixty4
Glad you got out of it so quickly. I spent five years in Pick at the turn of
the century and it set my career back quite a bit.

Just out of curiosity, how did you get out and what are you working on these
days?

~~~
SeanLuke
> Just out of curiosity, how did you get out and what are you working on these
> days?

I quit and went to grad school. I'm now a computer science professor.

~~~
csixty4
Awesome.

------
sytringy05
Great story!

I worked at a large aerospace manufacturer who had their most important
production systems (just for a single factory) running on an ancient VAX.

It was written in the 60's and the original author's son, who was the sole
person on the planet who knew anything detailed about it - he was the
architect, the dev, the BA, the sysadmin and so on - was due to retire after
35 years with the company.

The DR plan was similar to this - they had found a model of the same computer
in a museum somewhere that still worked and they had dibs on it.

In the end they shut down the factory, one of the major factors in that
decision was this system. Just too hard to replace it

------
unoti
It's the people that make a job good or bad; the technology plays a
comparatively small role.

I've worked with people I'd do anything for, including spending a couple years
wading through old technology. You do it for the team, because you love your
people. The same is true using new technology: it's easiest when you're doing
it for your team.

~~~
navanit
Or your end users. Everything becomes easier when you really care about the
demographic you're serving.

------
JohnTHaller
I was completely expecting 8 inch floppies or punch cards to show up once I
was half way through reading.

~~~
sthorn
LOL, we debugged our PL/M with an Intel ICE ("blue box") that used 8"
floppies. Good times.

~~~
jsnell
One of these? [http://www.computinghistory.org.uk/det/731/Intel-
MDS-80-Micr...](http://www.computinghistory.org.uk/det/731/Intel-
MDS-80-Microcomputer-Development-System/)

Wow, that looks just amazing.

------
jackgavigan
In a previous life, I was a systems engineer at a company[0] that had several
client with what would be charitably termed "legacy" systems, including SCO
UnixWare and "Olivetti Unix".

0:
[https://news.ycombinator.com/item?id=10048801](https://news.ycombinator.com/item?id=10048801)

~~~
kjs3
Is that Olivetti X/OS or their SVR4 port. X/OS is _really_ legacy.

~~~
jackgavigan
It was the late '90s so it was probably the SVR4 port (I actually think a lot
of those "ports" were little more than rebranding).

~~~
kjs3
Yeah...pretty much everyone with a vaguely ISA/EISA/MCA box in those days just
shipped a 99% unchanged reference port of i386 SVR4. It was interesting that
one of the exceptions was, of all folks, Dell. They had a really nicely done
SVR4 distribution with a hugely improved installer, a lot of open source stuff
prebuilt and lots of bug fixes. Ran it for a while as my primary desktop and
liked it alot. Shame Dell didn't stick with it.

------
amyjess
I went to college at the University of Texas at Dallas.

Our online Student Information System (SIS) was one of the very rare online
services with _operating hours_. It wasn't available at night (and possibly on
weekends too, but I graduated eight years ago, so my memory is fuzzy).

At one point, I started asking around, and I found out that it was because SIS
ran on an old mainframe that couldn't read and write the disk at the same
time. During operating hours, it reads the disk and makes all changes in
memory. When operating hours are over, it writes all its changes to disk.

This was in 2003-2007 and they still had that mainframe controlling all their
class schedules and student data. UTD was considered the biggest tech-oriented
university in the southwest, too.

------
Neil_in_Chicago
Ten or fifteen years ago, I spent a few years maintaining a "system" in FoxPro
for unix on bastard hardware . . . I once had to maintain a system using
"Westi" 1.1, Westighouse's knockoff of CICS . . . The day I completed my first
serious project and linked it, I crashed the company because the operating
system was named "test", so I'd just overwritten it. About the time I was
fired, I figured out that the entire system was bootleg.

IF NO ONE WILL ADMIT EVEN KNOWING WHO BOUGHT THE IDIOTIC PILE OF MISMATCHED
PARTS, WALK OUT _IMMEDIATELY_.

------
api
More like Not Invented Here syndrome to the level of fatal pathology.

~~~
Sanddancer
It's its sibling, Already Invented Here. Every kludge makes sense in the
context of the system state when it was implemented, but that leads to a
system where /everything/ ends up a kludge. Management didn't want to invest
any time in fixing a system that already "worked", so 30 years of temporary
fixes turned it into an absolute monstrosity.

~~~
ethbro
Both of these are important observations. And, as far as I can tell, both of
them exist largely because of corporate politics.

If you could design a corporate culture that minimized the incentives towards
both, I think you'd be a long way towards having a more sustainable company
viz technology.

~~~
webjprgm
There was one time (2006 at a company of less than 50 people) where I
absolutely refused to build the new pricing features in a billing system
(which would have been a big hack on a long series of smaller hacks) unless
they let me refactor it (meaning rebuild it from scratch). After waiting for a
few months without their new pricing features they finally gave in. That's at
a young, nimble company where the co-CEO had a programming background. I can
only imagine how hard a fight that would be at a larger, slower company.

~~~
api
I've worked for a government data center where they ran instances of Ultrix in
a commercial VM to run old data processing software that ingested and barfed
out CSVs that contained UUencoded blobs of image data. All this was written in
a mixture of very _very_ old Perl, shell scripts in some gawd-awful Ultrix
shell CSH script, and C code full of weird -isms that made it difficult to
port to anything made after the rise of mammals. This was of course mission
critical. At least they virtualized it instead of continuing to run physical
Ultrix boxes, which was apparently a _huge_ fight with management.

A side dish of LULZ was that they were experimenting with moving the VMs to
EC2, so there is probably Ultrix in EC2 now.

This stuff is really, really common in large "mature" industries and
government.

------
russnewcomer
I think the most interesting lessons of this story have to do with the
following expectations.

First, the expectation of support timelines. It seems in tech that support is
promised for products far beyond when the vendor should reasonably be expected
to support them. Companies make large investments in technology that
ultimately aren't long-term enough.

Lesson: Any company doing a major project where there is significant cost (as
a percentage of the overall cost) in hardware or software, and that selected
hardware or software is not commodity level or has low market penetration in
their industry should seriously reconsider the projections of TCO, as long-
term support will get increasingly expensive over the life of the project
(Contrived example, supporting Windows XP via virtualization vs OS/2 Warp)

A second expectation is the timeline of viability for technologies. Non-
technical business people of the past generally seemed to have an expectation
of viability that was around 15-20 years. More recently, stated expectations
seem shorter, but are actually not.

Lesson: Businesses that make serious investments in technology need to
consider length of that investment, and technical people should probably take
those estimates and double or triple them, because that's probably how long
hardware/software will be in use. Companies providing technology or support
need to have realistic timeframes of support.

As a corollary to this lesson, I have spent some of my career thus far doing
IT support-type roles for small organizations, and while I haven't encountered
quite the same level of obsolescence as the author, I have seen people using
NT 4 for critical server infrastructure in 2012, I've set up DOSBOX[1] in
Windows 7 for clients to continue to use that version of Lotus 1-2-3 they paid
for and have been using since 1987 , and I've tried to figure out, in 2011,
why Netscape Navigator on, I think System 7, ( using dial-up! ) wouldn't
properly render a webpage. People keep tech a long time, and people in the
tech world need to expect that, and plan accordingly.

The final expectation is one that technology companies have, which is that
tech should be replaced. While there are many use cases where tech gets old,
breaks, has to be replaced, there are lots of cases where it can be functional
for almost indefinite periods of time, and industries that will expect such.

Lesson: Tech companies in should not neglect the future, but should also
strive to design objects of technology that can last a very, very long time.
Companies that do so will find a niche and make money because there will
always[2] be enough customers to support them.

[1]It was going to get some use on an iPad via LogMeIn and there was some
reason I've forgotten why DOSBOX was a better choice than the command prompt

[2] Subject to prevailing economic winds, of course. You can still buy buggies
and buggy whips, but it's not a growth business.

~~~
xj9
I agree with you, but most software engineers seem hell-bent on ignoring the
discipline that every other kind of engineer is _required_ to learn in order
to exercise their craft. I don't mean to say that software engineers are not
engineers, but I _do_ think we should put more emphasis on shipping correct
programs rather than merely functional ones.

~~~
collyw
Management is never interested in that.

~~~
xj9
Sad but true

------
ddoscampaign
Reminds me of a contract gig I passed on to support Microsoft's creaky
inheritance of Danger's Hiptop/Sidekick stuck in cheapest maintenance mode.
Everyone associated were trying to rotate out and it seemed like morale was
less than zero.

Also never work for slave-driving shops like Taos (aka Tause Mountain) unless
you really need the money because it will crush your soul. (eg master cool
techs and be awesomely personable to get cooler gigs/referrals.)

~~~
rietta
That transition was a real mess. We had some games in the T-Mobile Sidekick
store and getting paid the royalties by Microsoft was way more difficult than
it should have been after the transition. It took months to get added as a
vendor and I had to even send them a "final invoice" using the numbers that
they had provided. The next collection step was going to be at least small
claims court. The Danger team did built a great system for its time and by the
time Microsoft took over, it went downhill fast. And then Microsoft's Kin
flopped too.

~~~
hga
_The Danger team did built a great system for its time and by the time
Microsoft took over, it went downhill fast._

Microsoft managed one of the biggest cloud computing screw ups in history to
date:
[https://en.wikipedia.org/wiki/2009_Sidekick_data_loss](https://en.wikipedia.org/wiki/2009_Sidekick_data_loss)

 _The incident caused a public loss of confidence in the concept of cloud
computing, which had been plagued by a series of outages and data losses in
2009. It also was problematic for Microsoft, which at the time was trying to
convince corporate clients to use its cloud computing services, such as Azure
and My Phone._

I've heard good things about Oracle's RAC, but it's understandably intolerant
of your screwing up its disks (SAN mis/re-configuring) when you aren't
properly maintaining backups. I also heard the consultants you have to hire
after you manage such a feat are expensive.

~~~
bro-stick
Yup. I managed some dataguard (not RAC) instances on AWS for Palm pre HP.
Thankfully we had DR plans and snapshots to cover our asses.

Edit: Fixed HA techs.

~~~
mattzito
How did you do RAC on AWS without shared storage?

~~~
bro-stick
Thanks, fixed in edit. I was mistaken, it was dataguard and better snapshots
using archive log mode.

~~~
mattzito
I wasn't totally skeptical - we did RAC on AWS in tests back in the day using
a third node as an iSCSI target, but it was a) sketchy as hell, b) not at all
redundant, c) not something I thought Palm would go for.

~~~
bro-stick
It might work on something like OEL with ZFSonLinux using zfs send/recv.
Larger implementation might want to investigate drbd or something like OCFS2,
GPFS, AFS or Lustre (none of which probably plays well with cloud
environments). Maybe Gluster but with trepidation. (It was an AWS consulting
shop with banking / military chops, whom could sell ice to enterprise
eskimos.)

------
myth_buster
I'm wondering whether the use of

    
    
      ingenious (adj)
      (of a person) clever, original, and inventive.
    

in place of

    
    
      indigenous (adj)
      originating or occurring naturally in a particular place; native.
    

is a Freudian slip or sarcasm.

Edit: Never mind; they used it with _in-house_

~~~
kelvin0
Sarcasm would be my bet ...

------
hadeharian
What people don't always understand is outside of the tech world, there are
people who still think computer upgrades are unnecessary expenditures. It's
true that the theoretical lifetime of a computer is probably something like 50
years, minus the hard drive replacement and the power supply which will
generally go bad much quicker.

The more difficult thing is in a field like aeronautics any minor changes have
the possibility of generating catastrophic events. In these situations, the
cost of replacement, and testing, and phasing stuff in is enormously high and
the risk is obscene when change needs to be made.

------
peeters
> That idea was nixed since the machine was old and fragile, we couldn't risk
> poking around in the inside.

Well that pretty much sums up everything doesn't it. I don't see how a
rational brain could produce that thought.

~~~
egwor
I once worked somewhere where we never switched an old machine off. It was 15
years old and would cost £15K to replace. They were recommended to never
switch it off - on the basis that it was most likely to fail on power up.

------
zem
wow, my worst job infrastructure experience ever was when we were all forced
to run a windows virtual machine to run outlook as our mail client, even
though we were developing on linux and there was no real reason to prefer
outlook over a linux mail client. (our company was making software for the
windows vm market, and the ceo got the bright idea that everyone should run a
windows vm to dogfood our product.)

it was a pretty miserable experience, but this kind of puts it into
perspective :)

------
markbnj
I won't say it was my worst job ever, at all, but I can empathize. I joined a
consulting company a few years back and ended up doing performance
optimization on a legacy point of sale system written twenty years before, in
C, for a large home improvement supplies company. Some of it actually proved
interesting, but in terms of career moves it wasn't the best.

------
bbcbasic
Excellent! It feels like I am on the Daily WTF. A severe WTF followed by a
thread of everyone's battle stories.

I've worked on .NET most of my career. Biggest old-tech WTF is JS in IE5, or
hours of fun getting those DCOM balls spinning again.

------
veddox
I love it :-) What a time travel! I've been reading a bit about computing
history of late, and that article sounds like it's straight from the pages of
some old Jargon File entry...

------
andreapaiola
In 2007 for a big bank I fixed an intranet site for a version of Mozilla
browser (pre-Firefox) that run on an old version of Debian that were used in
their branches.

------
hahamrfunnyguy
The first part sounded actually kind of interesting, being stuck in this role
for eternity would suck though.

------
DonHopkins
Would the author mind sharing that tape dump from an old ITS system please?

~~~
hga
See if you can get Devon, plus the people worried about privacy etc., to get
their act together.

~~~
DonHopkins
I remember running across a Turing machine emulator implemented in TECO in
Minsky's home directory that I'd REALLY love to get ahold of.

~~~
hga
Ask and yea shall receive:

    
    
      MSG: APL    1     
      DISTRIB:  *BBOARD
      EXPIRES: 03/17/81 23:08:54
      MINSKY@MIT-MC 03/11/81 23:08:54 Re: too-short programs
      APL is compact, I suppose.  So is TECO.  When I wrote the following
      Universal Turing Machine, which works, I actually understood it.
    

[ I've interpolated the non-printing characters as displayed by (Gnu) EMACS,
escape is ^], ^^ is one character, as is \356: ]

    
    
      i1Aul qq+^^0:iqm^[29iiq\356y0L1 00L1 11L2 A1L1
      y0L1 0yR2 1AR2 AyR6 yyL3 00L0 1AL3 A1L4 yyL4 0yR5 11L7 A1L4
      yyR5 0yL3 1AR5 A1R5 yyR6 0AL3 1AR6 A1R6 y0R7 0yR6 11R7 A0R2
      ^[j<sR^[;-d-2ciql-^^^[ci"ed^^^[cii^[ciuq'^[>
      j<sL^[;-d-2ciql-^^^[ci"ed^^^[cii-2c^[ciuq'^[>jxblx1lx2lx3lx4lx5lx6lx7hk
      iyyAyyAyy^[32<i0^[>ji110101110000010011011^[ 1uq<htmbqq=>
    
      I do not advise attempting to understand this code, which is
      almost as bad as that for the Universal Turing machine.
    

ADDED: or [http://ancell-ent.com/share/minsky-TECO-turing-
machine.txt](http://ancell-ent.com/share/minsky-TECO-turing-machine.txt)

Please ack receipt of this and/or send me email (in my HN info); for others,
note this is ITS TECO, which I was told was by far the most powerful version
of it (fortunately, by the time I showed up learning it was no longer really
necessary).

~~~
DonHopkins
OOP ACK! It was a shot in the dark, but I am SO GLAD I asked!!! Thank you
Harold!

It looks just like I remember. ;)

