
Ask HN: What's the largest amount of bad code you have ever seen work? - nobody271
I think I&#x27;ve broken my own record with this one ~2500 lines of incoherent JavaScript&#x2F;C#. Works though.
======
dejv
I am maintaining one application in construction industry space. That
application was created 25 years ago by construction worker that never wrote
single line of code before, but because he caused a lot of problems on
construction site they give him Programming 101 book and let him build it.

15 years later the app was close to half milion lines long of huge bowl of
spaghetti code. Only comments in whole codebase were timestamps. I don't know
why he dated his code, but I find it fascinating: he never deleted basically
anything so you can find different timeframes of when he discovered various
concepts. There is use-exception-instead-of-if period, there is time when he
discovered design patterns, there is time before he learnt SQL so all the
database queries was done by iterating over whole tables and such. I am sure I
will find commented Hello world somewhere in the code someday.

I am working on this codebase for 10 years. Code quality improved and major
issues get fixed, but there is not enough budget to actually rewrote whole
system, so after all it is more or less huge spaghetti monster and I get used
to it.

~~~
ThePhysicist
I have to salute this construction worker for building a solution that is
apparently so valuable for the business that they can’t simply replace or
rewrite it. This probably means that it solves a real problem for them, and
adding 30.000 lines of code per year without any formal training or much
tooling is no small feat either. I understand the criticisms and laughs here
from the “real” software developers, but damn it’s just impressive what people
can create on their own given enough time and motivation.

~~~
dejv
It is impressive indeed. The coding started just in time when Windows 95 were
released. There was no Stackoverflow and they don't even really have internet
back then. The programmer (as far as I know) didn't even speak English so he
has access to book or two in German language and code snipets in help section
of Delphi. At the same time creating applications with UI just started, so
there was very little experience available, espcially in rural Austria.

Company did tried to migrate to other software few times, but the software is
just too specific for given industry and legislation of small country that the
companies who tried to create similar software usually went bankrupt soon.

~~~
meshugga
Rural Austria? Please provide a company name ... or at least first and last
character :)

------
wcarss
Years ago as an intern at Microsoft, I had code go into the Excel, PowerPoint,
Word, Outlook, and shared Office code.

Excel is an incomprehensible maze of #defines and macros, PowerPoint is a
Golden Temple of overly-object-oriented insanity, and Word is just so old and
brittle you'd expect it to turn to dust by committing. There are "don't touch
this!"-like messages left near the main loop _since roughly 1990_.

I had a bug in some IME code causing a popup to draw behind a window
occasionally, and a Windows guru had to come in to figure it out by remote-
debugging the windows draw code with no symbols.

I learned there that people can make enormous and powerful castles from, well,
shit.

~~~
PaulHoule
"Don't touch this" around the main loop can mean being able to make promises
about responsiveness, reliability, etc.

Frequently there are critical code sections where it is much easier to tell
people "don't touch it" rather than training people how to work on it safely.

~~~
mannykannot
When that is the case, would it not also be a really good place to explain why
not, or provide a link to the place where such an explanation is provided?

~~~
thepp1983
In reality there often isn't time.

Getting it done > Getting it done properly as far a management is concerned.

~~~
Ace17
Ever had this discussion with a coworker?

Coworker: "I hadn't enough time to do it right"

You: "Given enough time, how would you do it differently?"

Coworker: "............" (crickets)

IMHO it's not related to deadlines only ; the "not enough time" argument is
often a comfortable fallacy, keeping us from facing the limits of our current
skills.

I found it to be especially true with testing. I've lost the count of how many
times I heard "we didn't had time to write (more) tests". But testing is hard.
And when given enough time, these developers don't magically start doing it
"right" overnight.

~~~
carlmr
>I found it to be especially true with testing. I've lost the count of how
many times I heard "we didn't had time to write (more) tests". But testing is
hard. And when given enough time, these developers don't magically start doing
it "right" overnight.

Bingo. It's not necessarily only skills though. It can be myriad reasons and
"no time" is just the easiest excuse they can think of. In big companies I've
often seen the company process prescribing such bad tools that a good TDD
testing strategy is impossible to do with those tools, but they won't move
away from them, because somebody in purchasing already bought 10,000 licenses
for this bad tool (which is often just a bad GUI, which doesn't really help,
except for selling the thing).

The worst tool was a bad GUI where you couldn't even define functions you want
to use, that had slow (>1h), non-deterministic, test execution, for a unit
test.

~~~
thepp1983
To even write unit tests effectively you need to write your code in a certain
differently.

In C# this normally means using IOC + DI.

Also almost nobody I know does proper TDD. I know it is very convincing when
one of the TDD evangelists shows you how to write something like a method to
work out the nth number in a fibonacci sequence using nothing but just writing
tests.

In reality most 95% of developers that even write tests write the logic and
write the test afterwards to check the logic does what it should.

~~~
carlmr
>To even write unit tests effectively you need to write your code in a certain
differently.

>In C# this normally means using IOC + DI.

I've become quite partial to functional programming in the last few years.
Side effect free functions with functions as interfaces for DI lend themselves
perfectly to TDD and data parallel async without worrying too much.

C# is now slowly taking over most of the good features from F#, but I think
the culture won't transform so easily.

------
boyter
The worst program I ever worked on was something I was asked to maintain once.
It consisted of two parts. The first was a web application writen in ASP. The
second portion was essentially Microsoft Reporting Services implemented in
80,000 lines of VB.NET.

The first thing I did was chuck it into VS2010 and run some code metrics on
it. The results were, 10 or so Methods had 2000+ lines of code. The
maintainability index was 0 (number between 0 and 100 where 0 is
unmaintainable). The worst function had a cyclomatic complexity of 2700 (the
worst I have ever seen on a function before was 750 odd). It was full of
nested in-line dynamic SQL all of which referred to tables with 100+ columns,
which had helpful names like sdf_324. There were about 5000 stored procedures
of which most were 90% similar to other ones with a similar naming scheme.
There were no foreign key constraints in the database. Every query including
updates, inserts and deletes used NOLOCK (so no data integrity). It all lived
in a single 80,000 line file, which crashed VS every time you tried to do a
simple edit.

I essentially told my boss I would quit over it as there was no way I could
support it without other aspects of work suffering. Thankfully it was put in
the too hard basket and nobody else had to endure my pain. I ended up peer
reviewing the changes the guy made some time later and a single column update
touched in the order of 500 lines of code.

There was one interesting thing I found with it however, there was so much
repeated/nested if code in methods you could hold down page down and it would
look like the page was moving the other way, similar to how a wheel on TV
looks like its spinning the other way.

~~~
specialist
When I do code necromancy, instead of understanding the code, I try to
understand what it does, how it interacts with the world, and then recreate
that behavior.

Meaning I capture all the I/O and recreate it. SQL, HTML, PDF, CSV, whatever.
Serialize everything, before and after. And then do diffs on the outputs to
see if new code reproduces expected behavior.

Much easier than dead code removal, code deduping, incremental changes,
backfilling tests, etc.

Once someone captures, documents what the code is actually doing, the real
refactoring begins. Removing unnecessary queries. Grooming data. Simplifying
schemas. Aligning the app with the biz. Etc.

~~~
hprotagonist
This is often the only sane thing to do.

It’s my default approach with thorny scientific code: get everyone to agree on
what output should be for a bunch of relevant inputs, get the original systems
output, hash it, and then write a bunch of tests in a new project that all
assert that these inputs produce outputs whose hashes are as follows..

then never look at the guts of the horror again.

------
rollulus
At my first gig I teamed up with a guy responsible for a gigantic monolith
written in Lua. Originally, the project started as a little script running in
Nginx. Over the course of several years, it organically grew to epic
proportions, by consuming and replacing every piece of software that it
interfaced with - including Nginx.

There were two ingredients in the recipe for disaster. The first is that Lua
comes "batteries excluded": the standard library is minimalist and the
community and set of available packages out there is small. That's typically
not an issue, as long as one uses Lua in the intended way: small scripts that
extend existing programs with custom user logic (e.g. Nginx, Vim, World of
Warcraft). The second is that Lua is a dynamic language: it's dynamically
typed, and practically everything can be overridden, monkey patched and
hacked, down to the fundamental iterators that allow you to traverse data
structures.

This was the playground for the guy to create his own reality.

Lacking a serious standard library, he crafted his own. Where a normal world
e.g. file rename function would either do the job or return a error to the
caller, he chose a different approach. Functions were autonomous highly
intelligent pieces of code that tried to resolve every possible problem,
entangled with external logic, so grokking the behaviour of the most
fundamental things was challenging - let alone understanding fragments of code
composed of library calls.

Lacking a OO model in Lua, he built his own. I can spend a lot of time
describing with what was wrong with it, but it suffices to say that each
object had SIX different 'self' or 'this' pointers, each with slightly
different semantics. And highly entangled with external unrelated logic of
course.

I'll save the stories about the scheduler and time series database he built
for another time.

~~~
humanrebar
I've seen personal reality building happen in Lua several times already. It's
very seductive to intelligent solo artists who are given a lot of freedom.

To be fair, I've also seen it happen in C, C++, and JavaScript.

~~~
julianz
"personal reality building" is the greatest quote in this whole thread, it's
perfect. Will be using.

------
hprotagonist
Behold, academia.

The only maintainers of this code, ever, have been grad students and postdocs.
I estimate there have been about 12-15 generations worth. This code has
supported hundreds of publications in its lifespan.

A codebase that began life in 1987, in C. First ported to matlab in 1999.
First source control was added (as SVN) in 2015. Between 2015 and 2018, there
were 6 commits total, yet 3 people graduated out of the lab from it. Probably
100,000 loc total, of which I estimate maybe a third is ever used. 1400-line
matlab functions are normal-ish. I've found loops nested 11 levels deep.

It's a series of psychophysical experiments. Each experiment exists in at
least 4 different versions side by side in source, each named slightly
different, often by incorrect datestamp of last modification. Version control
across machines is not well maintained, so you have to diff everything before
you can copy or move files lest you accidentally blow something away
completely.

Oh, and it's mexed and wrapped for use on a mac on exactly one snow leopard
machine, hardware from 2007.

edit: I think this counts as a job, not a student experience, because I am not
a student. I just have to clean this mess up once in a while.

~~~
mlboss
I think it is a problem in general with code for experiments. You just need to
change a tiny bit for new experiment and you don’t want to ruin earlier
experiment.

~~~
humbledrone
There's a very simple solution: guard your tiny bit of code with e.g. a
command line flag that defaults to "off", and commit that.

It's a little more work sometimes, especially when your experiment changes
things structurally, but it pays off over and over.

~~~
alexbecker
Isn't this what git branches are for?

~~~
nl
No!

There's a great article somewhere about how the normal version control flow
doesn't really work for this style of computing.

You want to keep both "versions" of code live and active in the same place at
the same time (often in the same notebook).

People end up with methods named methodName, methodName2 etc, which isn't very
good. But once you see the workflow you understand why normal version control
doesn't work either.

There should be a solution to this, but AFAIK there isn't yet.

------
madmax108
A customer-facing dashboard. Yes, a dashboard. How bad can a dashboard be bad
you ask? Well, for one, the dashboard had tabs, and each tab was a separate
webapp hosted on a separate server. And each team was responsible for
developing and maintaining the webapp that their team was incharge of (i.e the
User team in charge of Users webapp, Feature1 team incharge of Feature1
webapp).

Now add on the fact that different teams had varying levels of frontend
competence. This led to some webapps being in (badly written) React, some in
Angular, some in JQuery and one in Angular2 as well. Some were java API
backed, some were NodeJS backed and one used Ruby in the backend. Oh yeah, and
each had different datastores as well.

Now alongside this, there's no central auth framework, so each webapp had
their own way of how to determine user auth (there was a shared cookie
thankfully on the *.company.com domain), so there were 6-7 possible login and
logout pages as well.

When the company had a brand redesign and needed all customer facing stuff to
re-align with new design, we had to literally re-design 6 dashboards which
have used different paradigms and different tech stacks.

To this date, the dashboard still exists and is used by customers (Main
dashboard for a company valued at $500M+) and the most common issues are
issues related to Auth (eg. random logouts when switching tabs), data
inconsistency (They have crons which update data from one DB to another, but
it's not immediate) and an inconsistent design and UI behaviour (since JS is
also different for each app) which pisses off many users.

Till date, I'm not sure who signed off on this pointless dashboard design.

~~~
afraca
To some extent this sounds like Spotify. I have been told it's basically
Iframes stitched together, and each frame is owned by a team [0] , I do think
Spotify has some better auth communication though.

0: [https://www.quora.com/How-is-JavaScript-used-within-the-
Spot...](https://www.quora.com/How-is-JavaScript-used-within-the-Spotify-
desktop-application-Is-it-packaged-up-and-run-locally-only-retrieving-the-
assets-as-and-when-needed-What-JavaScript-VM-is-used)

~~~
lowry
Aha! This is the Spotify model!

------
gnulinux
We have absolutely no idea how to write code. I always wonder if it's like
this for other branches of engineering too? I wonder if engineers who designed
my elevator or airplane had "ok it's very surprising that it's working, let's
not touch this" moments. Or chemical engineers synthesize medicines in way
nobody but a rockstar guru understands but everyone changes all the time. I
wonder if my cellphone is made by machines designed in early 1990s because
nobody was able to figure out what that one cog is doing.

Software is a mess. I've seen some freakishly smart people capable of solving
very hard problems writing code that literally changes the world at this very
moment. But the code itself is, well, a castle of shit. Why? Is it because our
tools (programming languages, compilers etc) are still stone age technology?
Is it because software is inherently a harder problem than say machines or
chemical processes for the human brain? Is it because software engineers are
less educated than other engineers? .....?

~~~
pavlov
It's like that in other fields of engineering too, _when they are making
something they haven 't built before_. That's the essential part: for example,
a lot of construction is really just rebuilding the same thing that's already
been built 100,000 times in the past 100 years.

When they attempt to build something new, it often ends up like software –
tremendous overruns in both cost and schedule.

C.f. the only new nuclear power plant being built in the West, which is $4
billion over budget and ten years late:
[https://en.wikipedia.org/wiki/Olkiluoto_Nuclear_Power_Plant#...](https://en.wikipedia.org/wiki/Olkiluoto_Nuclear_Power_Plant#Unit_3)

The reason why this feels more commonplace in software is that we're usually
designing something new. Software has essentially no reproduction costs, so
there's no reason for anybody to design software that's a carbon copy of
something you can already download or buy off-the-shelf. That's not the case
in engineering of physical products or works. New buildings are needed all the
time, even if they're performing exactly the same function as the building in
the adjacent lot.

~~~
augustl
> It's like that in other fields of engineering too, when they are making
> something they haven't built before.

Exactly this. I like to use an analogy of building bridges :)

    
    
      - OK, we have this valley and we want to drive cars over it. What do we 
        do?
      
      - Hmm, we could just make the road along the bottom of the valley
      
      - Wouldn't that cause the cars to fall because of the steep angle?
      
      - Good point. Maybe if we built the road with a lot of twist and turns
      
      - Then it's a slow and long drive.
      
      - Maybe we could build some sort of catapult setup to throw the cars
        across the valley
      
      - That would make safety a concern though. Is there a way we could use
        helicopter rotors to suspend a road in mid air over the valley?
      
      - Or what if we attach the road to each side of the valley and make a 
        road that's strong enough to not crack in the middle under its own weight?
      
      - Yeah, that sounds like a good idea. How do we make sure it can hold its
        own weight, though?
      
      - .....etc

~~~
majewsky
> Maybe we could build some sort of catapult setup to throw the cars across
> the valley

This was played for laughs by German satire news website "Der Postillon" (like
The Onion, but in German): "Department of Traffic to replace ramshackle
bridges by jumping hills"

Nicely photoshopped picture at: [https://www.der-postillon.com/2014/11/lander-
ersetzen-marode...](https://www.der-postillon.com/2014/11/lander-ersetzen-
marode-brucken-durch.html)

------
mb_72
At one stage a company I worked for was considering licensing the code for a
school time-tabling application, rather than paying the company to do the
(fairly minor) changes we required to meet (non-USA) state requirements. The
company was started by a couple of teachers, the same people who wrote their
product. It was 10s of ks of Pascal code, but with not a single variable name
or function name that made any sense; everything was A, AA, AA1, X, XX, XX2
etc. I spent a few days looking at it, then recommended we keep paying the
somewhat steep cost for the modifications. Then at least if anything broke it
was on them to fix it.

Incidentally we had a small falling out with this company, and they were
refusing to update their executable until this issue was resolved. This looked
like affecting some hundreds of schools and their timetables. I did some
checking, and it turned out their 'non-updated' executable was doing a simple
date check on the local PC; if it was past a certain date, the executable
refused to run. So I did a quick hack in our application that involved: \-
setting the local PC date to prior to the 'cutoff date' \- running their
executable with the required parameters, and grabbing the results \- setting
the local PC date back correctly

This led to interesting negotiations as they were puzzled why their 'gun to
our heads' no longer appeared to be working, and things were resolved to the
benefits of both parties soon after.

~~~
meshugga
Code obfuscation + blackmail negotiations ... awesome company to do business
with.

------
LanceH
I took over a Perl project where every SQL call was an exec to a java program
which would make the query.

The largest madness was a J2EE mess where persistence was achieved by taking
your current object, sending a message bean to the server, it would touch the
database and return the result which was being polled for (making it
synchronous). The amazing thing is that the client and the server were the
same J2EE instance. So Class A could have just called Class B. Instead it was
A -> turn it into a message bean -> send it to the "server" (same machine) ->
unwrap it into A again -> transform it into B -> message bean it back to
client -> unwrap into B.

Literally three months of 8 people ripping all of that out and replacing it
with A.transform() // returns B

Oh, and at the same time, none of this was in source control. It was being
worked on by 12 people up until that point. They didn't know who had the
current files which were in production. So my first job was taking everyone's
files and seeing which ones would compile into the classes being used in
production, then checking those into source control. Up until then, they just
kept mental track of who is working on which files.

~~~
C1sc0cat
Was there a reason they did not use DBI to call the code ?

~~~
LanceH
I had been told he bragged about it being "job security" since nobody could
understand what he had written. I never spoke to him directly as I was his
replacement...

The java mess was a pattern copied from within the company where it was used
correctly. The main product of the company was built around some very complex
scheduling software. It was a black box and communication was entirely through
inter-server beans. This was copied to intra-server, which makes no sense at
all, and it was used everywhere.

~~~
mattkerle
"job security" coupled with "I was his replacement" says that his plan worked
about as well as his code...

~~~
LanceH
Indeed. It was quite the entry to the company for me as well. "What are you
working on?" "Fixing Dave's code." "Good luck."

But my most auspicious entry to a company was where the guy I was replacing
had been arrested for stealing data from a competitor. My first two days were
spent recovering logs, downloads and other traces of what he had done and
handing it over to the lawyers. (the circumstances were not mentioned in the
hiring process)

------
mattzito
My first job was sysadmin of a third tier ISP back in the dialup days. The
account management and provisioning system that ran EVERYTHING was probably
close to 100k lines of csh. Everything was done via a UI that the shell
generated as a sort of curses-style interface.

What was horrible about it was that it controlled everything from who got a
website, active domains, what POPs users could dial into, metered billing, you
name it. And it did all of this by manipulating flat files of pipe-delimited
data on a central server, then rcp’ing those files to the various machines,
then rsh’ing to the various machines and kicking off THEIR scripts, which
parsed the source files and generated their own files, which called another
set of scripts that parsed THOSE files and generated the software config
files.

This included doing things like updating init scripts so that new IPs got
added to interfaces, and what email server a user was provisioned on, so it
had to generate new exim configr with routing rules.

All this to say that it all worked, but I dreaded having to go in to
manipulate anything. Adding a server at least had a dedicated procedure so
that was fine, but anything else was a nightmare.

Case in point - as part of a gradual plan to remove this nightmare, I swapped
out the radius server that they were using for one that could support a
database backend, and modified the local config generator script to make a new
config for the new software as a stopgap until I could get it into a database.

The config file had a series of fields that just had numbers in them, and
after much digging, it seemed like that controlled whether a terminal dial in
user was presented with a menu of options, and what options. I had to
reimplement that logic for the new software, made a mistake, and accidentally
removed the option for UUCP for the 10 customers that were still using UUCP.
One of them was on an ISDN line and their mailer decided to continuously
redial looking for the UUCP, tacking up thousands of dollars in carrier rate
charges for the weekend that it took anyone to notice something was broken.

~~~
abledon
Wow this is legendary. I’d love to direct a short film or tv series that
revolves around an IT/software team using a massive csh codebase like this.
I’d love to generate some training montage / diagram sequence shots of the
system being built by the characters maybe make some cool blender / adobe
premiere overlay screen splits of the high level architecture as the team
references certain aspects of the system

~~~
deevolution
Make it like the IT crowd, but for a software team. That would be golden.

~~~
MrDOS
I'd take something less fictionally dramatic and more along the lines of
reality TV (ala home makeovers/Kitchen Nightmares/Bar Rescue): a team of crack
engineers untangling the mess and laying out the best practices for future
development. The concept is even ripe for booze sponsorship.

------
Rebelgecko
The project around 10 million lines of code. Some of it was written in a very
specific version of Fortran that was a PITA to compile. One fun experience was
opening a file and seeing it was created on my birthday. Not in the sense just
matching the day and month of my birtdhay. The code was literally written on
the day I was born.

~~~
jedimastert
I recall rewriting a several thousand line program that was written in a
preprocessing language that compiled to fortran that was homerolled by some
random guy that was just so ahead of the game at the time that he basically
wrote an extension of fortran with all the macros he made and rolled into one
thing.

The rewrite was 100 lines long.

------
pengo
Ten years ago I was called in to remediate a new web application which had
been subcontracted to an Indian development company. The PHP developers who'd
put it together evidently didn't know about classes, and each page in the
application was hundreds, sometimes thousands, of lines of spaghetti code,
most containing the same duplicated (but subtly changed) blocks providing
database connectivity etc. Security had not been a concern either; passwords
and credit card details were stored unencrypted in database text fields.

I was called in because, while most of the application worked, some of the
requested features were not yet complete. When I made my initial
recommendation (scrap the whole thing and start again) I was told the client's
board would not agree to that because of the money already invested and the
fact that the board had seen a demonstration proving that "most of it worked".

It took two developers eighteen months to beat this sorry mess into a
maintainable state while ensuring it remained "usable". It would have taken
one third of that time to rewrite it from scratch.

~~~
aussieguy1234
10 years ago PHP didn't have classes

~~~
cpburns2009
PHP 5 was released 14 years ago in July, 2004. One of its new features was
OOP. Classes were supported at that time. In fact, a comment from 14 years ago
shows an example class [1].

[1]:
[http://php.net/manual/en/language.oop5.php#46290](http://php.net/manual/en/language.oop5.php#46290)

------
dustinmoorenet
I use to work on hospital lab software.

* It was over 20 years old by the time I started

* It was written in Fortran

* Variable names were single and double digits

* Each fortran program would run in isolation but had a shared memory process

* It was formally a terminal program but a weird Java frontend was created so everything looked like Windows GUI

* All program names were four letter acronyms

* All data was stored in fixed width binary "flat" files

* It previously was under CVS version control, but each install slowly drifted apart, so each site had it own unique features and bugs.

* I once had to move a significant feature from one install to another using only patch files generated from the work done on the original install.

------
ohthehugemanate
The worst for me was a rescue project: a site for a US tech sector Public-
private partnership. Nothing too complicated: recurring donations, paid
events, a small members area. They had sent it to an Indian firm to build in
Drupal 7 - not a lightweight system to begin with.

I would like to say "cue the stereotypes for Indian developers" and we could
all have a good laugh. But no. This is more like Heart of Darkness. They must
have traveled to the darkest corners of the subcontinent to find a mind
capable of the eldritch horrors we found there. We started keeping a wiki of
design patterns, to save us WTF time. Here are a few.

* Memory management. What's that? This site required 16 GIGABYTES as a PHP memory limit in order to load the front page for an anonymous user.

* Security. What's that? Part of the reason it required so much memory, is that it would include a complete debug log of the site's entire transaction history, including PII and credit card numbers, in every session. Meaning any vulnerability was a critical vulnerability.

* They would arbitrarily include the entirety of Zend framework to access a single simple function. This happened several places in the codebase, with different versions of Zend committed.

* Can't reach the ERP to get the price for an event? Let's set it to $999,999.00 and proceed with the transaction.

* Invoice numbers were random numbers between 1-1000. Clashing numbers meant a fatal exception that would fail to store the invoice or payment record... But not until after payment had been processed. Birthday paradox means this happened a lot.

* The developers used arcane bits of Drupal API to do totally mundane things. Like, if you know about hook_page_alter, you know there's a setting in the UI for the frontpage. But we'll just use hook_page_alter instead.

* Write queries using Drupal Views, rewrite the query in code, override the views query with their custom (identical) version using an unusual API hook, just to add a sort.

I could go on, but I think you get the picture. Eldritch horror of a codebase.

~~~
Avalaxy
Holy crap. You already earned yourself an upvote for this sentence:

> * Memory management. What's that? This site required 16 GIGABYTES as a PHP
> memory limit in order to load the front page for an anonymous user.

~~~
andyhasit
I'm tempted to log out and created 5 new accounts to upvote for each bullet
point :-D

------
lucozade
I used to own a C++ application that was a morass of abstractions and
indirections so it was impossible to reason about. It took a number of hours
to compile.

On one infamous occasion we were making a, relatively small, patch release.
The debug version worked fine but the release version crashed systematically.
Even when we backed out _all_ the changes we had the same behaviour. We were
screwed.

Until one of the team had a bright idea. She stripped strings from the debug
build and tested it. To our surprise it not only worked, it was only slightly
bigger than the previous release version and it was also slightly faster! We
shipped.

This experience was the trigger to make me go all-in on a full re-write that I
had been contemplating. One of only a couple of times in my career that I've
made that decision on a major piece of software.

The re-write was a huge success. It was also about 10% of the original in
terms of LoC. The day our testing finished, we held a ceremony where we
deleted all the old code from the current version.

This caused a slightly different issue. At the time, code metrics were
starting to get fashionable but LoC wasn't yet the pariah it became.

So, a couple of days later I got a concerned call from the metrics guys.
Apparently, we had deleted more code than all the other teams combined had
added in the previous measurement period. This caused their metric calculation
to barf. Their solution? We should add all the code back in! This led to a
somewhat heated argument that ended up with me persuading them that deleting
code was good and they should, at least, abs(LoC) it. It didn't make the
metrics any more useful but meant that we had an application we could reason
about. Happy days.

~~~
shoo
> Apparently, we had deleted more code than all the other teams combined had
> added in the previous measurement period. This caused their metric
> calculation to barf. Their solution? We should add all the code back in!

ah yes, when bad metrics become targets

------
zhengyi13
From 10+ years ago when I worked at PayPay, webscr (the frontend at the time,
where you'd log in) was a total of 2GB of C++ that would get compiled into a
1GB CGI executable, deployed over some ~700+ web servers.

Debug versions would never get compiled, as I'm told the resulting file was
too large for the filesystem to handle.

Apparently a great deal of the code was actually inline XML.

They _knew_ this was a bad pile of technical debt, too: at one point, a
senior/staff engineer gave a company presentation where they brought a fanfold
printout of the main header file for this monstrosity, and literally unrolled
it across the entire stage.

~~~
cgijoe
I think you meant to say PayPal?

~~~
zhengyi13
Oh whoops, thank you, yes!

------
thepp1983
I've worked on a CMS that was partially done in .NET, Iron Python and used an
XSLT templating system to generate HTML for the front end.

The architecture looked like something from the Early Java days.

The system used Iron Python / C# in the following way.

1\. A web request would hit the CMS 2\. There was a massive switch statement
to work out how the query would be rewritten 3\. If it the url was prefixed
with processor it would attempt to find the processor in the db. 4\. The code
would then find the python script associated with the processor. 5\. The
processor would then spin up a command line instance in a hidden command
window on windows server. 6\. The processor would have to return XML that had
to be built up using strings (not element tree for you). 7\. This would return
the XML to the C# which would then try to render into the XSL Transform.

If at any time this failed. Silent failure. There was no way to debug easily
(There was a magic set of flags that had to be set in Visual Studio or wise
you couldn't debug the python scripts).

To get the software to build on a new machine. It took a contract 4 months to
reverse engineer an XP machine. None of it was documented anywhere.

It used ImageMagick to generate thumbnails on the fly which doesn't work to
well with windows server.

The lead engineer was an alcoholic. He used to go to the local pub for 4 hours
in the middle of the day and come back smelling like a brewery.

~~~
goostavos
I actually laughed while reading this. I've encountered something eerily
similar, but it was a Ruby app. Almost exactly your steps 1-3, however, what
was returned from the db were ruby method names, which were then invoking
other pieces of the code through some deep, twisted ruby magic.

Uncovering what the hell was actually happening was like peering into the mind
of a psychopath.

Once myself and 3 other engineers spent 4.5 hours trying to figure out how to
send an email from this treacherous app (a feature which had stopped working
months before we showed up). After those 4.5 hours, none of us even came
close.

To top the whole thing off, page load times were in the minutes. The standard
joke from the users was that they "get to take lots of coffee breaks."

Much to the (new) manager's credit (and my sanity), we got buy in to just
build a new version and let the old one die. I left the team shortly after the
2.0 beta launch, but God I wish I would have stuck around a little longer so I
could have seen the official end of life for that calamity of an app.

~~~
thepp1983
In a weird way I kinda like these totally mental systems. It really ups your
skill level for debugging if nothing else.

I've worked with lots of proprietary CMS systems and I kinda got into a groove
with working with them. I knew exactly how to manipulate the system within the
parameters of said system.

I kinda found it challenging. When a system is well built. I find it boring.

------
TheRealDunkirk
I wanted to tackle a rewrite of a legacy LISP program, used by AutoCAD, to
take in a list of 3D coordinate points, and output a DWG of a 2D projection
(from which to create a jig on which to assemble a prototype). Cool stuff,
right? My intent was to create a "proper" script in AutoCAD to actually create
3D objects from the input data, and let the program do the projection to a 2D
drawing. (I'm still not sure if that would have been possible; I never got
that far.)

I had never written in LISP before, so I bought a book!

I was having trouble reading the program on the computer, so I printed out the
roughly 15,000 lines. Not a lot, I know, but it was about an inch and a half
thick stack of paper. I started going through it.

It consisted of LOTS of subroutines. Thousands. Each one neatly formed; no
more than a couple hundred lines. It gave me hope.

It read in the text file, created a blob of a string, and then passed that
blob to the first subroutine. Then it passed to the next subroutine. And then
the next, and so on. As far as I could tell, it never called a subroutine a
second time, and it never returned to the starting method. Given the
strangeness of LISP, I couldn't figure out what it was doing, or why.

The guy who wrote it had retired, and we didn't get along anyway, so I didn't
try to chase down what his thinking was.

I gave up.

To my knowledge, they're still using that program 17 years later.

------
le-mark
Bad is not a good measure. But taking 'bad' to mean large, convoluted, zero
documentation; I was once at a financial services company that had a 25
million+ LOC mainframe cobol application that had been under active
development since 1969. This was a batch and CICs system. It was spaghetti on
every level; database (db2, vsam, isam), the screens of the app, the batch
jobs, the cobol. It was truly astounding. It was also the source of about $500
million in revenue. They were doing software as a service in the 1970's. It's
still going today. Customers in that space don't have many options.

~~~
pmarreck
Sounds like a market opportunity. What vertical market is this, more
specifically?

~~~
vaylian
Market opportunity? Sounds more like the best way to throw away your sanity.
Some things are just beyond repair.

~~~
pmarreck
Since when is a greenfield app on a new product addressing a market with only
old entrenched players “throwing away your sanity?”

~~~
vaylian
Sorry, I did not realize you meant to create a new software from scratch. I
thought you were talking about supporting/extending/refactoring this old pile
of madness. Thanks for the clarification!

I suppose you were referring to the statement "Customers in that space don't
have many options". Your statement is reasonable and there might be market
opportunities to create new software. But I somehow have the feeling that it
would take a tremendous amount of time to recreate something that checks the
same boxes as the old system. But I do not work in that sector so my judgement
might be totally off.

~~~
pmarreck
Thing is, a competing product does not (at launch) have to match the
entrenched one feature-for-feature. It just has to tick the MOST IMPORTANT
boxes while being advantageous in other ways (such as being built on FAR newer
tech, more reliable etc.)

~~~
vmchale
Being built on newer tech isn't an advantage from a customer POV. It's only an
advantage when the newer tech is better, and tbh I have a hard time believing
that you'd match the performance and security of a mainframe that easily.

~~~
le-mark
Being an order of magnitude cheaper would be compelling to these same
companies though.

------
Softly
In my first year of university, I wrote a Java swing game which had classes
for literally every part of the GUI (I was young, carefree and unwilling to
yield on the principals of OO I'd been taught earlier in the year. I think it
had somewhere in the region of 100 classes. Of which only about 3 had the real
logic in them.

Now I've put it in perspective, I went on to an intern position between years
2 and 3 of uni. I was handed a _lovely_ piece of code which had:

\- Around 300 classes \- 3 or 4 layers of nested generics \- Factories,
factory factories, generator factory factories \- 90% of the parameters were
passed in from the build engine running the code, so it was impossible to run
locally, ever. \- 0 tests \- Some 100 pages of documentation, which had been
lost until I was about halfway through my placement (and mostly documented how
to set up and run it, not how to maintain it)

Seriously, this thing was designed to the extreme, made to be generic to every
single scenario in existence.

So what did it do? It took items from a customer facing system, transferred it
onto the internal work tracking system. Then when they were updated in the
modern system, mirrored the relevant updates back to the legacy.

The best part? Every time the internal work tracking system updated (once
every 6 months), this thing broke horribly and it was practically impossible
to fix. Even if you managed to set up stuff so you could work in a development
environment, it still connected to the customer facing system, so you had to
be incredibly careful what you did during testing.

It wasn't the biggest in terms of LOC, but it astounded me just how much
effort (apparently the guy who wrote it squirreled away for a year to write
it, then moved to Canada, and was famous in the department for having one too
many beers during a outing to a local Indian) went into designing this
behemoth.

I still have the occasional nightmare about it!

------
Rjevski
I once had to look at a client’s code to determine if/how we’d go about taking
over their application. Their only developer threatened to quit and this is
when they realised it would be best to outsource this and reduce the bus
factor.

It was a huge _folder_ (not repo - and there were zip files of different
“versions” of the code in there). The main monster was a huge Visual Studio
solution with hundreds of targets, one would be an application for entering
some data, the other was for entering data from a hardware device (a scale if
I remember right), etc.

The main source of truth was an MSSQL database to which all these apps would
connect _as root_. There is no backend as such to ensure access control &
consistency, and any misbehaving app could essentially trash the entire DB.

Database credentials were hardcoded in every app’s main entrypoint, with
earlier “versions” of the credentials commented out below.

I thought that surely these must be either staging DBs or at the very least
there would be network-level access control meaning the DB wasn’t accessible
from outside... but no - I managed to connect to their production DB _as root_
from a random, untrusted location. I do not know if MSSQL uses encryption by
default but I would bet good money there was none and they were essentially
connecting to their DB as root, over plaintext, from hundreds of different
locations across the country without any kind of VPN.

In terms of code you obviously have your standard & expected “spaghetti
monster” with UI & business logic scattered everywhere. What struck me the
most was an _empty_ exception handler around the main entrypoint.

In the same folder there was also source for an iOS app. Didn’t look at it but
I don’t see _any_ valid reason why this should be in the same place as the
Windows apps.

Thankfully I no longer work there and even if I were I had no major C#
experience (which gives me a very convenient excuse not to touch this mess).

~~~
HeyLaughingBoy
Ha!

Are you me? :-)

Had almost the same experience, minus the database. Friend of the owner wanted
to buy a company and asked us to evaluate their code to see if it was
maintainable enough to add new features. I got a zip of hundreds of firmware
projects each representing a different version. They were all on the same
basic platform but with different hardware features #ifdef'd, or customized
for a particular customer. The code itself wasn't _that_ bad (not that good
either!), but their developer clearly had no idea what Version Control meant.

In the end I gave the thumbs up and he bought the company, then ended up
having to redesign the product from scratch since much of the originally
designed-in components were no longer available. He did his Due Diligence for
the software, but ignored the hardware side!

------
8fingerlouie
I worked for a company that makes sortation devices (conveyors etc), and i
inherited an old product from a previous colleague.

The software was largely a standard base with modifications done for the
individual customer to suit their business needs.

In this particular project, it was a batch sortation, meaning we received a
large batch file from their mainframe, which would then be parsed and executed
by the controller software.

Everybody feared this particular project, and estimates on new functionality
were sky high, but I didn't think much of it until i had to modify the batch
parsing code. I was met with 22000 lines of C code in a single function. This
single function was modified multiple times each year, usually adding more
code in the process.

It took me the better part of a month to refactor into "manageable" chunk
sizes, and in the end i was left with a 1700 LOC function that was still "too
big", but nobody really understood how it worked, and we couldn't test it, so
i just left it at that.

After my refactor could implement new functionality somewhat faster, but in
the end it was still a very complex algorithm, so despite being in smaller
functions, you still had to be very careful when modifying it.

------
badjobthrowaway
(I am a regular of HN. I don't want this attributed to my acct.)

Previous job. Was sysad. This code runs most of the academic US internet.

Everything was Perl5. There were over 150 different major applications written
with it, ranging from 40 lines to 500k lines. The older the recent commit, the
worse. Touching any of this would cause errors, either in itself OR in
associated applications! You'd be working on thing A, get it working well, and
4 weeks later thing B would fail horrendously.

The worst was a tie between a variable that was a 6 line long SQL query, which
was packed to the brim with function calls that ended up expanding a query to
something like 50 lines long.

The tie was a gem in code that wasn't touched for 7 years. This wasn't at the
top, or even in a config file. It was hardcoded in the middle of the perl5
program...

    
    
         $db_server = (servername);
         $db_user = root;
         $db_password = (password);
    

Other dishonorable mentions are as follows:

1\. No primary keys for the main database....

2\. Goober had the idea of storing pictures in said MySQL database. 70 GB of
pics...

3\. Redhat 4 still in production, along with RH 5.

4\. Its everyone for themselves. The goal is you hobble it along enough for
the next oncall. Let them get hit with it.

5\. Running iron from 10 years ago. Contracts pull in $$$, but you're dealing
with paleo-datacenter crap

6\. Just retired LTO3 tapes. Now they have "shiny" LTO5....

~~~
zimpenfish
You know what's terrible? This could almost be any one of about three of the
Perl contracts I've had in the last decade (apart from the US part, obvs.)

~~~
badjobthrowaway
Yeah, the more I dig around how perl shops work, this seems endemic.

Now, I work in a heterogenous Windows/Linux shop with non-paleo hardware.
We're even deploying on AWS in a limited fashion.

This place has its warts too, but everywhere has _something_. But that
previous place... I'm surprised it still hasn't crashed and burned. Their
networking was solid tho. Just anything with Linux was a tire-fire.

And as one more gem, there was a program that terrified the fuck out of me. A
fellow engineer showed me this update tool that would update a router
remotely. Great. Well, if you add the -n (customer) flag with the customer
number, it would update __all __the routers for that customer!

It was spectacular, and terrifying at the same time. I asked them for their
testing procedure, and it was a -t (customer-testing). If they forgot the
-testing, well.....

------
binalpatel
At a former employer someone created a framework to try and create a generic
process to process and load data (i.e. ETL). It was written with the best of
intentions, but was terrible.

This framework was given to an offshore team, and they were told to use it to
farm out hundreds of requests. The framework was inflexible enough that they
each started adding snippets here and there to make it work for those
projects, with no code review.

When I joined there were well over a hundred different projects, all using
this framework at the core, most having little bespoke tweaks to make the code
work. Every failure was essentially a new failure in a new way.

It was a useful experience - it's one of those experiences that teaches me via
a negative example. This was the worst example of how "roll your own" resulted
in incredible technical debt I've seen.

~~~
CoolGuySteve
Had something similar happen to me. I gave a new guy a task of 'figure out
luigi or airflow to help us process nightly data better than crontab'.

He came back with a custom python/c++ framework where literally every function
was called handle(), and in c++ it used type inference to figure out which
handle() to call.

Apparently this is what he wrote for his last firm.

We quickly found something else for him to do but he didn't last much longer
after that anyways.

------
andai
My own! For a university assignment we had to make a simple Excel clone as a
group.

We divided the work and I ended up working on the formula parser. I spent a
week thinking about it and couldn't figure it out (I wanted to work it out
from scratch). Eventually I had a flash of insight: I know how to parse simple
formulas, so I can use string replacement to recursively rewrite a formula
until I can parse it.

By the time I had written all of it, I didn't understand how it worked
anymore, but it did work!

FormulaParser ended up being longer than the rest of the codebase combined,
and I eventually learned the other groups did it with a regex and ~50 LoC...

------
jacquesm
Most large enterprise systems that are more than a few years old tend to be
pretty bad, and almost all of them work just fine, as long as you respect the
'dragons be here' signs and you don't attempt to fix that which isn't broken.

Millions of lines of code (or even 10's of millions of lines) are really not
all that exceptional. The original programmers are typically in some home or
are pushing up the daisies.

------
letientai299
I came here was about to complain about some projects of my previous
companies. But reading this thread makes me feel ashamed for getting angry
with them, compare to how much more pain you guys went through.

Thanks for all the sharing. Young devs (like me) should really read and
appreciate this thread.

------
logfromblammo
My previous job. More than a million lines of code for a glorified CRUD app,
with more than a $7 million annual budget.

Accruing technical debt was a process feature. More bad code that everyone is
afraid to touch means more budget for terrified developers and testers, and
insane networked database design means more budget for servers and sysops. The
fear leads to meetings, the meetings lead to suffering, and suffering leads to
the dark side. It still works according to spec, and is human-fixed quickly
whenever it doesn't, but the poor quality of the codebase is likely costing at
least $2 million per year.

~~~
abledon
Read that last paragraph in the emperor’s voice from star wars episode 6.

------
overcast
I'm guessing by bad, you mean ugly. I'm fairly convinced there is no "good"
code, or that it's such a minuscule subset that you'll never likely encounter
it in your career. Every year I look back on last years stuff I've written,
and I can find ten ways to clean things up.

In my opinion, if it works, and provides the utility it was designed to bring,
then it doesn't matter. If it makes money, then who really cares!

~~~
nathan_long
> In my opinion, if it works, and provides the utility it was designed to
> bring, then it doesn't matter. If it makes money, then who really cares!

This makes sense. However, if the code 1) is hard to understand (according to
the developers available) and change and 2) needs to be changed, it costs
money.

Eg, read about the expense banks are now incurring trying to maintain COBOL
systems. Whether the code is "bad" is debatable. But the fact is that they
have a hard time finding people who can work on it.

~~~
overcast
Sure, but you're acting under the premise that "if we just did it 'right' the
first time, we wouldn't have this mess". What I'm saying is that only under
very few circumstances does it ever workout that way. Particularly with long
standing systems and their software. It just builds over years, nothing you
can really do about it.

~~~
nathan_long
> Sure, but you're acting under the premise that "if we just did it 'right'
> the first time, we wouldn't have this mess".

I think you're right that every long-lived code base will have warts. And I
don't think that means that the original builders were wrong-headed.

But if you've got a decades-old system that nobody understands anymore, you've
got a huge liability. You can't ship features to compete, you can't fix bugs,
you can't comply with new regulations. You can't even rewrite confidently
because you don't know what the old system does.

There must be things you can do as a code base ages to keep it maintainable,
allow incremental rewrites, etc.

~~~
ilovetux
You may not want to reinvent the wheel, but you have to change the tires.

------
itsreallyme
300,000 lines of Python…

The company had bought Ellington, a Django based CMS, but the team basically
rewrote the entire thing using multi table inheritance (unintentionally), so
everything in the database had two copies, and we had over 70 tables, hundreds
of gigabytes, disasters every week and tons of bugs... I discovered this more
than a year later and nobody was even aware. Even the DBA wasn’t aware the
tables were duplicates.

------
dfinninger
Not the largest, but the most insane.

At an old employer there were 15,000 lines of batch script across 14 .bat
files on a Windows laptop. Old director of IT used it to onboard new
customers. It basically copied a DB and turned "CHANGE ME" in some columns to
the client's name.

It had it all. 5k lines of date validation, 3k lines of "UI", 400 goto
statements, hard-coded passwords, versioning by incrementing the file names
(leading to a bunch of code that was never called), and to top it off a static
IP granted to the laptop that used as a part of authentication.

Took me two weeks to unravel it and replace with ~20 lines of Ruby.

Later, all of my complaining on Facebook led an old professor to invite me
back to give a talk on the importance of code quality!

------
farhanhubble
More than a million lines with a single file containing more than 100,000
lines of spaghetti code consisting of macros and badly indented C code in a
single function! This powered the GUI for an entire generation of low cost
phones (pre Android era).

------
neya
A while ago, I was tasked with maintaining production code written in R by an
enthusiastic junior developer. He loved R so much that it blinded his ability
to use the right tool for the job.

Instead, he wrote web applications in R, instead of Python or Ruby, which my
company had many developers who had expertise in, and eventually handed it
over to to me. He even persuaded our bosses to invest into R Studio Server and
had an instance installed in one of our machines. It's not only the choice of
the programming language that made me furious, it's also the quality of code.
He also mixed up snake case and camel cased variables all over the code. In
addition, the same name would refer to different things, eg. `abc` and `Abc`
and `a_bc` would mean totally different things. And stuff that could be
written in a simple Sinatra or Flask application were written in R Shiny.

As a non-R person, I quickly learnt the language, (while mentally cursing it
all the way for the bad choices it had made and the terrible inexplicable
syntax it had) but getting used to this bad code was quite a challenge. We had
several top tier clients whose reports were critical and reliant on this R
code and it would frequently, randomly fail while maxing out on memory, no
matter how much you threw at it. Debugging was another issue and I struggled
with this codebase for 8 months while this junior developer had moved on to
other technologies.

Eventually, my main role _almost_ switched to devops which I hated, because I
enjoyed writing web applications and good code that doesn't require
maintenance nor devops much. Eventually, I realized I couldn't take
responsibility for this anymore as it would cost me my reputation and I really
didn't like the way the company handled the situation as well. They were quite
supportive of the junior dev encouraging him to move on to newer technologies
while he half-assed everything and threw it on other people's heads who
already had other responsibilities. They did this so that they could show off
at meetups "We use the latest tech stack..blah blah" while adding 0 value for
clients.

So, I quit the company, along with a dozen others and never looked back. But,
I did learn quite a lot..my my.

~~~
hjek
> In addition, the same name would refer to different things, eg. `abc` and
> `Abc` and `a_bc` would mean totally different things.

That's not too unusual. In Java `Camel` would usually be a class and `camel`
an object, and in Prolog `Camel` is a variable wheras `camel` is an atom. Not
sure about R though..?

> And stuff that could be written in a simple Sinatra or Flask application
> were written in R Shiny.

I don't know R Shiny, but the examples looks neat and simple.[0] Are you sure
this is not just a case of "I don't like X" rather than the code being bad?

[0]:
[https://shiny.rstudio.com/articles/basics.html](https://shiny.rstudio.com/articles/basics.html)

~~~
neya
Well, that nomenclature works well for classes and objects, but if you use mix
up cases within an object, I think we call can agree it's really bad code?

As for R Shiny, those examples all look fine, but wasn't the case with my
code.

------
pmarreck
Here's a metaquestion:

Is it possible for _any_ codebase to _NOT_ eventually (given enough time)
become a crufty pile of garbage?

I suspect (but have no real evidence... yet) that SOME of this spaghetti
garbage is due to the traits of procedural, OOP, mutable languages. But this
would then imply that things like functional-language codebases have much
longer lifespans... and I don't have evidence for that... but I'm hoping
someone can chime in

~~~
deanmoriarty
Linux kernel is massive and has been around for decades now and it’s probably
millions of lines.

It certainly has a lot of cognitive overhead since you need to always keep in
mind the context in which the code you are writing will be running (to reason
about concurrency etc.), but it’s relatively easy to understand and well
written.

~~~
yjftsjthsd-h
Yeah, I think some FOSS projects qualify; I'd say OpendBSD.

~~~
brohee
OpenBSD has a very sane culture in which deleting code, or reformatting code
without the compiler object change, are good things.

Most organizations don't see the value in that.

------
api
The worst I've personally seen was a many tens of millions of lines C++ Qt GUI
app. Won't mention the name to protect the guilty but it's something some
people here might recognize.

It didn't really need to be that big. A lot of its size was a result of
pathological over-engineering. Apparently a previous (of course) engineer who
built much of the original app thought boost::bind and functional binding was
awesome. He also absolutely loved template meta-programming.

The code base was full of templates that contained templates that contained
templates that contained... layers upon layers upon layers of templates,
generics, bind, and so on. The horror. The compile times were awesome too.
Local builds on an 8-core machine took 15-20 minutes.

I once made a commit to that code base where I replaced two entire
subdirectories of files and tens of thousands of lines of code with one
function. It took me weeks to understand and then finally to realize that
_none of it was necessary at all_. It was all over-engineering. The function
contained a case statement that did the job of layers of boost::bind and other
cruft.

I definitely had a net negative line count at that job. The experience helped
to solidify my loathing of unnecessary complexity.

It also made me respect languages like Go that purposely do not give the
programmer tools like complex templating systems, dynamic language syntax,
etc. It's not that these features have no uses, but honestly their uses are
few. I've used Go for a while now and have found maybe two instances in tens
of thousands of lines where I missed generics. I imagine I'd miss operator
overloading in heavy math code, but that's about it. The problem is that these
features are dangerous in the hands of "insufficiently lazy" programmers that
love to over-engineer. I'd rather not have them and have to kludge just a
little than to deal with code bases like the one I described above ever again.

------
gargravarr
I never actually saw the code that makes this work (thank Christ) but at the
previous place I worked, we did everything in SQL. Believing SQL Agent to be
'too limited', my boss had taken it upon himself to reimplement it... in
SQL...

Just let that sink in for a moment.

There were 20+ tables all modelled after how SQL Agent does its own
scheduling, 30+ stored procedures for interacting with it, and it was all
intended for use with a GUI that was (naturally) never written. A relationship
diagram of the tables sat on my personal Wall of Shame board for most of the
time I was there. And yes, I had to use it from time to time. It was installed
in every database for 200+ clients, in production, UAT, development, you name
it.

The most delicious irony is, the only way such a thing could run
automatically... was to use a SQL Agent job to poll it once a minute.

~~~
gargravarr
And yes, I'm aware you can programmatically create SQL Agent jobs in SQL, so
he couldn't even use that excuse. The only thing it had over SQL Agent was
that it could be 'event driven'. All that ultimately meant was that the job
would only run if a condition was met.

Yeah...

------
poulsbohemian
As many experienced commenters below have already noted, nearly any piece of
substantial, revenue-generating, long-lived code will ascend to a place full
of dragons and mysticism. The more interesting debates revolve around:

1) How did it get that way, and what can we learn from that? 2) Is it
inevitable that all software will end up like that? 3) How can an organization
ever successfully sunset or move to a more maintainable system? Or should they
even aspire to that?

~~~
nathan_long
[https://www.reuters.com/article/us-usa-banks-cobol/banks-
scr...](https://www.reuters.com/article/us-usa-banks-cobol/banks-scramble-to-
fix-old-systems-as-it-cowboys-ride-into-sunset-idUSKBN17C0D8) describe banks
struggling to maintain ancient COBOL systems as the developers who understand
them are dying off.

I don't know exactly what they should have done and when, but it seems like
rewrites are going to be necessary, and it sure would be nice to start
rewriting a system while you still have someone who can explain what it does.

~~~
poulsbohemian
In the post-Y2K years, I heard a lot about how the next order of business was
going to be to replace all those systems in health care, banking, etc,
especially as a generation went into retirement. But, here it is 2018 and
there are still articles Food for thought.

------
bogdanu
My worst experience was with a PHP app (no framework), 60K LOC per file, 6
dimension arrays, variables and comments in 3 different languages, elseif
statements spread over 2-3k LOC, no kind of separation of concerns (HTML mixed
with php, db calls, etc).

It was a 3 months nightmare with an arrogant as fudge client/developer.

------
bayindirh
I have seen a complete project. The project was maintained by one person.
Former developer left and transferred his work to the next.

It was a C project, but the second developer was coming from a Java
background. So his half is all written with Java notation and naming
conventions.

The thing was maths-heavy and had no comments. Instead they were consulting to
a documentation book which also had file names and approximated line numbers.

The error tracing was done with function call stacks, so every function pushed
some data into a global stack before a critical operation and, popped the same
data if everything went without any problems.

Some libraries they were using were ad-hoc patched to taste, and grafted into
the code tree. So, debugging was 5x harder and longer.

The functions were not divided into headers in a logical way, they were ad-
hoc. So finding something was dependent on an source indexer or a grep sprint.

Last but not the least, the developer was so arrogant that the bugs were
resolved by threatening to force him to clone his code and sent it out to
another developer to debug and clean. Otherwise he pretended that the bugs
were not present, because it was running on his test bed.

... and yes. It was in production.

------
bungie4
A nested if statement approximately 75 levels deep all because the author
didn't understand that an ID can be unique. So he manually checked the value
(which meant it could never be changed without a code change).

He did't understand the concept of a join. So he'd nest queries in VBScript
with join key supplied from the outer query to the inner. Row by row.
Essentially, a manual cursor.

Same programmer wrote an ASP portal app. The login of which got most of its
security because they didn't know how to iterate over a returned dataset. Same
code would set a cookie for access IN THE PRESENCE of a password. It could be
wrong, you would still get access. Worse, the logout function didn't delete
the access cookie, it just redirected you to the login page. Meaning you could
impersonate anybody if knew thier username. Included admin.

I once corrected bug, by using a view. I sent him the view. He had no concept
what a view was "That's like a stored procedure right?'. I'm shocked he knew
what a stored procedure was.

He's still in business and the software is deployed worldwide. He refuses to
fix it. He's a multi-millionaire.

------
mixmastamyk
Worked on a file-sync system written in Python 2.6/7 by B-players in a hurry.

Lots of microservices, before it was cool, that was fine. Shitty code
everywhere, commented code, dead code, 5 blank lines here and there. Many
lines over 100 chars, lots pushing 150 or 200+. Didn't understand how to use
argparse or logging but tried. Crazy mixed-case
WTFExtremelyLongSillyNamesEverwhereSendThis at the command-line interface,
instead of verbs like send.

Had a custom ORM that didn't want to look like one and took 5 times the code
to do similar things. Little handling of exceptions, things like wrong
permissions or IO like tar file creation might cause a 6 hour job to crash.

Daemons couldn't be shutdown gracefully, had to tail their logs until they
paused for a moment, cross fingers, and kill process, often kill -9. Old
daemontools made it more difficult. Bad timing could mean you are in for 3 to
6 hours of manual job cleanup work. Would happen a few times a week anyway,
cutting into dev time. Still you could count on it to work about 90% of the
time.

Token test suite and docs. Embarrassing web interface that would look
amateurish in the 90's. Original developer made us do a standup everyday at
10:30am just when getting into the zone. They felt worthless for a while, and
then it dawned on me why, we were all working on different projects.

The punchline: spent three months of 60 hour weeks taking out the trash,
writing tests, paying down debt. Spent the next month or two with another dev
designing/writing a vastly improved V2 with graceful shutdown, Django-style
ORM, and quality as headline features.

A few weeks before we're about to knock it out of the park and deliver, the
old author of V1 comes back in a panic, says we need to finish at end of month
as a huge project is finishing. Doesn't seem to make sense, big changes at end
are a bad idea. Takes over control of project, designs/implements V1.1,
pushing aside our improvements and whips it up in a few weeks while I sit
there with nothing to do. 6 months work of 60 hour weeks flushed down toilet.

After picking jaw off floor and offering a few choice words I left the job by
mutual agreement a few weeks later and didn't look back. Good times.

~~~
dcow
My only code style rule is don’t write lines more than 100 char which usually
means “one statement per line”. It floors me how people think it’s acceptable
to write code that you have to scroll not one but two axises to try and
comprehend. Reading code should be like reading a book. And forcing yourself
to write code that grows vertically instead of horrizontally usually solves
all the problems and more that people dumping countless hours into auto-
formatters are trying to grapple with.

~~~
mixmastamyk
He'd code fullscreen with a 16:9 monitor, so felt "it's not a problem."

------
walrus01
Not exactly code but people would be scared if they found out how much of the
Internet, at OSI layers 1, 2 and 3, is held together by the metaphorical
equivalent of duct tape and twine. At OSI layer 1, sometimes almost literally.
Major ISP outages that have been caused by somebody tripping over the $5 power
strip that was powering the authoritative nameserver.

~~~
yjftsjthsd-h
> duct tape and twine. At OSI layer 1, sometimes almost literally

How about barbed wire?
[https://news.ycombinator.com/item?id=15910263](https://news.ycombinator.com/item?id=15910263)

------
mikekchar
I worked on the DMS switch in Nortel (if PRI broke for you, I'm sorry, I tried
my best!). 31 millions lines of absolutely horrific code. You'd think
_somewhere_ in that mess there would be some redeeming code, but if there was
I never found it.

To give you some examples, I originally came on as a contractor because they
had some refactoring they wanted done. The entire system was home built
(including the programming language) and there was a file size limit of 32,767
lines. They had many _functions_ that were approaching this limit and they
didn't know what to do, so they hired me. Probably you can imagine what I did.

One time I went to a code review. They were writing a lot of data into some
pointers. I asked, "Where do you allocate the memory"? The response was, "We
don't have to allocate memory. We ran it in the lab and it didn't crash,
proving that allocating memory is a waste of time". No matter how much I tried
reasoning with them, I couldn't convince them. The code shipped like that.

One of my more amusing anecdotes is that when I worked there the release life
cycle was _five years long_. The developers would work on features for 3
years. The developers were responsible for testing that their own code worked.
There was no QA. After 3 years, we would ship the code to the telcos
(telephone companies) and they would test it for acceptance for 2 years. We
would fix the bugs that they found.

I started working there at the end of a release cycle, so people were only
fixing bugs. I got an interesting bug in that I couldn't find any code that
implemented the feature. The feature had apparently been implemented at the
beginning of the cycle (so around 4 years before), by someone who was now my C
level manager. I started looking at the other features that person had
implemented. There was no code. It seems that this enterprising person had
started work and realised that nobody would check his code for 3 whole years.
He just checked off all his work as done without actually doing anything.
Since he was an order of magnitude faster than everybody else, he was
instantly promoted into management. When I reported my findings to my manager,
he made it clear I wasn't to tell anybody else ;-)

Such a messed up place. _But_ the switch worked! It had an audit process that
went around in the background fixing up the state of all the processes that
ended up in weird states. In fact, when I worked there, nobody I worked with
knew how to programmatically hang up a call. If you were using a feature like
3 way call, etc, they would just leave one side up. Within 3 minutes, the
audit process would come by and hang up the phone. Tones features "worked"
that way -- by putting the call into weird states and waiting for the audit
process to put it back again. You could often hang up after a 3 way call, pick
up the phone and still be connected to the call.

Most people don't know it, but because of some strangeness with some of the
protocols, telcos used to "ring" their main switches with Nortel DMS switches.
This would essentially fix the protocols so that everything could talk to
everything. So, if you ever made a long distance telephone call 20 or 30 years
ago, it almost certainly went through a DMS switch. The damn thing worked.
Somehow. I have no idea how, though ;-)

~~~
nobody271
This one might take the cake. Thanks for sharing.

~~~
mooreds
I agree. It's like a buffet. Multiple stories, each frightening in their own
way.

------
stabbles
Nobody mentioned Wordpress yet? Last time I checked it was a terrible pile of
procedural code

~~~
Endy
Without a doubt. I've run into so many little "glitches" in WP that it really
soured me to CMS backed websites. The next site I run is going to be 100%
hand-coded.

... and it'll probably load a lot faster than the PHP+CSS nightmare that is
WP.

------
agentultra
A web application written in C++ before there was the STL or many libraries
written for C++... in 2007 or so. It was probably 200k LOC or so of CGI
application code, libraries to render PDFs, manage a filesystem of XML files,
an XML parser... none of it documented or tested.

The developer who was leaving the company dropped a copy of Michael Feathers'
_Working Effectively with Legacy Code_. There was a small amount of Python to
wrap the API to the C++ code using Boost against which a small suite of unit
and functional tests were being developed. I learned a lot on that project.

I never fully understood how the C++ code all worked but having that API
interface in Python helped to grok parts of it (and eventually replace it with
a few hundred lines of Python code at a time).

------
billwear
Actually, if you want to see clean code that works well, try the traditional
ex/vi source. It's something like 20K lines, but very neat functional
programming all the way down.

~~~
jackalo
I just want you to know that through rabbit holing 'traditional ex/vi source',
the following happened...

1) Read a Wikipedia page for vi.

2) Noticed that Bill Joy used a Lear Siegler ADM-3A terminal.

3) Discovered that Lear Siegler was the result of a merger between Siegler
Corporation and Lear, Inc.

4) I know that Lear, Inc. was founded by William 'Bill' Lear.

So, not only do we have Bill Lear to thank in a way for vi, but your username
is oddly familiar...

------
pacoWebConsult
12k line code-behind for an ASP.NET page that duplicates PDF files, driven off
of a spreadsheet, and adds some additional information at the top of each
page. This has generated the majority of a major state's ballots for at least
the past 4 elections.

------
cat-turner
In my past life I was an oceanographer at a consulting company. Essentially we
wrote Matlab code that processed data we get from ocean and meteorological
models. Many of these models, used by NOAA, are all written in Fortran. While
most were well documented, one thing you could not get around is the use of
parameters for these models ( think of global variables used everywhere) and a
never ending game of go-to statements. I had to figure out how this one flood
model worked... literally had to read 5,000 lines of Fortran. This is what we
use to determine potential flood impact from hurricanes, today. Software
written for oceanography is a whole other beast.

------
dotdi
I was given the opportunity to work with iOS at one of my former employers and
I quickly agreed (I was a backend developer back then). It was a rather
popular application at a rather large Austrian media company, with around 50k
unique monthly users.

Boy did I regret that decision!

\- 100KLOC

\- Initial development outsourced to India. Comments, variable names in
Indian.

\- Subsequent development outsourced to Belarus. Add comments, variable names
in Russian.

\- "Why use ObjC OO features when we can write buggy and incoherent C"?

\- Global, implicit state everywhere. Tapped on something? Hope it didn't mess
up the state you are relying on.

\- Obviously no tests, and no testers.

\- Inheritance chains of up to 20 classes.

\- iOS kindly forces MVC, but you can obviously write empty controllers and
all spaghetti-logic into your views. Needless to say, that was what they did.

\- Complete lack of proper structure. Several views iterated up their parent
controllers to the desired one, grabbed its views, iterated over them until
the right view was found and something was done to that view.

\- Building and running the application was controlled by 10+ env variables
(the other dev was fired after he pushed a dev build to the App store, which
mysteriously passed review. "Whoopsie, forgot to set one of the env variables
correctly").

\- about 80% of the logic was copy-pasted for the iPad build instead of
reusing anything. It was not a different target, it was a separate project.

~~~
charlesdm
You were "given the opportunity" huh? :)

How long did you work on that before you threw in the towel?

~~~
dotdi
About six months. I managed to actually get a new feature into that hot mess
while fixing bugs that would take days to weeks to track and handling
compatibility with a new iOS major version, but then I got an offer to work in
the space industry and quickly left.

A while later I heard they threw everything away and started an in-house
rewrite in Swift. Some people do seem to learn from their mistakes.

EDIT: clarification on the opening statement.

------
wbsun
Two case:

1\. Millions lines of messy Java code with annotations and dependency-
injection form a multi-shard distributed job with tens of distributed
downstream backends, some of which provide fake always-success synchronous
RPCs hiding the fact that the underlying operations are actually asynchronous
and may fail very often. What makes it worse to work on these code is it was
built with a single transactional underlying database but then due to
reliability issues of the single database, data are now across at least three
different transactional databases. This causes endless race conditions and
concurrency issues to fire everywhere. The production release of this
distributed job used to be once every week, now multi-months is normal, and
quarter rollback is not a surprise to people.

2\. Almost million lines of messy C++ code with several .cc/.cpp files
containing tens of thousands C++ code, some class implementations are across
multiple .cc/.cpp files. I have always been scared when touching some of these
giant .cc/.cpp files. People who have been working on the code for years can
still easily make ignorant mistakes when adding/modifying a small feature
(with so-called fully unit-test overage of course). There are multi-million
lines of testing code, which is almost 10x of the code be tested, but most of
them are bogus and almost test nothing, hence silly mistakes are everywhere,
everyday. Even the original author of the code base needs 5 follow-up fixes in
order to make a 10-line behavior change work.

For both code bases and the jobs running these code, people are now talking
about breaking them into microservices, by converting function calls in the
existing code bases into RPCs. I can foresee a tremendous number of service
outages are coming...

------
andyhasit
I might have a slightly different record, if you count lines in a single
source file...

One client wanted to hire me to continue developing his AutoCAD plugin written
in AutoLisp (a cut down version of lisp for writing macros in AutoCAD). He had
all his code in a single file which was around 63,000 lines long. (I
calculated this as 504 meters of screen/paper top to bottom).

This was for a product that was in production, used commercially, and he'd
been going for 15 years, adding bits as he went. There were loops hitting 300
lines, well over a hundred global variables, no consistency, and duplication
like you wouldn't believe because he hadn't grasped the idea of moving code
out to reusable subroutines.

I asked him what he did when his clients found a bug. The answer was "knuckle
down for a few weeks".

I just couldn't believe that this was his daily work for 15 years, and he
never thought to learn anything about programming other than what was
immediately required to solve the problem in front of him...

Another unbelievable part of this, is that AutoCAD has an absolutely superb
IDE built into it (i.e. it's fricking free!) which has features like the
ability select & run code in current scope with minimal clicks which make for
the fastest development environment I have ever played with, as in, you have
no idea how much functionality you can churn out in a day. But he didn't like
it, so he edited his code in....wait for it... WordPad! (That's a Windows
built in Rich text editor where double clicking on an underscored_word only
selects up to the underscore because it's not made for code, and that on its
own makes it impossible to work with).

I tried to modify his code for several hours but had to stop myself because it
was madness. So I told him the only way forward would be to rewrite everything
from scratch after which point I could make it do anything he wanted. I
reckoned it would only take 4 weeks to rebuild his 15 years of mess - partly
because the IDE makes development so damn quick, and was even willing to do it
on a fixed price, but he declined, which I think was utter madness.

The thing is, he had 15 high paying clients, and two other part time employees
helping with other aspects of the business, and drove a brand new Audi A3,
which means he probably holds a world record for highest ratio of money earned
to quality of code written, at least in the CAD subcategory :-D

------
a-dub
NaN kloc.

Nobody could even describe definitively where all the code could be found in
version control. The best part was that nobody trusted version control either,
so even if you did find something that looked relevant, there was a good
chance it was dead and only served the purpose of confusing you or forcing you
to memorize unstructured structure just to be able to get around.

It was madness inducing.

------
shadoxx
Here's a story I haven't shared in awhile.

One of my first gigs actually getting paid to code was getting hired on as a
last ditch effort to save what was (unknown to me at the time) a failing
business. It was a company that basically relisted real-estate auctions on
their own site, coded entirely in PHP, by a single developer who had read "How
to Code PHP in 24 Hours". They had one client that was basically keeping them
afloat. It was so disorganized that at one point I was tasked with making a
quick YouTube commercial in Adobe Premier for a client, showing off the
features of our whitelabel product with their logos, edited from a stock
template. I do not know Adobe Premier. I digress.

The main PHP file (yes) was 20,000 lines of code. Want to add a new feature?
Copy that file into a new file and save it as newfeature.php. Database
operations weren't transactional, there was no change management, and for
about a week we were using production systems to code until development
environments were made for us.

There was other shady stuff going on too, like using over a hundred proxy
accounts to scrape content from other listing sites. I refused to touch or
even look at that logic in the codebase, and it was always talked about in
kind of a hushed way. I was young, didn't know any better, would nope the eff
out if a similar opportunity came along at this stage in my career.

They folded shortly after laying off pretty much everyone but the CEO and the
lone coder. Dumpsterfire would be an understatement, but my coworkers were
chill and helped make the best out of a bad situation.

EDIT: Oh yeah! I forgot to mention the hardcoded password the was site wide
that we used as a sort of "impersonation" feature. You could type in any user
account and this password, and it would log you in no problem. No, we did not
have auditing controls.

------
rageagainst20
When I took over my current job a outsourcing company who employed developers
based in Islamabad (Pakistan) wrote most of the code. It's C#, they didn't
understand the concept of null, by ref or by value.

It was interesting. The database was Mongo and they had some horrible Entity
Framework port that they forked on Github and used which wasn't maintained and
at the time in 2015, 3 years old.

The data layer had business logic. Let's say they were getting a user record
from the database but they didn't want to include the first and last name
they'd write a method called

GetUserWithoutFirstAndLastName();

That would be in the User repository. Another requirement would come up to get
the user but not include the user's language for instance and they'd create
another method

GetUserWithoutFirstAndLastNameAndLanguage();

Ended up with about 70 or so methods which basically gave different levels of
hydration for the User object.

The frontend was written in Extjs which took about 25 seconds to load.

They had no tests.

~~~
abledon
C# 8 just announced optional nullability as a feature

------
jtolmar
All code is bad and Google puts almost everything in one repository, so I'm
going to say that ;-).

For more traditionally defined bad code, I worked with a team that rewrote an
existing service from scratch using with 40000 lines of spaghetti. The service
that it replaced was 80000 lines, and was replaced despite working perfectly
fine because it was a total clusterfuck with dependencies interwoven across
every service anyone had ever heard of. This was a payments system for a major
online retailer. All of this code has been removed since.

I also used to participate in Java4k, a competition to make games fit in 4
kilobyte jar files. Writing almost everything inside a while loop in a single
function is basically table stakes for that.

~~~
geezerjay
> I worked with a team that rewrote an existing service from scratch using
> with 40000 lines of spaghetti. The service that it replaced was 80000 lines,
> and was replaced despite working perfectly fine because it was a total
> clusterfuck with dependencies interwoven across every service anyone had
> ever heard of.

Your post should be at the top of the thread. Rewriting projects from scratch
tend to do more harm than good, and the only reason this problem isn't
addressed very often os that the people invested in reinventing the wheel
don't admit to having caused more problems than they solved, and the ones who
developed the old systems have moved on and thus are unable to say anything in
their defense.

------
ryanmarsh
Any large company. I’m being serious. As a development coach I see a lot of
code and I must say the DEFAULT is “how does any of this even work?”

~~~
TideAd
I've been working on fixing some old C++ code at my company so that it
compiles with the most recent version of the compiler. To my surprise, the
errors that the new compiler gives me are very reasonable. But I always ask
myself, "how on earth did this ever compile in the first place?"

The crazy part is going into the commit logs and seeing that a lot of the
people who wrote this have been very successful. They've become vice
presidents, retired rich, and written influential papers.

~~~
flukus
I've been doing the same, porting (getting to compile and seemingly run) an
old c and c++ codebase from 32bit solaris to 64bit linux. For a while I was
spending my days alternating between "how did this ever compile" and "how did
this ever run" when debugging. It took a lot of modifications to get it to
compile on a now 6 year old version of GCC, then I got to debug a lot of
issues with type sizes, endianess and posix differences. The team responsible
for it is still in favor of keeping it 32bit unfortunately, so we can't even
do a desperately needed valgrind run. I wish I could look at the history but I
don't have access to the clear case repository I think they're still using.

We've got another codebase of similar vintage that is preventing us from
deploying spectre patches...

------
baron816
I’d like to hear people describe why the code they’ve seen is bad.

~~~
mikekchar
In my experience (despite what you might imagine) the biggest factor has been
lack of experience for the developers. You can get some pretty weird ideas
when you first start out. Those weird ideas can gain traction in a group and
then it becomes "the way" to do it. Over time, the system gains cruft and
there is no practical way of addressing the problems. It takes _really_
experienced people who specialise in legacy code to systematically improve
large systems -- and it takes a donkey's age. It's just not a sustainable
enterprise.

We tend to think that harsh deadlines and unthinking managers are to blame,
but writing good code over the long term is incredibly difficult. Group
dynamics are hard to deal with and as you add (or replace) people on the team,
you are bound to eventually go off the rails even if you started off well.
Which is not to say that you shouldn't try, but our industry is really
immature. Most developers are quite young and even when you have a couple of
older people on the team, they may or may not have the skills for long term
design evolution. Maybe 50 years from now it will be the norm to write good
code, but I think we'll still be writing legacy messes for a while.

I should point out that even the worst code I've seen written this decade is
at least an order of magnitude better than the average code I saw when I
started my career. As an industry we _are_ improving!

------
JoeAltmaier
A 10,000-line C++ module for transmitting hundreds of different packets to a
radio module from the driver, all cut-pasted copies of the same code with
different structure name. Bugridden (bad copy-paste-edit bugs).

I replaced it with an 11-line template.

------
clausok
I worked on an Excel\vba swat team at an investment bank. Whenever someone in
the various business units got themselves into more trouble than they could
handle with an Excel model, we'd try to set things right. We saw a lot of
wonderfully creative vba code from industrious programming beginners (or, as
we liked to call them, "Macro Recording Cowboys". The most amusing I saw was a
model where the vba code used a Greek mythology variable naming convention:

Dim Hermes As String, Artemis As Long, Odin As Range

------
apnkv
A few weeks ago I had to reproduce one of the recent papers on deep learning
theory. It had ~3000 lines of code over around 10 files, but when I just
removed stale and unused calls and spaghetti (just like static analysis does),
it reduced to ~1000 lines with identical functionality.

Original code also computed the most time-consuming routine twice to thrice
each run and had many many typos, e. g. "calculte_infomration". Imagine this,
but in every second function/variable.

However, it worked.

------
gassiss
Three years ago I was hired as a Business Analyst, with 0 background in
anything related to IT. Our team was responsible to maintain a legacy base in
COBOL.

As time went by, everything started to make sense and I was able to grasp
almost everything about the environment and the tech stack. Even perform a
little bit of system analysis in COBOL to try and identify some gaps in the
code base.

But what always intrigued me is that even those senior developers in the team
with 25+ years of experience with COBOL wouldn't ever touch this one program.
The program responsible for 90% of the logic of the product of that company.
Every now and then, this program would ABEND for whatever reason and sometime,
no one could figure out why. They respected (cute way of saying they were
afraid) this program so much that, instead of refactoring it, they would just
throw an if statement and let it run.

This program was built in the 60s and had I don't know how many hundreds of
thousands of lines of code. It is still running to this day.

Now I don't have the expertise to say if that was bad code, and even if I had,
I didn't deal with this program enough to say this anyway, but I was very
intrigued why would this particular program ABEND out of nowhere, and no of
these super experienced developers would have the guts to touch it.

------
YeGoblynQueenne
There should be an award. Like the Turing award, but not quite.

The trophy should be a plate of spaghetti, in gold.

~~~
kieckerjan
Hell yeah! Like the Golden Raspberry Awards or the Bad Sex Awards, but, you
know, for code!

------
osrec
I worked in investment banking as a quant. Once had the misfortune of working
with a regulator-approved custom built market risk system from the 90s. It had
code in every language you can imagine, from cryptic Perl, to Java, to C. The
thing worked, but development cycles were a year long, primarily to allow for
regression testing. It was a monster to debug, as very few people have the
brain space to simultaneously comprehend that many moving parts.

------
wernsey
My first job out of university was to program IVR systems for a telecom
company using an in-house developed framework.

An IVR application would typically play a recording to the user, then wait for
the DTMF signal, then maybe take the user to a sub-menu, prompt the user to
choose another option and do a database query and then play another recording
and so on.

The IVRs had to repeat a menu if the user made an invalid selection, and
"press 'star' to return to the main menu" and so on.

So most of the applications I maintained were thousand line C++ functions that
looked something like this (paraphrased):

    
    
      void ivr_main(int ch) {
      main_menu:
        play("mainmenu.vox");
        d = get_dtmf(ch);
        if(user_hung_up(ch)) return;
        switch(d) {
        case '1' : goto menu_1;
        case '2' : goto menu_2;
        case '3' : goto menu_3;
        case '4' : goto menu_4; 
        }
        goto main_menu;
      menu_1:
        play("menu1.vox");
        d = get_dtmf(ch);
        if(user_hung_up(ch)) return;
        switch(d) {
        case '1': goto menu_1_1;
        case '2': goto menu_1_2;
        case '*': goto main_menu;
        }
        goto menu_1;
      // etc...
      }

------
Latteland
After many years as a software engineer, I can only say that every large old
application is a pile of messy junk. Whether it's database software at
Microsoft, visualization software at (another company), a database at a third
company, another database at a 4th company, another database at a 5th company,
they are all filled with junk and crap.

------
joddystreet
The one I am currently working with \- Multiple calls to the database (a lot
of times in a loop) \- Rather than reference, or variable passing, DB id is
passed as a parameter to functions, so multiple calls to the database for a
single object per API call \- Database design was normalized initially, for
"optimization" most of the database has been denormalized. \- Queries without
index \- If we were to create an index, probably have to create index on every
column \- one of the function has more than 36 if else (including nested if
else) \- Does not work with database partitioned or multiple databases \- OLAP
like queries running wild on OLTP systems, just because we can \- Fastest API
call - 1 second, slowest - <i dare no say> It started as an outsourced project
and the same codebase is being dragged on, with features being added every day
and with no coherent plan.

------
boomlinde
I've been patching a configuration user interface web application that uses
the same codebase for a bunch of different product categories. Some 40 files
totaling 15000 lines of HTML/js combining filepp preprocessing (a C-like build
time pre-processor) and SSI (server-side includes and conditional rendering
with directives snuck into HTML comments) split over several repositories that
are all tightly interdependent. Adding to that, obviously no one wants to
claim ownership over this codebase and changes are frequently made by people
that understandably don't know exactly what they're doing and reviewed by
other people shooting from their hips. Any change to it has to be considered
for every product that uses it, which I've never seen a full list of.

Not too many LOC but the badness/LOC ratio is terribly high.

------
cgh
Microsoft Biztalk. By "work", I mean it didn't crash. It also didn't do
anything useful.

~~~
arethuza
If SAP is how Lucifer interacts with our world then Biztalk is how Cthulhu
seeks to enter our world by contaminating the minds of innocents with its
eldritch horrors.

Mind you, it may have got better than my experiences with it, which as you can
probably guess weren't very positive.

------
jancsika
If you search youtube for something like "pure data dance music" you'll find
some excellent spaghetti diagrams that produce music/video.

Since the environment provides realtime feedback, it's a drag to try to
refactor the current diagram into a reusable abstraction. Instead, many users
optimize their creative time and just keep adding functionality to the diagram
in a single graphical window. By the time they are done there is text
overlapping other text and a bunch of lines obscuring most of the diagram.

Even with the ones that have a simple set of controls for a sequencer or
whatever, there is usually a "guts" module that hides all the spaghetti.

Also, for any running program you can instantly make it twice as spaghetti by
doing "select all" and "duplicate." :)

------
dapreja
ADP Retirement Services first generation of PES (Plan Entry System) program.
It was/is an internal tool with all the horror stories that you hear about
internal tools. The people who made it wrote obscure logic with encrypted
variable naming with the main purpose of having job security, it's a core tool
in their business model but treated like gutter trash, built with visual basic
and embedded angular code to make it look more modern, the supposed rewrite
talk went on for 8 years, and to put icing on the top it was QAd/maintained by
underqualified offshore old-school contractors (you cant even call this group
devs/engineers).

------
joecool1029
The leaked Windows source some years back was pretty damn ugly.

And it was only a small portion of the OS.

------
btbuildem
Microsoft Word back in '99 -- can't really estimate the amount of code, I
didn't have the experience to handle something like that. What I saw of it and
what I worked on, that alone made me reconsider my career path.

------
andrewf
If Foo1.php, Foo2.php and Foo3.php are all copies of the 1000-line Foo.php,
each with a different smattering of bug fixes, each included by a different 10
- 20% of the overall codebase... am I allowed to count all 4 separately?

------
rocfreddy
I worked for a cell phone company. They had a large piece of software that is
used by all customer-facing employees to manage accounts (billing, network,
provisioning). Over its life, the software had been rewritten in many
languages. Prior to my joining the team, someone thought to take a .NET
application and rebuild it as a web application (Spring MVC and ExtJS4).

The first day on the team, before even looking at the code base, I ask our
development lead what unit testing framework we are using for the Java code
and what we are using for the front end code. He gives me this funny look and
tells me to speak the guy onboarding me. I of course go and ask the lead
onboarding me and he gives me an answer that turned my world upside down.

"Automated testing is a waste of time. You will spend too much time writing
test then developing code and delivering stories to the business"

I need to point out that this guy eventually goes on to an executive level
position. proving that it is not the quality of work you deliver, but the
optics of delivering quality software that counts in large corporations. I
digress.

I receive my SVN credentials later that day and come to the realization that I
have made a very poor career choice. The front end code alone is over 5k
Javascript files with functions spanning thousands of lines long all full with
100's of nested asynchronous callbacks. Not only is the code crap, but the
tools that we had were not used correctly. For example comments.

//01/01/2018 - Fix bug - Begin //01/01/2018 - Fix bug - End

The code base was riddled with these. What story was this for? would have been
useful to pull the Jira story and see what you were trying to do. I guess I'll
spend the day going through your 5k line function that does everything, but
nothing. Or, better yet, let me check the SVN commit date. Maybe they
committed some good notes for me there.

01/01/2018 - Fix bug

Damnit!

I did my year, which is the minimum you can be in a job before posting out. I
left without a second thought. When I left the business was complaining
because the testing team was larger than the development team. I wonder why?

------
INTPenis
Bad code is a relative term. It might look awful but the design might be
decent and of course it works or it wouldn't qualify.

I'm maintaining two semi-large applications right now that I wish I had the
time to fix.

The bad stuff is basically no documentation, no attempt at pep8 compliance,
written before decorators existed in python and very dependent on Python 2
syntax.

But it works and it's actually a very good distributed monitoring system that
was ahead of its time. Closed source but it would remind people of prometheus
if it was released, yet it was several years ahead of that product.

------
geekbird
The ugliest, most incomprehensible code I've ever see was when I had to mess
with RedHat's version of Anaconda (for building distros) back in the early
2000s. It was worse than the multi-thousand line TCL screen-scraper that I had
to occasionally fix in 1999. It was the first code I'd even seen written in
Python. You couldn't really trace anything, it was all so nested and
idiomatic. It made the Perl code I inherited from a long-time Perl guru
downright comprehensible. I've looked askance at Python since.

------
blobmarley
Had to maintain a web code from the pre JSP era,i.e, servlets and writing UI
code in Java classes (think writing html in s.o.p). The beauty of it all was
that there were only two classes for the entire application: one was the
servlet, and the other held all the UI code for all the pages. Requests
bounced between the both of them, held together and dictated by a series of
"if" conditions checking for "string equals" in a method with 50 variables,
all strings, named s, s1, s2..s49. And no comments anywhere.

------
zie1ony
Magento.

~~~
zhengyi13
... which was theoretically better than OSCommerce :/

~~~
leesec
...or Websphere

------
h4y44
I was in an outsourcing team working for Panasonic door-phone system, the
source code was big, but what my team 's done was to develop some more
functionalities including video call. It was about ~20 KLOC written in C for
the UI & stuff (what we were ordered to do, since the behind layer was not
disclosed to us). The source code was a mess since it's written by low-
experienced programmers and even intern students. Later I quit that job :)

------
thom
I once had to wade into a PHP app for a government procurement website that
was just horrifyingly messy. I used to maintain records for the highest line
number on which the opening <html> tag appeared in a template (well into the
thousands) and the highest number of opening/closing PHP tags (hundreds in a
single file). There was no structure beyond bunging stuff in the session and
hoping it all hung together.

The app ran on Mac servers.

------
tluyben2
A million+ lines of Java code for a product we (in a previous company) had; it
started out nice and design pattern etc but after 5 years with ~200 different
coders adding/changing things and no time to refactor, it became bad enough to
trigger a phase out of the product and replacement by another product. I
actually moved roles to figure out how to not make the same mistakes: those
lessons work for them until today.

------
Walkman
Two projects with 300000 lines of Python code each at a small company handling
physical access control systems with smartcards and stuff. Once I looked the
most critical part; the logic which decides if a door should be opened or not
and it was implemented with the assumption that in Python, the bitwise
operators work the same way as in C (they doesn't at all).

------
Tade0
An Angular app with ~50k LOC.

Barely any tests and several hand-rolled components, some of which were merged
only because the author managed to implore somebody to give them an R+ after a
few sprints of the branch just sitting there.

A total of 21 people were involved in creating this system and for some insane
reason we still had daily standups with almost full attendance.

~~~
dasmoth
What's the problem with "hand-rolled components"? Isn't that pretty much the
essence of building non-trivial frontend stuff? Or does this have some special
meaning in the Angular world?

~~~
Tade0
Not in the case of components which could easily be replaced with a well-
tested and proven solution.

Notable among the ones in this project was the date picker - there's a decent
number of those available and yet somebody made the decision to hand-roll it.

The result was a mess that for some reason had a 300 LOC service as a part of
it. Needless to say minimal tests.

It was a waste of man-days in a project that was already over budget.

~~~
rhinoceraptor
There are a lot of date pickers, however if you have a specific UI and UX to
implement, it might not be worth trying wrangle one of them into doing what
you need it to do.

There's so many integration points (business logic, i18n/l10n, styling,
keyboard navigation, accessibility, etc) that I think it's often easier to
write your own. Hopefully now that most browsers support <input type="date"
/>, we can just use that most of the time.

------
DaveSapien
My codebase I'm working on right now. Went from a proof of concept demo, to
small mini game, to a bigger mini game, to full game, to SAS product. With no
time(budget) to rewrite it up until delivery to a customer. Its pretty
frustrating, as its my fault, but budget and tight deadlines forced my hand.

------
bigredhdl
So since we are on this subject, does anyone have advice for wrangling
spaghetti QT apps into submission? We have two of them here and of course
their creators aren't here anymore and we find it very difficult for
developers that weren't involved in their development to grok all the shared
state.

------
icedchai
Server code written in C++ that called into a lower level C library. There was
a 4000 line case statement that handled message replay. I am not sure why it
wasn't broken out into functions, but it was "sensitive" code and refactoring
it was considered a no go.

------
chad_strategic
Drupal 8

~~~
chad_strategic
I never worked in Drupal 7, but I can't image it was much better.

~~~
lowry
Drupal 6 was actually a pretty consistent hooks-based system that exploited
the fact that PHP has a fast and feature-rich implementation of the function
lookup table.

Drupal 7 started the migration to object-oriented code and was halfway through
the messy rewrite when it was released. Drupal 8 finished where Drupal 7 left.
That's where the majority of the developers left as well (pun intended).

------
UziTech
My first PHP app was an invoicing application. It is ~3500 lines of php in a
single file. Most of which is a few heredocs with javascript to be in lined on
the client side. I haven't touched the code in 10 years. I still use it for my
freelance invoicing.

------
voldacar
As a budding programmer (hobbyist), what can I do not to fall into the
mistakes others talk about in the comments? Is there any particular set of
mental habits that could lessen the chance of spaghetti occurring, or is it
just a matter of when rather than if?

~~~
ijuhoor
All these comments are good. Also ALWAYS watch out for 'Quick Win': When you
do something quick and dirty, mark it up for 'refactoring' (i.e. Tech Debt).
When your list of quick and dirty builds up, go and clean it. Do not
accumulate tech debt, try to keep it low otherwise you will pay full price
later (examples up here).

Another great tip that saved me from tons of refactoring: "The wrong
abstraction is worse code duplication". Meaning sometimes duplication of code
is better than trying to create the wrong architecture (as long as you mark it
in your quick and dirty list).

~~~
superhuzza
You don't just pay full price on technical debt, you pay it back with
interest!

------
En_gr_Student
Windows Vista. It is practically a virus, and ran ~5x slower than XP on the
same hardware.

~~~
arethuza
It wasn't even consistent - a colleague had weird errors with networking on
Vista and I didn't even though we had identical laptops.

Microsoft support was basically "yes it does that sometimes".

------
hnruss
Largest amount? 13k lines in a single Java file, none of which was
boilerplate. Who knows how many lines were in the whole project... a million
seems about right.

------
atilaneves
C functions that span over 10k lines with 46 parameters.

------
jedimastert
Not as big as some of the examples here, but still kind of interesting. I
worked for a city government, and the entire backend code for our work-bids
site was a few hundred files of PHP written with some sort of Dreamweaver
framework that was deprecated years ago. Old enough that the entire site had
to be run on PHP 5.7. I ended up rewriting the entire thing with Laravel (the
framework of choice for the other webdev that was in the IT dept). It really
didn't take that long; he just had too much on his plate to worry about it
before I showed up.

------
spsrich
has to be itunes. What a bloated pile of crap that is.

~~~
samfriedman
any entertaining examples from your time working on it?

------
brianpgordon
Once upon a time, there was a search product and one of the data sources that
it could search was a Solr/Lucene database. This should be no problem, since
search is what Solr does. It should be as simple as passing the user's query
through to Solr and then reading the response. The problem was, it was
important to know exactly which parts of any matched records were relevant to
the search.

The Guy Before Me™ decided that the best way to implement this would be to
split the user's search into individual words, perform a _separate_ search
query through Solr's HTTP API for each _individual_ word, and then do a bunch
of very clever and complex post-processing on the result sets to combine them
into a single set of results.

This led to endless headaches due to horrible performance. Imagine if you
wanted to implement web search this way. How would you synthesize the results
for the search "boston plumbers" given the search results for "boston" and the
search results for "plumbers?" You would need tens of thousands of results for
each search term to find even one match that applies to both terms. Now scale
this to getting hundreds of results to present to the user. Now scale this to
n search terms.

I was tasked with making this take less than 8,000ms for a simple query. I
spent a while getting to understand how this code worked and building out
performance tests so that we could determine how it would behave under load
(we didn't have any users yet). The results were pretty grim. I presented two
possible options for moving forward:

1\. Move this crazy result-set-intersection logic closer to the data. I could
build a custom Solr plugin to do this stuff inside the Solr server so that we
didn't need to copy gigantic result sets across the network from Solr to the
application server for every query.

2\. Delete ALL of this nonsense because _literally exactly_ what this whole
mess of code was meant to accomplish is already implemented in Solr. They call
it highlighting. It's one of the marquee features of the program. I can't
stress enough that this is precisely, perfectly, unequivocally, the exact
thing that all of this complexity was meant to accomplish.

My manager thought it would be a shame to throw away all of that very
expensive code and lose the flexibility of an in-house solution. So we went
with option one. I spent the next month writing a Solr plugin that reproduced
the original logic. It was still slow as mud so I sharded the data across
multiple Lucene servers and distributed the algorithm across them with a
map/reduce sort of scheme.

In the end, it all worked great. It was fully ten times slower than the
solution already built into Solr, but it worked.

The startup later ran out of runway trying to build a big-data-sized in-memory
distributed database from scratch to speed up search. The founder (also the
lead developer while I was there) insisted that everyone use raw C-style
arrays and a custom in-house hash table implementation because he thought STL
was too slow. Basically, "not invented here" was in the DNA of that company.
I'm surprised we even used commodity hardware and didn't design some kind of
in-house search coprocessor that would do everything in silicon.

~~~
mooreds
... it all "worked great". Wow, what a mess.

------
bayindirh
Obligatory XKCD: [https://xkcd.com/303/](https://xkcd.com/303/)

~~~
mooreds
There's an xkcd comic for everything.

~~~
bayindirh
Absolutely.

Maybe, if we pitch the idea to Randall, he may prove it. :)

~~~
dhimes
Ha ha Turing Complete xkcd...

------
shapiro92
Huge ecommerce company. Code written from the 90s in .NET was ported into PHP
using an auto-library... nothing else to say

------
sigi45
ah my godness that brings back bad memories... :|

I was at a project were every new developer said "i have never seen such bad
code". Srsly! 3 new People said it and me as well.

\- Bad testability \- no tests \- hidden 3 bad bugs surfaced in just one year
\- ...

I had very little trust in that code. And it was not much fun. On Upside:
Cleaning it up was a great feeling.

------
oraguy
Oracle Database 12.2.

It is close to 25 million lines of C code.

What an unimaginable horror! You can't change a single line of code in the
product without breaking 1000s of existing tests. Generations of programmers
have worked on that code under difficult deadlines and filled the code with
all kinds of crap.

Very complex pieces of logic, memory management, context switching, etc. are
all held together with thousands of flags. The whole code is ridden with
mysterious macros that one cannot decipher without picking a notebook and
expanding relevant pats of the macros by hand. It can take a day to two days
to really understand what a macro does.

Sometimes one needs to understand the values and the effects of 20 different
flag to predict how the code would behave in different situations. Sometimes
100s too! I am not exaggerating.

The only reason why this product is still surviving and still works is due to
literally millions of tests!

Here is how the life of an Oracle Database developer is:

\- Start working on a new bug.

\- Spend two weeks trying to understand the 20 different flags that interact
in mysterious ways to cause this bag.

\- Add one more flag to handle the new special scenario. Add a few more lines
of code that checks this flag and works around the problematic situation and
avoids the bug.

\- Submit the changes to a test farm consisting of about 100 to 200 servers
that would compile the code, build a new Oracle DB, and run the millions of
tests in a distributed fashion.

\- Go home. Come the next day and work on something else. The tests can take
20 hours to 30 hours to complete.

\- Go home. Come the next day and check your farm test results. On a good day,
there would be about 100 failing tests. On a bad day, there would be about
1000 failing tests. Pick some of these tests randomly and try to understand
what went wrong with your assumptions. Maybe there are some 10 more flags to
consider to truly understand the nature of the bug.

\- Add a few more flags in an attempt to fix the issue. Submit the changes
again for testing. Wait another 20 to 30 hours.

\- Rinse and repeat for another two weeks until you get the mysterious
incantation of the combination of flags right.

\- Finally one fine day you would succeed with 0 tests failing.

\- Add a hundred more tests for your new change to ensure that the next
developer who has the misfortune of touching this new piece of code never ends
up breaking your fix.

\- Submit the work for one final round of testing. Then submit it for review.
The review itself may take another 2 weeks to 2 months. So now move on to the
next bug to work on.

\- After 2 weeks to 2 months, when everything is complete, the code would be
finally merged into the main branch.

The above is a non-exaggerated description of the life of a programmer in
Oracle fixing a bug. Now imagine what horror it is going to be to develop a
new feature. It takes 6 months to a year (sometimes two years!) to develop a
single small feature (say something like adding a new mode of authentication
like support for AD authentication).

The fact that this product even works is nothing short of a miracle!

I don't work for Oracle anymore. Will never work for Oracle again!

~~~
Izkata
A sentiment among members of a former team was that automated tests meant you
didn't need to write understandable code - let the tests do the thinking for
you.

This, and stuff like your story, are why I don't trust people who promote
test-driven development as the best way to write clean APIs.

~~~
acroback
TDD need to die. This is a curse.

There should be integration tests along with some property based tests and
fuzzy tests. Usually catches a lot of things.Invest in monitoring and alerts
too.

TDD is like relying on debugger to solve your problem. Is debugger a good
tool? yes,it is a great tool. But using it as an excuse to avoid understanding
what happens under the hood is plain wrong.

The problem lies in industry where software engineering is not given any value
but whoteboarding and solving puzzles is.

Software engineering is a craft honed over years of making mistakes and
learning from them. You want code asap, kick experience engineers get
codemonkeys in and get a MVP.

Quality is not clever algorithm, but clear conscise logic. Code should follow
the logic, not the other way around.

Clear > clever.

~~~
ddfx
I think TDD is the best way to develop (yet). Obviously tests are code, and if
you write crappy highly-coupled tests you will end up with only much more
messy code. This is a clear example of bad testing. The greatest advantage of
TDD is in design, everything should be modular and easy to unit test, so you
could:

\- reproduce bug and verify your bugfix in matter of ms with proper unit test

\- understand what code does

\- change and refactor code whenever you want

You can tell from what is written that they are not following TDD. Redesign
that codebase in an easy and clean to test design would require an exponential
effort and time compared to have it done step by step, but it would be worth
it

~~~
astrange
A unit test is the least useful kind of test. It requires your design to be
"easy to unit test" instead of simple, and if you change something and have to
rewrite the test you might miss some logic in both pieces.

Plus the tests never break on their own because they're modular, and each time
you run a test that was obviously going to pass, you've wasted your time.

As long as you have code coverage, better to have lots of asserts and real-
world integration tests.

~~~
ddfx
Integration tests are much slower usually, and you are testing tons of things
at the same time. Something breaks (like in that example) and you have no idea
of what and why went wrong.

If you unit test _properly_ you are unit testing the business logic, that you
have to properly divide and write in a modular fashion. If you want to test a
more complex scenario, just add initial conditions or behaviors. If you can't
do that or don't know how to do that, then you don't know what your code is
doing or your code is bad designed. And that may be the case we read above.

Tests rarely break because they help you not breaking the code and
functionalities, and they are so fast and efficient on making you realizing
that that you don't feel the pain of it.

I can't imagine any example where "easy to unit test" != simple

~~~
brootstrap
in my work with python, easy to unit tests usually makes things a bit harder.
You want functional methods , not mega classes with 100s of class variables ,
and each class method operates on some portion of those class variables. It
makes it impossible to truly isolate functionality and test it. While coding
though, it is very easy to make a class and throw god knows what into the
class variable space and access those variables whenever... However if we have
staticmethods , not reliant on any class , just the arguments provided, and it
doesnt modify any class state, the test are great. We can change/refactor our
models with confidence knowing the results are all the same.

------
waleedsaud
Thank you for the question it gave us so much knowledge in the replies.

Wish you all the best

------
zerr
A million lines of mostly 90s style C++ codebase - works like a charm :)

------
sonaltr
One of the startups I worked at had 3 frameworks in PHP - all grown, all
messed up in all the wrong ways - and that was just the backend. We had a
homegrown framework written in JQuery for the frontend as well. never Again.

~~~
nickthemagicman
And probably making a ton of money so biz didn't want to replace it.

------
Jaruzel
Every line of code I've written over the past 30 years.

------
lumost
100k lines of semi generated, semi handrolled java/c

------
admin1234
I made scones with the wrong flour once. ( _¿_ )

------
cybervegan
Amavisd. About 11,000 lines of crufty perl.

------
sandwell
The most insidious cases are those where bad processes are developed around
bad software, leading to a vicious cycle of co-dependency.

I once worked for a reasonably large business that ran all of their invoicing
and stock management through an MS Access project. The whole thing was wacky.
half the business logic was implemented in VBA. The other half was stored
procedures, but there was no obvious pattern (predictably, anything that
involved a task the DB was optimised for was written in VBA). "Deployment"
consisted of saving to a network drive and waiting until everyone opened it
again in the morning. Data integrity seemed optional and inconsistent. The
symbol naming convention could be described as cryptographic. The only
documentation was a comment over each function stating the original
developer's name and a timestamp. It had a proud splash screen stating
"Developed by Dave Davidson" that was shown for 5 seconds on startup - this
was of course completely simulated and there was no reason to have a splash
screen. I could never quite fathom the magnitude of this guys delusions of
grandeur or why they would want to put their name to it.

The worst part about it all was that for the most part, it worked. The parts
that didn't work were well known to the people using it and worked around. So
processes were developed around it, and over time these became so deeply
ingrained in the teams using it that they couldn't imagine working any other
way. Most of this consisted of taking telephone orders, printing off the
invoices that were generating in Access, and then rekeying that information
into an accounting system (for efficiency purposes of course, these printed
copies were passed to another team member with notes scribbled on the original
document. Mistakes were commonplace and accepted as a CoB).

Part of my role was to implement a web based ordering system. We did a
reasonably good job but of course had plenty of our own WTFs. The biggest pain
was integrating this with a particular team. They could not imagine a process
that did not involved printing off orders and rekeying. After a while we
realised that the reason we got so much push back was that once fully
integrated, our system would make half of the team members redundant.

With support from management we went ahead, and over time the wrongs were
righted. When I left there was still a deep level of distrust in the new
system. Mistakes that were daily occurrences in the old world were "proof that
the new system won't work". Orders were still printed "just to make sure I
don't lose it". New bugs were treated as if the sky were falling. I would
spend more time managing expectations than writing code. But we got there in
the end.

Buggy software can be fixed. Buggy humans are an entirely different kettle of
fish.

------
bitwize
Windows.

------
ivthreadp110
SAS Viya. Enough said.

------
bigiain
git ls-files | xargs wc -l

<puts face in palms, starts weeping gently>

~~~
k7f
cloc .

------
weatherlight
The Internet.

------
jbigelow76
While not related to LOC, warranted nonetheless:
[https://xkcd.com/2030/](https://xkcd.com/2030/)

------
aerovistae
The entire Athenahealth stack.

~~~
__float
Perl, right?

~~~
aerovistae
The whole damn thing.

------
superdex
if it works, it's not bad code

~~~
k_
I don't think I want to work with you, ever :)

------
PavlovsCat
My own CMS. Here's the function signature of the heart of it, which depending
on circumstance may call itself recursively.

    
    
        function a_nodes_list_trees(
        	$site_url,
        	$base_url,
        	$mode,
        	$site_id,
        	$max_subtree_depth,
        	$nlevels,
        	$node_id = 0,
        	$tree_id = 0,
        	$nleft = 0,
        	$nright = 0,
        	$nlevel = 0,
        	$current_subtree_depth = 0,
        	$flags = 0,
        	$order = 'tree',
        	$inline = false, /* needed when loading stuff via ajax*/
        	$skipped_types = [],
        	$items_per_page = 10,
        	$min_tag_count = 2,
        	$min_author_count = 2,
        	$is_site_root = false,
        	$skip_pagination = false,
        	$ignore_404 = false,
        	$skip_display_options_and_batch_ops = false,
        	$display_bottom_pagination = true)
    

As you can tell from the default values, it started out having 6 arguments.
And it has things like this in it:

    
    
        	$mysql_the_rest = 'FROM
        			'.DBTP.'node n
        				LEFT JOIN
        					'.DBTP.'node n2
        						ON
        							n.tree_id = n2.id
        						AND
        							n2.perm_view & ' . A_PERMS . '
        			'.$mysql_join.'
        		WHERE
        			'.$mysql_where2.'
        			'.$mysql_where.'
        			n.perm_view & ' . A_PERMS . '
        		AND
        			n.site_id = ' . $site_id . '
        		';
        	$total_count = $A->db->fetch_one_column('c', ' SELECT COUNT(*) c '.$mysql_the_rest);
    

.. you know? The whole CMS is 20k lines of PHP, with HTML, PHP, and MySQL all
happily living together in the same files (it's not that I don't have
templates, I just have plenty HTML in the PHP, too)

Yet, it works like a charm, PHP updates made it faster even, and I can use it
for everything I needed so far, and use its output in a variety of ways. I
still want to rewrite it, but it seems a lot of work to just shave off a few
ms and have nicer code, with the same result for the visitor, and also having
to write something that migrates the content. I suspect with enough content,
it will slow down, and then I'll think about the next iteration. But it's
still a mixture of pride, plain being happy to have it, and groaning whenever
I fix a bug or add a feature.

~~~
d0m
An easy refactor for your function is to use a "Builder" for your params,
i.e.:

options = NodeBuilder .author({ min: 2 }) .flag({ .. }) .pagitation({ .. })

function a_nodes_list_trees(options) { ... }

Because in the end, if your function is fast and working correctly, it's fine
even though it's a little bit messy. The problem is more all the code calling
this function and having to pass dozens of params in the right order, and then
it's a pain to start adding/changing parameters. Also, when calling this
function, you probably need a bunch of temporary variables to "build" all the
params; all those temporary variables could instead live inside that builder
object.

~~~
PavlovsCat
I think I'll just use arrays, one for the caller, and one globally that has
the default values. Fortunately that function isn't called in that many
places.

Making and then really using the CMS helped me with knowing what I would want
in my next CMS, and just keeping the whole in mind from the start would
probably make it a lot better by itself. My thinking is that the longer I put
that off, the more languages improved and the more I hopefully learned in the
meantime ^^

------
excalibur
Windows

