
Ask HN: Share your dev horror stories - _jellyfish
Share your worst dev horror story!<p>Last year we collected dev horror stories on the CircleCI blog for Halloween https:&#x2F;&#x2F;circleci.com&#x2F;blog&#x2F;dev-horror-stories&#x2F; and it was so much fun we want to put the call out again.<p>What&#x27;s your worst? The scariest might end up on our blog roundup this year, feel free to make a note if you&#x27;d rather your story not be shared :)
======
mbrock
First job out of college. Homemade crypto plugin for SQL Server sometimes
corrupted client data, which was life & death important. Couldn't recompile
the plugin.

Discovered race condition causing use of fallback crypto key. Taught tech
support to decrypt with fallback key.

Looked through compiled machine code. Discovered race condition triggered by
particular two-byte opcode sequence that happened to be readable ASCII and
unique.

Taught tech support to search and replace that opcode sequence with Notepad++
in the DLL, changing a comma to a tilde, thus solving the problem once and for
all.

------
jetti
First dev job after 2 years of working non-tech and doing data entry. It was a
debt collection agency (took whatever dev job I could get) that specialized in
student loans. Well, it turns out they were doing interest calculations in SQL
Server. Funny thing, well not so funny, is that the output was generating
interest that was incorrect. I sketched out the numbers and did the math on
pen and paper and found the interest should be and what they were calculating
were different. I fixed the bug and things were fine after that.

The other horror story was at my current job. There was a data transformation
that needed to happen to take data from one system and put it in another. Both
databases happened to be on the same server so the person who designed this
decided to use several SQL stored procedures to accomplish this task. This is
where the fun began. First, if there was already a column named something,
such as "BirthDate" and this individual wanted to use a column with the same
name, instead of re-using the column that already exists and populating it
with the correct data, he would create a column named "BirtthDate". Now that
example is made up, but he would frequently purposely misspell words to make
unique column names to avoid conflicts with columns in the same table that had
the same name. He also would do calculations in his stored proc but then would
never use the result, or he would set the variable holding the result to a
static value lower in the SQL stored proc. It made things very interesting.

~~~
felicianotech
"BirtthDate" Oh geez. That's a scary path to go down.

~~~
jetti
Yep! It then becomes a "which one is right" situation. Luckily I moved teams
and don't have to deal with that anymore. I was close to quitting partly
because of having to deal with that too.

------
malux85
Inherited an app that was built in Marmelade - a cross platform (Android /
iOS) framework. C++ codebase.

(Note the following is a criticism of the previous developer - not of
Marmelade)

It was the most rinky-dink setup I’ve ever seen - unrepeatable build system
that relied on binary files all over the file system (in .config folders,
temporary directories, hard coded paths that only existed on devs laptop)

The texture compressor was able to fit all of the sprites into a sprite sheet
with literally kb’s of texture memory left over, we couldn’t change a sprite
because it would cause the very exact pattern of all of the other sprites to
no longer fit together by the packer

Random crashes of the app we’re common, massive libraries would be included
only to use a tiny math function within - to do a trivial calculation causing
100,000x slowdowns

Deployment involved manually building a bunch of object files, then copy +
pasting them to other folders, then building more and moving them all around,
then delete assetX from here, delete assetY from here, run packer manually,
copy files here, run linker - none of this was documented

The code was a mess - major over abstractions 10 layers deep that could have
been 2 or 3, each intermediate was instantiated seemingly randomly through the
class hierarchy, all over the place, no consistency

There were lots of anti patterns throughout the code base - some god objects
holding state they shouldn’t have, which led to lots of unreproducable race
conditions. Memory leaking common, so steady memory growth until crash.

The dev was a total nut, huge rants over nothing

I laugh looking back at it now, but at the time it was painful

------
muzani
I worked with a SaaS company, the type that builds software that other
companies depend completely on. This company was around for many years.

They pushed new code straight to the production server/DB. No safeguards.

They didn't implement any source control. All new code is emailed to the
project manager and engineers filled in a form explaining changes.

Project management is done on a checklist. In Excel. Schedules are in
spreadsheet tables. It was all managed in emails. People would have dozens of
copies of the schedules and often pulled the wrong one in a rush.

Dsspite all this email traffic, the company emails had very low size limits. A
week of work and the inboxes would be full. So people would regularly drop
stuff.

Staff were not native English speakers. They regularly typoed things and were
not consistent. On some days, the API would be "product" then "prodduk" then
"products" (actual variable name changed for privacy). So, app developers had
to change their code to work around this API.

------
Jtsummers
First job, I'm going to be vague as we did fix it but it was still rather
horrifying to be a part of.

I was in V&V at the time, not yet a developer (professionally). We had a
maintenance task, an old system that was built some 15+ years before any of us
[0] had started working for this company. Fine, it's normal. Pull up the
various documents (requirements, specs, design, test plan). Test plan? Hah! It
was only 50 or so pages. It was nowhere near comprehensive enough to verify
the system behaved correctly. So we started expanding it. We set up our test
stations and started flipping switches and developing new plans. Crap. Within
a week we found a problem. Some background: These are safety systems. We made
(for this particular project) the sensors that would be positioned in various
places (think smoke or heat detectors) _and_ the box that aggregated the
results. We then sent the aggregate results to a control panel.

So I said we took input from multiple sensors. The test plan had each sensor
tested individually. Trigger a smoke alarm on A, see it gets reported. Trigger
a smoke alarm on B, see it gets reported. We decided to trigger multiple
simultaneous alarms. I mean, it wasn't an _explicit_ requirement but it was
certainly a reasonable _derived_ requirement.

Odd. Only the first alarm shows up. Trigger A, see A. Trigger B while A is
still on, still only see A. Alright, try the other way. Shit. Same behavior,
but you only see B.

Ok, fine. What happens if we clear the alarm on A and _then_ trigger B. Oh
god. B never gets reported.

This is the _primary_ responsibility for these systems. If they aren't doing
this, they have zero value. Essentially, you would only be able to deal with
one emergency. And until a system reset, you would never be able to see any
others.

Another guy and I raised quite a stink but we were both too young to want to
risk our careers by escalating this too high. We left it about 3 levels above
us (the systems engineer). Within a year he and I were both gone, that had
left a bad taste in my mouth and he had been needing a change for a while
anyways.

Being fairly vague because the quality of _new_ software was actually pretty
high. And from my former colleagues it's only improved since I left. But those
old systems...

[0] 100% turn over the year before I got there, I did _not_ know that going
in.

------
_jellyfish
Thank you so much for your (truly terrifying) stories! Our roundup is up:
[https://circleci.com/blog/dev-horror-stories-part-
ii/](https://circleci.com/blog/dev-horror-stories-part-ii/)

