
The shittiest project I ever worked on (2013) - luu
https://blog.plover.com/tech/prudential.html
======
bonoboTP
Having followed HN and similar dev culture (Dilbert etc.) for some time now, I
feel a growing contradiction that I cannot fully resolve.

On the one hand big corporations are presented as these incompetent behemoths
that are locked in their internal petty office politics, managers are clueless
as to what is being or should be developed, and tons of money is wasted, they
are wooed by empty marketing etc.

On the other hand these organizations are immensely wealthy and successful.

It may actually be that they are right, and for non-tech-centered companies
this whole software stuff is really peanuts and their mental energy is better
spent on other business-related stuff. So they will come up with random low-
effort comments on all sorts of details because ultimately they (rightly)
don't care. What actually matters is to negotiate deals like referral fees, or
coming up with other contract ideas that will bring a lot of revenue in.

Or maybe it's just an equilibrium, a local optimum. You don't have to produce
"good" stuff (in the developer's sense) without waste, precisely because the
other company that you're targeting is also inefficient in similar ways.

I'm really not sure but I think it's important to look beyond just "haha,
stupid managers, they can't make up their mind". There must be deeper reasons.

~~~
CPLX
I've had this same thought, and have a pet theory about it. I think the answer
is something like momentum is so unbelievably useful in business that having
it is enough to be immensely wealthy and successful.

As someone who recently created yet another startup, I am reminded daily that
literally _everything_ I do has to be done for the first time. It's like
molasses covering every part of the machine. When a client needs a proposal
you have to write one from scratch, when someone wants payment terms or to
negotiate a contract adjustment you have to figure out contract language. You
don't have expense reimbursement forms, you don't have job descriptions, you
don't have _anything_ to use as a starting point or template for anything you
do.

Big companies have all that. They have clients that are planning on using them
again this year. They have meetings they took two years ago that are about to
be a sale. They have a guy on the third floor that has seen this problem
they're about to have once and knows how to avoid it. People instinctively
trust them, so they don't have to explain themselves as much. People
instinctively fear them, so their contracts get upheld. People that used to
work for them now work at potential customers and call them up.

And on and on and on. Having a slightly busted software development process
is, in fact, peanuts compared to all this.

Or, perhaps another way to say it is that there's a lot of different kinds of
"software" besides the digital ones-and-zeroes kind we think about here. A
group of humans all organized around a business purpose with tons of
experience and momentum is a high functioning machine as well.

~~~
tmaly
This idea of momentum reminds me of Jim Collins Flywheel Effect

[https://www.jimcollins.com/concepts/the-
flywheel.html](https://www.jimcollins.com/concepts/the-flywheel.html)

I have to disagree with the broad statement that a busted software development
process is peanuts. In highly regulated industries like finance, bad software
can cost you billions in fines from regulators.

In these industries, they have to invest in technology and good processes if
they want to stay in business.

~~~
edubs25
> In highly regulated industries like finance, bad software can cost you
> billions in fines from regulators.

You're assuming the regulators are low-level enough to identify a flawed
system. A lot of regulations are applied at a policy level (aka process level)
and auditors (internal or external) are not always able to identify severe
gaps between what's documented in a policy and reality of day to day. Good
people and technology matter way more than process.

~~~
bumby
A good process is supposed to protect you from not having good people (or, at
least people who can't be good every day).

Sure, if we hire perfect people there's no reason to even have a process. In
reality, when that breaks down you have to rely on your process to close those
gaps. But, like you said, a policy/process is only as good as the paper it's
printed on if nobody follows it.

~~~
edubs25
_edit_ I'm using security as an example of a regulated aspect of software/IT
systems. I am aware there are others.

100% agree - My point was more that regulators tend to use policy and process
artifacts as proof of security compliance. The idea that more highly
regulated/audited IT sectors are inherently better at security is probably
false. The audits just aren't stringent/detailed enough and there's aren't
enough incentives for executives/boards to do more than the minimum. Which is
why the values/opinions/ethics of those decision makers becomes key.

The most secure 'company' I worked for was a large research university because
it was a priority from the director and we were constantly getting probed by
overseas IP addresses. On the other hand, the worst company I worked at in
terms of security was in the payment processing world. We were PCI DSS
compliant and audited almost continuously by various interested parties
(banks, card brands, third parties, etc.). I don't think any of our people
were bad/inept but at the end of the day our policy artifacts made us
sound/appear way more compliant than we were. The problem was upper management
didn't care (allow us time or $$) about closing the gaps in implementation
unless we failed audits or pen testing. We didn't fail but no one in IT
thought we were doing enough. We didn't have enough auditing and regularly had
inexplicable 'bugs' reported that honestly could have been evidence of network
intrusion but we had no proof either way.

------
aidos
Not me, but one of my favourites would be when a guy joined _our_ team. We
were an agency and he’d been hired by the client to eventually take over. On
the first morning I walked him through the codebase and showed him how it all
worked, how to extend it etc. After a while I said “any questions?” He
replied, “yeah, just one - am I meant to be a developer?”

Turned out that when the startup had brought him in as CTO they failed to
mention that he was eventually going to be the sole developer.

~~~
weka
Gotta ask -- how did the next few weeks/months pan out?

~~~
aidos
To his credit he just kinda sucked it up and got on with it. I was happy
because eventually I didn’t have to work on it anymore. Surprisingly, all the
money they wasted on buying rack servers and Oracle enterprise licences didn’t
net them more than 10 visitors a day and it feel apart. Once it all went pop,
him and I crossed paths on other projects too, got to trade stories about the
absurdity of it all.

------
bshimmin
I think anyone who has done any consulting with a big organisation probably
has one of these stories. As one of the other commenters says, if the end
result is something that the client is happy with, and they actually pay you,
you should basically consider this a success!

I spent six months building a single-page app (back when that was still an
exciting sounding thing) for a major US financial institution; by the time we
actually delivered it, the result was so watered down from the initial
proposal that I could've just taken some screenshots from what my designer had
produced six months earlier and put them in a PowerPoint deck and said "Done!"
\- because this was, in fact, what the VP at the company ended up doing,
except they were screenshots from the web app we built that no one other than
the VP himself ever used.

On the plus side, I became good friends with the VP and the project paid off a
significant chunk of the mortgage on my first house.

~~~
Jedi72
Good thing there are no real problems left to work on in this world or this
would be a gross mis-allocation of macro resources.

(No judgement on you OP this is the situation we're in)

~~~
clarry
Ah but if someone paid for it, the problem must be exactly that valuable to
them! And if it's valuable to someone, it must be a Real Problem!

~~~
weka
When it comes down to it, we're not paid to code. We're paid to solve problems
with code and if paying you $75/hr as a contractor to build me three webpages
with CI/CD capabilities is it, then off you go.

~~~
pault
$75? That's about half of what I would charge myself, and an agency is going
to be $300 minimum.

------
c3534l
This... this is just my life right now. Get a contract, spend all of my time
doing anything other than actually accomplish anything, and then after wasting
time adhering to project requirements that don't make sense, wind up
delivering something that could have been done in 10% of the time if people
listened to me at the start of the project.

~~~
rubber_duck
>wind up delivering something that could have been done in 10% of the time if
people listened to me at the start of the project.

I felt that way until I eventually got to a point where people did listen -
and it turned out to be false. I underestimate the effort required on first
glance and I oversimplify things in hindsight. Also I'm a developer - not a
business guy - I can guess a lot of things on the requirement side but a lot
of it is outside of my domain. This is something that I have to be aware of
now when I'm in a position to give such estimates.

Don't get me wrong - I think there's a lot of room for improvement in almost
everything I worked on before and after me - but that gut feeling of how I
could have done things was never on point for me. With the article in question
- too many people involved in the decision - no way to know the final result
for sure until people see it have a bunch of meetings, see the iterations and
decide on this - rarely do you have the person with deciding power
know/understand these things clearly from the start. And you need to account
for bureaucracy BS time-waste - it's just a fact of life when dealing with
such clients.

~~~
hamilyon2
Your comment was something like eye - opener for me.

------
vfc1
My shittiest project was a hostile takeover from another development team.

In the interview, the manager told me a bunch of BS that turned out to be
completely false: that it was a new project, that lots of new development was
expected, that there where performance challenges to be tackled.

When I got there, I found out on day one that the goal of the project was to
take over the code base from an existing development team, with which the
customer had lost trust because among other things they refused to use their
new shitty trouble ticket system (which was awful), and instead insisted in
using JIRA.

Also, they had a server running in Tomcat that they didn't want to migrate to
Websphere.

Turns out that we didn't even have access to the code, until we accidentally
found that the SVN repo was listed in an infrastructure powerpoint, and we did
have a VPN connection set up.

We were at the taking over consulting firm offices and not at the customer,
reverse engineering the existing 7-year-old codebase without any assistance
from the development team, which didn't even know that they were getting
replaced.

After months of reverse engineering and producing useless documentation so
that the client manager could say in some meeting that the system was well
documented, we ended moving in and mostly doing production support instead of
development.

Our managers were afraid that we broke something, so they did everything to
make sure that we didn't spend our time coding.

After a few months, I just asked out because it was not development work and
this type of work was actually harming my career and even had my consulting
manager actually shout to me on the phone and tell me that he would blacklist
me at his company.

Software consulting is some of the shadiest businesses out there, beware.

The amount of lies that gets told to candidates just to get them signed at the
dotted line to contracts without exit clauses (which was the case with me),
the use of outdated job descriptions, the lack of information that you have
about the actual job even if you ask a lot of questions in interviews, you
really never know what you are getting into until day one.

Here is a tip from what I have learned: ask a ton of questions about what the
job will actually be, if they answer evasively or strangely seem like they
want to avoid the questions and move on to other aspects of the interview,
that is a huge red flag.

~~~
itronitron
>> Our managers were afraid that we broke something, so they did everything to
make sure that we didn't spend our time coding.

that's like hiring football players and not letting them on the field out of
fear they'll score for the other team.

please write a book!

~~~
vfc1
Yes its insane LOL Managers were non-coders themselves, and they looked at
developers with suspicion.

This happens in a lot of projects, where developers are almost treated like
children or assembly-line workers.

I realized through small hints and queues over time that the goal was to take
over the maintenance of the project and make sure that we could fix things if
something broke, but not add any new features.

They even had a management term for what we were doing. They called it KIR -
Keep It Running! LOL

------
seanwilson
> The report I got about the demo was that the real estate people loved it, it
> was just what they wanted.

> “But,” they said, “how do we collect the referral fees?”

This is why you need to understand what problem a client wants solved and not
just build what their suggested solution is. Even if you build their suggested
solution perfectly, they're not going to be happy at the end if their original
problem isn't solved. Their suggested solution should only be used as a
starting point to understanding their requirements.

~~~
JoeAltmaier
That's a confusion about who the client is. This was a web page, to be used by
a public consumer. The real estate people were, in my view, obstructionist
old-guard trying to preserve their paper empire. Not the client. They
torpedoed an early example of the web removing the middleman and streamlining
an industry. Prudential's agents weren't ready for the revolution.

~~~
mattmanser
Actually, you're very wrong, as I've worked in that trade.

That industry relies very heavily on referral fees to this day. Part of that
is matching consumer with broker, and part is brand.

Today it's places like comparethemarket, back then it was Prudential.

So nothing's changed, the web hasn't made any efficiency gains.

The better solution than what they came up with is to have put in a unique
forwarding phone number per affiliate, and charge the affiliates a referral
fee per call (or perhaps per unique telephone number).

Whether that tech existed in the late 90s I don't know, but it's been
available for over a decade now, and was probably available back then in a
more analog form. What's nice is that now you get a webhook telling you about
each call, so all the billing is automated.

As for when you put your email address into a form on some comparison site?
Mortgage brokers bid on those leads. It used to be in the region of £20-50 per
lead, pre-2008, and I know someone charging a comparable rate now, plus a cut
of commission for subsequent remortgages.

~~~
JoeAltmaier
Ha! That just shows that web companies have figured out how to put themselves
back in as middlemen. So sad.

------
kingaillas
Great story!

Out of curiosity I wondered about the current state of the Prudential website.
So I searched, and found Berkshire Hathaway bought the real estate arm in 2012
and now they are Berkshire Hathaway Home Services.

I did a local search on the website, received pictures and basic information
(price) for each house... and for each one it lists the same toll-free number
as a contact.

So the site he built basically works the same way!

------
tmaly
As he said he was green, I think this really shows that early on, young
developers do not always understand the importance of requirements gathering
and problem understanding.

Just yesterday, a junior developer from my team came to me about a request to
generate some output from historical data. I went back to the business side
and determined it was just a simple switch of several columns in the input
data to get the results.

My junior developer at first thought it was some monumental problem that was
going to require a lot of work.

~~~
mooreds
Yup. Knowing when to go back and clearly define the requirements is a senior
type of behavior.

"We don't have time to think, just do!" is a recipe for disaster.

~~~
duxup
They said they wanted everything so the DB displays in a table ... everything.

~~~
IggleSniggle
Hey works for me, let me just get my web-scrapper going so I can--hey wait a
minute!

------
Balgair
Oh dear God ... I just finished this exact project this week (ok, 90%
similar)[0], everything he says is true.

The insanely messy data (bathroom intergers are a mess!), the firm essentially
being just a collection of smaller firms in one building, the massive issues
with 'finder fees', the near endless list of VPs that have to pass anything
off, the contractors not knowing anything, the 'real' employees also not
knowing anything, the hosting of websites, the brokers paying for useless (to
them) web-dev, it's all true! Nothing has changed in ~25 years!

[0] I know you're reading this _Jeff_.

------
mattkevan
After working in a web agency for years, my number 1 red flag that the project
is going to go badly is the quality of their current website they’re hiring
you to fix/replace.

If it’s solid but just a bit old or needs more features, you’re fine. But, if
it’s crap, look out.

The quality or lack of is never down to the the people who built it (unless it
was by the MD’s nephew or something), especially if the client blames them for
it.

It just shows they can’t run a project properly and whatever you end up with
will be just a slightly newer pile of crap. And they’ll slag you off to the
next lot.

Number two red flag is if they say at any point “You tell us, you’re the
experts!” This can be translated to, “We don’t know what we want but we’re
going to reject whatever you say anyway.”

~~~
mdpopescu
> Number two red flag is if they say at any point “You tell us, you’re the
> experts!” This can be translated to, “We don’t know what we want but we’re
> going to reject whatever you say anyway.”

I disagree with this being a red flag. That's exactly why I hired someone to
build me a small website. It's not because I don't know HTML / CSS / JS; it's
because I'm bad at (visual) design. I wanted someone who knows what they're
doing to use _their_ expertise and give me a good product.

~~~
mattkevan
That’s great, you know to hire an expert and let them get on with it.

The problems start when the client either can’t or won’t explain what they
want, then gets upset that what you produce doesn’t match what’s in their
heads. At that point they start trying to undo every decision and things go
south rapidly.

------
dvh
If he was paid by the hour it would be typical, perfectly reasonable
contractor project.

~~~
twervle
Consulting companies know this and they NEVER charge fixed price.

~~~
thirdsun
However there's a case for fixed pricing and it can work to your advantage:
Think of a difficult task or problem that you already solved in similar cases
- maybe you already have the heavy foundation of that work ready as a
template, maybe you automated complex, repeating steps. Or just think of a
simple task that has huge value for your customer. Pricing shouldn't always
depend on how long it takes you to finish a project.

------
DonHopkins
At least he didn't have to install Solaris on Sun executive's workstations.

[http://www.art.net/~hopkins/Don/unix-
haters/slowlaris/worst-...](http://www.art.net/~hopkins/Don/unix-
haters/slowlaris/worst-job.html)

~~~
skrebbel
I feel like I'm missing something in that article. Probably some historical
context. Is it really about execs at Sun pranking each other by installing
Sun's own flagships OS on each others' computers? Isn't that like "pranking"
Satya Nadella by making him use Windows 10?

~~~
bdsa
"On September 4, 1991, Sun announced that it would replace its existing BSD-
derived Unix, SunOS 4, with one based on SVR4. This was identified internally
as SunOS 5, but a new marketing name was introduced at the same time: Solaris
2."

I guess the execs didn't have a high opinion of their new product.

~~~
DonHopkins
The Day SunOS Died

    
    
        "Bye, bye, SunOS 4.1.3!
        ATT System V has replaced BSD.
        You can cling to the standards of the industry
        But only if you pay the right fee -- 
        Only if you pay the right fee . . ."
    

[http://www.poppyfields.net/filks/00070.html](http://www.poppyfields.net/filks/00070.html)

For context, the guy who wrote "The Worst Job in the World" email was Michael
Tiemann, one of "open source's great explainers." ;) Now he's pranking IBM
executives by installing RedHat Enterprise Linux on their mainframes.

[https://en.wikipedia.org/wiki/Michael_Tiemann](https://en.wikipedia.org/wiki/Michael_Tiemann)

------
silveroriole
The only thing that’s ever worked for me to avoid these scenarios is getting
all the relevant stakeholders physically in a room for a couple days and
drawing mockups, looking at their systems, learning the domain lingo, running
reports over their data, etc as we go. Otherwise the domain knowledge for a
proper solution just isn’t there, even for a ‘simple’ project.

Unfortunately I also hated every minute of doing this and have switched to a
back-end job where I never have to interact with a customer again!

------
juskrey
I can say, as a seasoned web apps outsourcing contractor, there are two sides
of the medal.

The bright side: clients usually come to me with just an idea, plan or
sketches, almost all of my projects were built from the ground up, I totally
technically owned them. This is a delight for any coder (and I code a lot, not
just manage the team).

The dark side: many startups and green business ventures fail, and the failure
rate is around 80%, even larger in the longer run. And I do the apps from the
ground up, nurture them.. but sometimes I feel myself a gravedigger.. out of
any 10 projects maybe 2 or three are still there after a couple of years.

------
zandokc09
It's an amusing story, I've seen this countless times as well. I think what
finally sunk in for me was that if you're focused on saving money or
efficiency on small projects then the upper limit of benefit to the company is
less than the budget for the project. Instead if you focus on optimizing for
the market, the profits can be unbounded essentially, and any wasted money to
get to market is simply part of the learning process.

And if you're the contractor having to run around as the plans change, well
just make sure you get paid enough :)

~~~
Xylakant
As contractor I like jobs where I can basically recommend “Just don’t build
this at all.” For example the young startup that wanted credit card payment
and automation for essentially a two digit number of invoices per year for a
target market where bank transfer would be much more common. Replaced with an
excel sheet. Customer happy. I don’t want to build crap just for the money
that’s in building crap. Let me actually add value.

~~~
aitchnyu
Umm... did you earn the kind dof money the pointless project would have gotten
you?

~~~
lapnitnelav
Probably no but they surely built goodwill with that company and as a result
can help them build a good reputation. This is often overlooked but people
will remember that.

Or they could just be satisfied with doing the right thing, rather than trying
to extract as much money as they could.

------
jmartinpetersen
Thanks for sharing. Good catch at the end. It's easy to forget this as a
developer. But usually technically details are just that in the bigger picture
- details. They're only important if the bigger picture is somewhat correct.

------
beat
As an aside, "Tell me about a project that failed" is one of my favorite
interview questions to ask. Failure is endemic in our industry. A candidate
that only tells you about success isn't telling you their whole story.

And a candidate who talks about failures in terms of what everyone else did
wrong, but never acknowledges any failures or learning experiences of their
own, is telling you something important, too... they're telling you that
either they never learn, or they don't tell you the whole story.

------
SubuSS
These are the instances where I feel it is definitely worth keeping some
people around who act as glue. (or old timers).

In our company - we have a tech lead role which is supposed to be the
technical point of contact for a given team. This person is expected to have
answers or have a way to find the right answers about the nuances of the team.

The good ones are invaluable. Whenever new projects come up, the modus
operandi is to write a proposal, review it with your current team for sanity
(on whether it achieves its purpose) and then with this group of relevant
stake holders / tech leads to ensure you're not getting some giant thing wrong
that can nuke some other part of the business.

The problem I see with this role though - we (I am the one for analytics and
one of the TLs for Data Infrastructure at Snap) have to handle 20-25 hours of
meetings around all these coordination AND ensure some work gets done in your
own area so that you are in touch with the code bases you are fiddling with.
So as much of an alluring title that is, you are giving up deep dive
development (like I was able to do building storage engines for DynamoDB).
Tradeoffs :/

And there are cases where the tech leads don't have the answers (new or
incompetent) OR they think of a weird case couple of months down the line, but
this works out overall.

I probably should write a blog about this - but anyway, I don't know how this
works out in the non-software world: Why wouldn't you have a knowledge
person/council around?

~~~
sequoia
> The problem I see with this role though - we (I am the one for analytics and
> one of the TLs for Data Infrastructure at Snap) have to handle 20-25 hours
> of meetings around all these coordination AND ensure some work gets done in
> your own area so that you are in touch with the code bases you are fiddling
> with.

If you figure out the answer to this PLEASE share it with me. This describes
my current role; whenever I focus on one aspect (eg actually writing a few
lines of code) the other (keeping up with planning) suffers.

My suspicion is that “tech lead” in this context basically means “manager,”
but employers know many devs are scared of the word manager, so we have the
highly ambiguous “tech lead” title, which allows dev managers to be in denial
about their actual role.

------
pbhjpbhj
There's a lot wrong with this whole thing but the end solution is the wrong
way around too - you don't require the client to then take a further step, you
get their contact details and then reduce the friction by calling them.

The solution should surely be "enter your phone number and zip code and we'll
contact you". Then the central referral agency contact the person immediately,
to keep the momentum & check the person's details, reward them for their
effort in finding the company, and arrange for the next step -- ideally an at
home meeting with the particular agent you booked them with.

Just "right, now call this number" is a bit daft, isn't it?!

------
pjc50
> "how do we collect the referral fees?"

This is the underlying problem, and the reason why "disruptive" startups can
be successful in the first place; making something more beneficial for the
consumer in a big organisation can be at the expense of one of the individual
units which can prevent it, even if the change would be advantageous for the
business as a whole.

------
AnnoyingSwede
This is something every freelancer have seen. Who ever introduces you to the
job rarely knows what the end-product should look like and it really proves
the value of creating a minimum viable product, showing it to customer and
pivot from that to where you might need to go. It this case the MVP would have
been the final product.

------
mathattack
Sounds like the OP fessed up to the problem at the top: despite hating large
companies, they signed up to a fixed bid project as an independent contractor.
Given how big companies work, that’s recipe for danger.

Scale is very powerful, but it takes a lot of work to get things done. Large
companies have a ton of waste, but getting to “this can work for all of us”
has a very big impact.

------
hyper_reality
Fun story containing a valuable lesson for developers thinking about going
into consulting, especially for larger companies.

------
vmurthy
Do you think this is why developers need to be made aware of user journeys? In
my experience, just spending some time on thinking through the user journeys
has done wonders to my understanding of features.

Edit : The article doesn't mention anything about user journeys. Hence the
thought :-)

~~~
arethuza
In an ideal world yes - but in some situations its just not practical. For
example, I once worked on a (large) project for a company I worked for where
we were the customer and developers were out-sourced about four levels deep -
they weren't even clear as to who the top level customer was let alone what
the user journey was.

------
Iv
In the end, something overpriced but usable was delivered. This project was a
success. I wish I could say that of all the projects I have seen.

------
yibg
Feels like things like this are super common in the consulting / non-tech
company world. A lot of small factions all wanting their little requirement
slotted in, and the whole product end up either being completely incoherent or
there are large parts that are wasteful.

For me it was a project a while ago with a telecom company. They wanted a
place to manage their customers' communications. One group was adamant that we
needed to surface customer's emails in this portal, instead of providing say
just a link to whatever their email provider is. Essentially they wanted us to
build a clone of gmail / hotmail / yahoo mail but only with SMTP / POP / IMAP
access to the data. We tried to tell them how expensive and not useful this
would be, but no had to build it.

Obviously we couldn't build a gmail clone in the time we had that was better
than gmail, and there was pretty much no incentive for users to use our
version instead of gmail proper. So after spending more than half of the total
project time (and budget) on this email functionality, six months after launch
we had a very small handful of users linking their email and most of them
didn't do anything with it. Even after being presented with the usage data, we
couldn't just get rid of this feature, because that would be admitting
failure.

------
eXpl0it3r
Wish the article would point out in the end, that this is a perfect example of
why you have to talk to the "final" customer at the start of a project.

------
Tistel
I just started at a large international firm. ~1 week in I found that a
collection of beefy AWS machines had been sitting around doing nothing for a
least 5 years (potentially 7 years). The base cost? ~100k USD (we are based in
Canada, so ~130k for us). Also, three of the machine (you have to have dev,
staging and prod!) where running a (presumably) expensive third party
financial software library. No one knew who to talk to about the cost of that
lib. When I told my boss, the CTO, they just shrugged and implied that it was
peanuts in the grande scheme of things. As far as I can tell the C-Level
people just break up their days going to meetings talking and kick all the
technical debt down the road. I find such waste upsetting, but, try to focus
it as fuel to get my own company going (if successful it too will be
disgustingly inefficient in 10 years!). _sigh_ 1/3 of the best years of my
brief spark of life on utter nonsense. probably best not to think about that.

------
madprops
I've been becoming less cynic about workplace bosses and their decisions. I'm
not really talking from raw workplace experience, but from noticing things
I've read. I think a boss order comes from a perspective that is _probably_
right. They have an understanding that a developer _might_ not have yet, their
instinct could be accurate, maybe most of the time.

------
ahallock
While it was frustrating, think about all the crucial experience (business
more so than technical) you gained.

I would rather face a few bad projects early on to learn how to manage
expectations, negotiate, handle sunk costs, and not get overly emotionally
invested. I've had a project pulled from production a day after launch.
Developing a thick skin is very important. If you are only thinking about what
the project means to you and the payoff seeing it go to production, as opposed
to the larger business concerns, you are being narcissistic and setting
yourself up for failure. Conversely, a lot of decisions are imprudent and bad
for the overall business and you have to know when to abandon ship.

------
IloveHN84
This is the typical case where starting small and simple helps you

1) save significant amount of time ahead

2) give your customer time to identify what they really want

3) help you iterate quickly on newer versions. If something doesn't work, it's
easy to throw stuff away

~~~
Gibbon1
All good advice.

Old army observation. No Battle Plan Survives First Contact With The Enemy.
Thus the sooner you get that over with the better.

~~~
shezi
As far as I recall, that's actually a quote: „Kein Plan überlebt die erste
Feindberührung.„ by General Moltke. Your translation is exact.

Just wanted to add a source:
[https://en.wikipedia.org/wiki/Helmuth_von_Moltke_the_Elder](https://en.wikipedia.org/wiki/Helmuth_von_Moltke_the_Elder)

~~~
yesbabyyes
It's also said, about Moltke the Elder, that he only ever laughed twice in his
life. The first time was when his mother-in-law passed away, the other time,
when he saw Sweden's Vaxholm fortress.

[https://en.wikipedia.org/wiki/Vaxholm_Fortress](https://en.wikipedia.org/wiki/Vaxholm_Fortress)

------
nojvek
Big companies are diverse. They have deep roots. Google and Microsoft can
afford to indulge in spruce goose’s for years and it wouldn’t make a
difference. As long has google has their search engine, they’re racking in
money. Same with Office.

In a big company, you are a cog, a little pawn. It doesn’t matter though. They
have a huge army and they can afford to burn people and it doesn’t make a
dent. They can just outright buy people. Everyone can adjust their morals if
they have extra zeroes at end of their pay check.

------
wazoox
My shittiest project was conceiving, developing and deploying WorldCup '98
live TV graphics. I worked 100 hours/week for 4 months, put on 10 kgs, lost
one tooth, and got like a 150$ bonus for the whole thing.

I've posted parts of the story as Quora comments, here's one:

A company named Compusport had secured the contract to provide visually
impressive graphics with statistics for WC98, but had the hubristic project of
developing the graphic stack from scratch — and that was too ambitious, and a
few short weeks before the World Cup the guy in charge with this part threw
the towel in… Uh oh.

My company stupidly proposed to help Compusport (the fools :D ). At the time I
was CG operator on French TV, and the only person in Europe comfortable with a
high-end, IRIX character generator program called Antero. So I teamed with a
colleague to build a PC GUI in Visual Basic to drive Antero through the
network, and that proved … tough :)

There was several technical hurdles: Compusport main selling point was
advanced statistics (it was supposed to be fully automated, IA powered ball
and players tracking. This didn’t actually work out, but this is another long,
painful and interesting story). So we needed to feed the statistics to the TV
screen, coming from a central server. Graphics had to be animated (then a
world first at this scale), so the CG system had to be remote controlled from
the mixing desk to launch animations and effects. Of course on the different
sites there were different mixers (Philips, Sony, GVG, Thomson…) working in
subtly different ways. We even had to build some custom electronics to
interface some of them with our system… An horribly complex network of various
hardware and computers using ethernet, ISDN, serial lines and video… a
complete nightmare :)

As a “rehearsal”, Compusport has planned to use the French Football Cup finals
on May 2, 1998. Oh, by the way he also “sold” the graphic design for this
event. With noone in sight to actually do it, you know, like a graphic artist,
and no budget either.

So here how the typical April day was: my colleague Eric was building the GUI
that remotely controlled Antero, communicated with the video mixer (so that
the FX operator could control Antero animations from the main mixing desk) and
the statistics server (to grab game data to feed to Antero) from 8 A.M. to 10
P.M. or later. Way later. I helped him on the Antero API (“to display that
page send this and this and then wait for ACK then send this and this”), while
during the day working with Marc Tatou to set up the graphics and the
animations : “make the red card come forward than flip. A bit faster. A bit
closer. That’s it”. Of course there was no GUI to program the animations, I
had to write scripts with coordinates and vectors in some weird domain-
specific language (apparently similar to the one used on some Ampex digital
effect hardware) and proceed by trial and error (that grows your beard).

The rest of the day (and night) was spent building the graphic charter for the
French Cup finals myself — I became very proficient in Photoshop quickly…

Finally on May 1st, at 4 A.M., I was done with the graphics, I tapped Eric’s
shoulder “look, I’m done!”. No response. I turned to him. He was sleeping
soundly, his forehead firmly stuck on his keyboard. We had lost him. Burnt out
without recourse. Fallen in combat.

Obviously the GUI wasn’t ready for prime time on national TV. The girl who was
supposed to be in charge of CG graphics during the match bailed out in terror
(understandably). So without having any sleep for the past 72 hours, I had to
manage it during the match, the old way (without any automation whatsoever)
which was quite tricky as Antero missed a critical feature, which is the timer
(it could only display the system time). I had to write a quick perl script to
manage the timer during the match… by resetting the clock at the referee’s
whistle and switching the display from a static 00:00 page to the rolling
counter. Would have been actually easier to use a good old Chyron Max! but
hey, that’s how it went.

After that, we had to find someone else to develop the remote control program.

To be followed…

~~~
barking
This is a fantastic story, I remember world cup '98 very well, did you get a
chance to enjoy France's victory at all?

~~~
wazoox
Not really. To go on with the story, there were 20 stadiums across the country
with the whole setup (remote control PCs, SGI Octanes, etc) so every day we
worked hard to either correct bugs or implement required features (because we
were so late), then I proceeded to push the new program to the central server.

Usually every day, before the match the guy in charge of the setup at the
stadium would call me to ask if there was any changes to the setup, programs,
etc. One day in Lyon there was to be the first match that could end with extra
time. We implemented the required functionality in the remote program during
the previous night, so I was home a few minutes before the match was about to
begin when I realized that Carlos (the guy in charge) hadn't called me. Uh
oh... So while the broadcast began, I called him "hey Carlos, you didn't call!
-Yeah, no worry, I know how tired you are I didn't want to bother you, I can
manage it... \- Sure, but did you install the new version, the one that
support extended time? -Uh, no... -Well, let's pray..."

You should know that the program was several tens of megabytes large, and had
to be downloaded through an ISDN line (64k). The very same line had to stay
free during the match to transmit statistics data to the central server, so it
was absolutely necessary to download the installer at least a full hour before
the match...

What do you think happened? I watched the match in anguish. Half-time, still
0-0. 89 minutes, 0-0. God hates us... End of regular time, 0-0, extra time!

Carlos calls back: "\- what should I do now? \- just reset the counter and run
the extra time like a regular new match."

The problem was that the program didn't manage the final score display, with
the scores for the two half-times and two extra time periods...

So when the match ended, I told Carlos: "-OK, so you have to run the final
score card manually. As the program can't send the correct data to fill in the
screen, you must enter the data manually"

Fortunately I had sweated long nights on these screens, I knew precisely how
many fields there were ... So I said: "-go to the Octane keyboard. On the
keypad, press 1001 + enter. Press Tab 4 times. Press 0. Press tab twice. Press
0. Press Tab 3 times. Press 2. Press Tab twice. Press 1. Press ctrl + "+" on
the Keypad. Press Ctrl+Enter. Say "On air!" to the director through the
intercom".

I was watching my TV, like 1 billion other people. The score card appeared,
with the correct values. I can't overstate how relieved I was.

So during the finals, for the very first time, the program was complete.
Nothing was missing. Everything went on perfectly smoothly. So sitting next to
the Octanes, I spent the whole match playing Unreal on a spare PC. I didn't
watch a single second of the finals. I was done with football once and for
all. I've never watch a single match ever since.

------
jlarocco
Maybe I've been around too long, but this doesn't seem too bad. The guy made a
lot of mistakes (flat fee, not talking to users, etc.) and then suffered the
consequences.

As far as the shifting requirements, management nitpicking, "dirty" imported
data, and all of that, it sounds like any software project at any medium or
large company. It describes a lot of projects at smaller companies, too, for
that matter.

------
unixhero
Reminds me of this: [https://projectfailures.wordpress.com/2008/06/24/project-
fro...](https://projectfailures.wordpress.com/2008/06/24/project-from-hell/)

Discussed here before:
[https://news.ycombinator.com/item?id=16793884](https://news.ycombinator.com/item?id=16793884)

------
ineedasername
I am embedded within an operational unit. It's a mixed blessing of sorts, but
one of the advantages is that, knowing the operational side very well,
something like the referral fee issue in the article would occurr to me pretty
quickly. Although the downside of a semi decentralized arrangement like this
is communication silos that must be actively dealt with to prevent
fragmentation of purpose.

------
lostmymind66
I decided to quit my job and travel around Asia about a decade ago and didn't
want to teach English, so I worked as a remote software contractor instead.

The project was for a person that was running a side-company while working
remotely as a project manager for a Fortune 100 company.

The development team was from various parts of India, Guatemala, and
Mexico..and I was the only one that spoke English fluently (The project
manager was from Guatemala and translated/managed all of the developers from
Mexico..they didn't speak any English). I only got the position because I cut
my rate in half, as my cost of living was really low, and I didn't want to
have a project that was too involved/stressful.

Some highlights:

The company owner was obsessed with the Scrum process, most likely because
this is what she used at her main job. We would have 3 hours of meetings
almost every day. On Fridays, we would have an extra hour, where we would be
required to tell fun stories or jokes. Since we were all from different
countries and most didn't even speak the same language, it was just...awkward.

The interview process was a written English test (I had to take one too). The
manager wasn't really that experienced and would give repo access immediately.
The result was chaos. We spent an entire month with a broken repo because
someone was overwriting the master branch constantly. I was paid during this
time to just keep checking in changes from one particular bug fix, which would
be working when I was finished with everyone's changes..and then the next day,
the master branch would be overwritten.

Turnover was high. I was there for a year and we went through over 50
developers. We had one guy come into our meeting room after he was hired, keep
interrupting us, and then eventually told us to fuck off. I think at some
point, I stayed because I wanted to see where the train wreck was going and
simply because I didn't feel like finding another gig.

The project was a way that made filling out government forms easier. There
would be major form updates on a monthly basis, which required lots of code
changes. The code base was originally written by an outsourced team, which was
horrible. There were two, parallel versions running: original and beta. We
were all working on the beta, which was supposed to be the better version.
However, it was a complete mess of spaghetti code. I found code, written by
the current project manager, where he had copy and pasted examples he found
online and left the comments in explaining that it was sample code.

When I received a new task to be completed, it was very difficult to figure
out what needed to be done. Bug fixes weren't difficult, but most feature
requests were tickets from months or even years earlier and some of the
requests weren't even relevant anymore. I would call the boss, to try to get
clarity, and she had this way of using corporate speak and talking in circles
to say lots of words, but never get the information that I need. When I did,
it was almost always wrong. I resorted to just completing the task wrong
(which I hated) and I would have to correct it later.

We all had to have our web cams on during the meetings. When the boss shared
her screen. Every...single...time, she would forget that she had personal
emails on the screen..and we would get to see her sexually suggestive emails
she was sending to her boyfriend that day. One time I saw everyone's pay in a
spreadsheet. Most developers on the team were getting paid $5/hour or less.

We were expected to work all major US holidays and weekends (I just flat out
refused to work on any weekend). We had a meeting on July 4th (since I was in
Asia, I this didn't really matter to me) one time and she was in the midst of
the aftermath of a party. She looked sloppy and there were people passed out
on the floor in the background.

When I finally flew back home to the US, she called me up one day and told me
that I was too expensive and she could hire 3 people from India for what it
costs to pay me.

------
JoeAltmaier
Another takeaway: Prudential skimmed fake 'referral fees' for a task that a
web page could do just as well. Of course this was in the old protectionist
days when the web was new. But what dense, backward thinking went into this
final decision to keep the old phone/paper system in the end.

------
perfunctory
Everybody who believes that private sector is by definition more efficient
than public should take note.

------
logfromblammo
They could have a contact form with something along the lines of "click here
to allow this agent to contact you", and then the affiliates pay for the
leads.

They were so worried about getting paid, they forgot to invent a new way to
get paid.

------
tempodox
Working from non-existent or hare-brained specs is the classic front runner of
project shittiness metrics. The article exposes an entertaining and realistic
example.

------
raheemm
I love reading stories like these. They are insightful and funny.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=6442359](https://news.ycombinator.com/item?id=6442359)

------
User23
This reads like Conway’s Law, a novel. I love it.

------
Ice_cream_suit
A relative works in a senior role at a medium sized hedge fund.

To a close approximation, software contributes zero to their bottom line.

------
itronitron
this is just a typical application development story

------
TheRealDunkirk
Just a niggle, but "senior web engineer"... in 1995? Uh... given that Netscape
Navigator was released in DECEMBER of 1994, can any of us say we were a
"senior" "web" ANYTHING in 1995? Sure, I was doing Usenet back on Unix
machines in the late 80's in college, but that's not "web." And in 1994, I was
dual booting Win3.11 (with Trumpet winsock) and Slackware, and getting on the
internet with a free SLIP account, but I was doing Usenet, IRC, FTP, and
gopher. Come on, now. "Pretty new" doesn't even begin to describe it. Am I
missing something here?

~~~
mjd
Yeah, you're missing that I was there really early.

I was in charge of all the programming and system administration for
pathfinder.com, at that time the central web site for all of Time Warner,
starting from spring of 1995. At the time I was hired I already had a lot more
experience than a lot of other people around; I had been a sysadmin and
webmaster for the computer science department of a large university.

I think “pretty new” is accurate. Both the CGI standard and NCSA Mosaic had
been released in 1993, with the <FORM> element specified in 1994. I had been
running a popular personal web site with innovative features since very early
1994.

------
otabdeveloper1
> Prudential Real Estate is a franchise operation. Prudential does not
> actually broker any real estate. Instead, a local franchisee pays a fee for
> the use of the name and logo and other services.

I don't know how contracting works, but for corporate project the rule is
always "follow the money", then once you figure out the money flow, only then
do the requirement gathering bit.

Doing it in the opposite order will result in tears.

------
draw_down
That is pretty awful. But you know, and maybe this is just me being too
fatalistic due to what’s happening in my own professional life, but most of
what we do is a waste of time in some sense.

Just shovel the shit, get paid, and move on. They want it changed? great,
change it. They want a form that does something stupid, fine, build it. A lot
of people spend their work day ringing up plastic crap that people don’t
really need, or whatever. Which is to say it could be worse.

Of course part of what sucks about this story is that the author failed to do
the “get paid” part correctly, but I’m speaking generally. I just can’t get
too attached to outcomes anymore.

~~~
RhodesianHunter
This is depressing to read. It's still a very in demand line of work. If your
day to day is that miserable try finding some place that isn't.

~~~
barking
I can empathise with his point of view. Sometimes all going the extra mile
gets you is a shrug of the shoulders or worse, grief.

If tv didn't exist, people would be more impressed with you for inventing
blurry 405 line b&w than high definition colour.

The latter makes it look easy, the former gives them some inkling that it's a
technically difficult task.

They'll be more grateful and impressed as a consequence.

------
labster
That was quite the shaggy dog project. I'd say more, but it would spoil the
disappointing ending.

------
rb808
Wow I searched the comments for agile and no hits. This is basic modern
project management - get something in front of the customers and users as soon
as possible and iterate from there.

~~~
xondono
Yes, because that never goes wrong...

Maybe it’s because I’m a hardware guy, but if anything, agile has normalized a
lot of bad practices (like developing before getting any specs) that can only
be made to work in software development.

~~~
rb808
Perhaps you're right, but the article here says that is why spec's often dont
work. The guy followed the specs, but the it wasn't what the customers wanted.

