
Ask HN: How to escape the vicious cycle of technical debt - tuco86
At my company time is always running short which is the excuse for all kinds bad or short sighted decisions, IMHO. Obviously this slows us down and makes time short again. I try to raise awareness of technical dept and common anti patterns, but i don&#x27;t come through. I talked to some of my coworkers and they see the same, but all of us seem unable to change a thing.<p>I wonder if anyone can give me directions on how to improve things. Especially with the product owners&#x2F;managers that have less technical background.<p>Or maybe a piece of good advice on how to deal with that personally. I also considered if I was the problem - by somehow misjudging the situation or something - but i think like myself, so don&#x27;t spare the criticism.<p>---<p>A current example of the type of problems we face.<p>A big move from our DC to a private cloud based solution is coming up because our hoster closes that location.<p>My coworker and I where charged with building the new infrastructure. We built a sweet CI&#x2F;CD infrastructure using gitlab and kubernetes. We did in ~5 months which took ~10 devs twice that the last time. So things are improving actually - yay! All our newer services where easily migrated, new prometheus monitoring is a treat. There is the last big monolith tho - two people and a short hard deadline while maintaining all the other services. So we sat together with the relevant devs and decided to fix away all those doodads in nginx and varnish configs and make the software usable on the new infrastructure together.<p>Now product owners start to complain that their tickets don&#x27;t get completed and some even started to pull rank and moved devs to other tasks - moar tracking. They seem unable to grasp that a site can actually fail when load can&#x27;t be handled, so they don&#x27;t get stressed like i do.<p>Currently the site takes ~20 seconds to render at some places. so there is no urgent need for improvements? I don&#x27;t get it, should i just watch it fail and relax some?
======
hluska
> Or maybe a piece of good advice on how to deal with that personally. I also
> considered if I was the problem - by somehow misjudging the situation or
> something - but i think like myself, so don't spare the criticism.

Before anything else, I really like and respect your attitude. As a developer,
I would like to work with more people like you.

Further, as a developer, seeing an application take 20 seconds to render makes
me physically ill. I would feel exactly the same way you do, and ask the same
questions.

The 20 second rendering time needs to be fixed. But, in a situation like this,
there are always two options. It should be the first priority, or other things
are higher priority.

Sometimes, I think it's good for developers to look at a business as a
predator. It doesn't particularly care about anything other than staying
alive. It has to take in as many or more calories than it expends or it will
die. And, in order to catch that many calories, it has to be ruthless.

Businesses are like that. If a business consistently spends more than it
makes, it will eventually die. It has to maintain positive cash flow over a
certain period, no matter what. (That isn't completely true all the time as
tax planning can come into it, but just accept this as a helpful half truth.)
This means that when it comes to fixing problems, it's not about fixing the
bullshit, it's about fixing the most expensive bullshit first.

A 20 second render time is fucking atrocious, but maybe, at worst that will
cost $5,000 a month. Whereas, maybe there is a feature that will bring in
$8,000 a month. In that case, the new feature wins, technical debt be damned.

All isn't lost though because again, you have a good attitude and I'd bet that
other people notice that too. If you are prepared to lose graciously, you can
bring up your concerns to management.

However, it's important to speak their language. They likely can grasp that a
site will fail under load, but they might not care. They might not care
because either something else is more valuable to work on, or because they
don't know how costly your problem is. If you want to win this argument, put
together some numbers and go at it. But, always remember that a starving lion
isn't going to pass up a sick gazelle because gazelle meat gives it gas...:)

~~~
cimmanom
That's a great analogy!

I want to suggest a related tactic, which is to relate your problem to a
secondary goal that another part of the business has.

For instance, your marketing team wants your site to be ranked in the top 10
on Google for a certain key phrase where it currently ranks #100. Marketing
can probably quantify the value of that improved ranking. But Google penalizes
sites that run slowly.

Not to mention the studies that show most users bounce if a page takes more
than a second to load; and a good marketing team will consider your
(presumable) bounce rate atrocious and be trying to improve that.

The marketing team can be your ally in the fight for resources to fix this
problem.

What's the cost per hour in customer service hours and in lost sales if the
site goes down entirely? Quantify the likelihood of that outage based on
current traffic rates. Recruit the customer service and sales teams to your
cause.

------
ntuandung93
People here have already given good advice how to deal with product
owners/managers. So I'm rather going to talk about the root, which is how to
deal with technical debt in the first place.

As a developer myself, I know it's hard to review the Pull Request with good
insight just by looking at the diff. What I want to say is that we sometimes
might neglect some technical debts by not understand fully the high-level
overview of the Pull Request. So in order to avoid that, you and your fellow
devs might want to spend more times to review Pull Request and talk about the
technical debt that will be introduced by those PRs. This will vastly reduce
the amount of debt you have to pay later in my opinion.

Apart from improving it in a normal way, there's a tool for that called
Softagram. What it does is that for every Pull Request you made it will
analyze your PR and then post a comment with high-level overview graph (or
picture) about that PR directly to your PR thread. So that it's much easier to
discuss with your fellow devs about what decisions you should make regarding
the technical debt. As a result, the process of reviewing PR is much more
enjoyable, faster, and more secure. You can check out the tool at
[https://softagram.com](https://softagram.com)

~~~
totall77
In the ideal world that's how it should be, in every pull request, the
reviewers should check the architecture and implementation to fix issues
increasing the debt. Instead to be fixed later with 10-100x costs.

~~~
ntuandung93
Hi, thanks for your thought. You might want to checkout how Softagram can help
you here:
[https://www.youtube.com/watch?v=_3OzOVIOmkQ&feature=youtu.be](https://www.youtube.com/watch?v=_3OzOVIOmkQ&feature=youtu.be)

------
mabynogy
> ...using gitlab and kubernetes...

It's not technical debt. It's recent, trendy and slow stuff. It seems some bad
decisions have been made. If you're asked, start by saying that. You need to
understand why it's bad before by yourself.

> Currently the site takes ~20 seconds to render at some places.

Make something custom to solve each case. Pick what you think is the fastest
this time (not the trendiest). The fastest is static html when it's possible.

------
e_carra
You should read "The Phoenix Project", I thinks it explains very well how you
could get out of some of your problems.

~~~
tedmiston
Can you add the high-level version of that explanation here so other HN users
have an idea what it is about, and decide whether to check out the book?

------
codeonfire
two devs and five months sounds like a waste of maybe $100k. Yes I would not
touch perf if no one is complaining about it. On the other hand product
"owners" routinely drive companies into bankruptcy.

