
It Takes 6 Days to Change 1 Line of Code (2015) - Mz
http://edw519.posthaven.com/it-takes-6-days-to-change-1-line-of-code
======
hacker_9
I want to fault this but I can't. Code reviewer who's not afraid to reject
code? Check. Proper testing of code changes? Check.

The process probably looks silly in this case because nothing went wrong. But
with a hard-coded variable being changed, who is to say it won't break some
other system by being changed? And if the variable was then moved to a config
file, who's to say again that it might not break in some obscure way? Such as
if the config file is not found in the uat/test environment? Additionally
after all this, the manager can change the variable in the config file as they
please without this process need going through again.

~~~
justinsaccount
The process is fine, but sometimes shit is broken and you need to fix it now
and clean up the existing mess later.

Why not instead of

> Shirley (Code Review): It is now against company policy to have any hard-
> coded variables. You will have to make this a record in the Parameters file.
> Also, there are 2 old Debug commands, an unassigned variable warning
> message, and a hard-coded Employee ID that will all have to be fixed before
> this module can be moved to production.

Add at the end:

However, I'll allow the change for now because this is an important update
that affects peoples jobs. I want to see every issue fixed before the end of
the week and I will not approve any other changes until these issues are
resolved.

There, win-win.

~~~
absrnd
Because everything is urgent, all of the time.. and so nothing gets improved.

~~~
ErnestedCode
"I know this is priority 1, but is it priority 1A, 1B, or 1C?"

~~~
ryandrake
Every numeric priority scoring system I've ever worked with at every company
I've ever worked with has devolved into this: Start with P1 (high), P2
(medium), and P3 (low). Sooner or later, an exec will swoop in and declare
something Even Higher Priority Than P1!!! So, "P0" gets invented, and after
some time, you have tons of P0 issues, meaning P1 is the new "medium", P2 is
the new "low" and P3 becomes "never". Then something comes in that is Even
Higher Priority than P0!!! And now, you're stuck.

For this reason, I always advocate reversing the numbers. The higher the P
number, the higher the priority. So you never run out of numbers to express
how much more urgent and earth shattering your next bug is. Try it!

~~~
umanwizard
I worked at a big company that had a big data warehouse that you could submit
jobs to: these jobs could be given one of five priorities from "lowest" to
"highest". Naturally, everyone set their jobs to "highest", so jobs that
generated some weekly email report that nobody really read were competing at
the same level as jobs that were actually business critical (e.g., necessary
for ad auctions to work correctly or whatever).

So they introduced a new priority level: "Critical". The pattern repeated
itself in short order.

It was a tragedy of the commons thing, and eventually it was literally
impossible for a job to ever finish if it was marked anything other than
"Critical".

So they introduced "Omega Critical". Now you needed to be on a permissions
list maintained by the database team to raise something to "Omega Critical". I
got on the list (because I was scheduling actually high-priority stuff). Then
everyone in my org would come to me and ask me to set random bullshit to Omega
Critical, because they knew I had permissions and "one more job won't hurt"...

I don't work there anymore, but I believe they have since developed even more
weird categories like "Q4 Omega Critical" and who knows what else.

~~~
zhte415
I've seen 2 pretty good solutions to this.

1\. VP bangs head with equivalent VP in other department, complaining there
are too many requests. Can work, depends on individual.

2\. Actually have a service level agreement. Requests are limited to X per
hour/day/week/whatever the process needs, and billed accordingly. Have a bit
of buffer in this, a little creative billing. Requests within SLA are dealt
with in agreed turnaround time. Have a dashboard that clearly shows this to
customers. Alert customers if they're consistently submitting what's outside
of SLA, and they will receive a degradation of service unless they provide
further funding.

Everyone knows if you pour too much water unto a cup it will overflow.

~~~
mdpopescu
Having a budget for each department and a cost for running the queries is a
great idea... I'll try to implement it the next time I'm in this situation. (I
have also had a ton of VPs - plus the CEO and the _owner_ \- come to me with
"very urgent now now now!" requests... it would have been great to tell them
"the top priority for me is this query worth $1000, can you beat that?)

~~~
zhte415
Sure. That's how external service providers work - being internal is in the
buffer, you can 'do a favour' when needed. If it is the owner making an
ASAP!OMG! request perhaps take the chance for a chat and to demonstrate you
have a system and that you're not only managing, but taking a stance of
leadership (and point out that as it is internally charged, they're not losing
any cash).

------
ztratar
This is why startups kill big companies.

Lots of stupid little processes that add up and make the managers feel power,
but end up killing the performance of the business.

Brings up the interesting differences between leadership and management.
Managing is what gets people into situations like this, whereas leadership (in
late stage companies) is what gets people questioning these processes instead
of moving like cattle into a slaughter.

I saw this exact type of behavior at General Electric back in 2011. People
wanted to argue about acronyms and process design all day, whilst Samsung ate
up 20% market share in 1 year. That business of GE no longer exists and was
sold to a foreign buyer. Move fast or die -- this rule does not only apply to
startups!

~~~
maxxxxx
Considering how inefficient big companies are it's surprising that they don't
get killed in the market immediately. You would expect that there are plenty
of nimble newcomers who kick their asses but that's not really happening.

~~~
Afforess
Two words: Regulatory Capture.

See Also: Boeing, IBM, CGI Group, AT&T, NRC, etc.

~~~
grzm
I don't read that as being the point of the article. Regulation is not even
mentioned or alluded to. Also, my understanding of regulatory capture has
little to do with regulation slowing down business. If anything, rather the
opposite. But that's off-topic anyway, and definitely not appropriate for this
week.

Edited to update comment on regulatory capture.

~~~
Afforess
I don't know what you're talking about, it's in there plain as day. The
company has a department called _IT Security_. Not IT. Not Security. IT
Security. Companies that legitimately care about security integrate security
personnel with all of their operations, into every department. Top-down
security departments exist to check boxes on regulatory forms. So I will be
shocked if "IT Security" is not a made-to-order department whose existence is
solely to satisfy things like PCI, HIPAA, etc.

~~~
grzm
I understand your point if you're referring to government regulation. I don't
think your argument makes sense if you're referring to "regulatory capture":

 _Regulatory capture is a form of government failure that occurs when a
regulatory agency, created to act in the public interest, instead advances the
commercial or political concerns of special interest groups that dominate the
industry or sector it is charged with regulating._ [0]

[0]:
[https://en.wikipedia.org/wiki/Regulatory_capture](https://en.wikipedia.org/wiki/Regulatory_capture)

In the case of regulatory capture, you'd think regulation would most likely be
_reduced_ rather than impeded.

~~~
dragonwriter
> In the case of regulatory capture, you'd think regulation would most likely
> be reduced

No, in regulatory capture you expect regulation to be increased so as to form
a barrier to entry to new competitors, while advantaging (comparatively)
established incumbents that have the inside track with regulators and the
regulatory process.

~~~
grzm
Makes sense. Thanks for the insight. My believe main point that using the term
"regulatory capture" is not what the commenter intended or is appropriate in
this case (as it's regulation in general that would affect this rather than
tangential effects of regulatory capture).

------
valine
Related anecdote: I interned for a medical device company over summer of 2015
and worked on a team that was in charge of maintaining several implantable
class 3 medical devices. The devices used a short range proprietary wireless
standard to talk with a intermediary device which could be connected to via
Bluetooth. One product had a pretty serious usability issue where connecting
to the intermediary device could take upwards of several minutes. After the
issue was discovered it took roughly an afternoon and several lines of code to
fix. However, in order to provide software updates for a class 3 medical
device each update needs to be individually approved by the FDA. It took 12
months from the time the fix was implemented to when the FDA allowed it to be
pushed out.

~~~
joshvm
What is the approval process? Do the FDA manually check the device (similar to
FCC certification for wireless things)?

~~~
valine
I never had the opportunity to be involved in the FDA submission process, but
from what I understand the FDA does not do any testing of their own. It's the
companies responsibility to prove to the FDA that their device is safe and
effective. All the testing is done by the device company, carefully
documented, and then reviewed by the FDA. I've heard stories that the
documentation for a class 3 medical device can be upwards of 20,000 pages
which are all printed and handed to the FDA in 3 ring binders.

------
cthulhuology
I this had happened at any of the companies I started, I would have fired
several people in the aftermath. The problem is the policy is broken by
design. Let's look at what is wrong here:

1.) priority is not just high, it is critical, communicating this is lost at
each layer (executive, planning, execution, process control, quality control)
2.) leadership is lax, the chain of command doesn't designate a clear single
responsible individual 3.) policy enforcement in this example actually
_increases_ the risk of an unsatisfactory outcome, by increasing the
complexity of the solution vs. what is in production 4.) quality control is
adversarial and ass backwards, code review is supposed to be a sanity check
"does this code do what the developer thinks it does" aka. "can some other
person understand it" 5.) test planning should not be the developer's
responsibility, quite frankly if QA can't figure out if it is working or not
you should fire your QA department. 6.) Ultimately, it is a total failure of
policy and management as it requires the President of the company to
micromanage the situation.

If you think any of this is fine, I'm sorry but your company is doomed to fail
(unless it is already so big it is too big to fail).

~~~
kevinstubbs
You would fire them, instead of trying to build up their skills? I'm not even
sure who you would fire in this situation, the IT director for not shepherding
the ticket all the way through, or Shirley for closely following company
policy? Either would be overreacting.

Slow/inflexible company policy has been built up, presumably under the IT
Director's watch. The president didn't communicate that it was extremely
urgent (and maybe it wasn't - the article doesn't say when the change actually
needed to be made by to make a difference). Ideally, management can recognize
that this change took a week because of their accumulated decisions, and lack
of urgency. They should figure out exactly how their company policy is slowing
them down instead of letting heads roll.

~~~
user5994461
Don't fire. Don't touch anything.

Just remember the chapter 12 of Peopleware.

Between two organizations, the productivity can vary by 10 times. It's not
about you, it's a social characteristic of the organisation.

Just leave and find one whom speed suits you.

------
lucasnemeth
For me the absurd part was this one:

"Philip (President): Our factory is underutilized by 10%. Either we start
building more of our backlog or we lay people off. I'd rather keep everyone
busy, build inventory, and get ahead of the curve before the busy season. How
can we do that?

Lee (Operations Manager): Company policy restricts us from building more than
3 months of backlog. If you just change that to 4 months, we'll have plenty of
work."

You can't have one week of 10% less productivity if not you will immediately
fire everyone? And you company policies only work if you change a line of
code? And this line of code is from some old legacy code that no one touch and
your whole process depends on it? No, 6 days of development is the least of
the problems here. This company is absurd.

6 days for the line of code wasn't optimal, but wasn't that bad. What this
story shows to me how bad management pretend things are IT fault.

~~~
rpeden
It didn't mention firing anyone. When I see 'layoff' used in the context of
manufacturing, it usually means something more like an furlough (i.e. the laid
off employees stay home and don't get paid until the company has enough work
for them to do). In this case, it seems they'd rather not do it, but if the
software change drags on long enough, they'd have to start layoffs.

As for requiring a code change to implement new policy; maybe not ideal, but
it looks like they're trying to move away from that with the requirement that
hard coded constants be moved to a parameters file.

And in general, using the company's production software to enforce business
policies is probably an important management control. Often when you see rules
like that, it's because the company got burned by something in the past and
wants to enforce a process that prevents it from happening again. In this
case, maybe in the past an operations manager was evaluated based on how busy
the factory does, and so just kept building up more and more inventory to game
the metrics he/she was being judged by.

I think it was a tad inaccurate of the manager to call it 'legacy' software.
It might be an old system, but it appears to be critical to the company's
operations, and it seems to be something that is worked on fairly regularly.
Even if it's written in Cobol and JCL, I'm not sure it would be fair to call
it legacy software unless the company is actively working on a replacement and
making only necessary changes to the old system in the interim.

So, while I understand your concerns, I'm not sure it's fair to pin this on
bad management. It sounds to me like a company that's probably been in
operation for a long time, and has processes in place to make sure nobody
pushes any changes to a production-critical system without proper approval and
testing. They could possibly streamline the process, but it doesn't seem
(necessarily) crazy based on the information we were given.

~~~
lucasnemeth
Ugh. that's sad. I'll be sure to avoid old manufacturing companies when
looking for jobs then.

------
kazinator
That can be okay if the _same_ 6 days can be used in parallel to change 999
_other_ lines of code, too.

I.e. "1 line in 6 days" is only a measure of latency, and not of throughput.

~~~
vacri
> _a measure of latency, and not of throughput._

You could drive a station wagon full of backup tapes quite far in 6 days :)

~~~
kazinator
It's not about how far, but how fast it is going. The bandwidth of that wagon
is about how fast those tapes pass a given point of travel. How they are
arranged inside the vehicle is critical. Tapes moving at 100 km/h side by side
have more instantaneous or "burst" bandwidth than tapes moving at 100 km/h in
a single file.

------
makecheck
A risk of allowing too much bureaucracy in commit processes is that engineers
try to avoid the slowdown as much as possible. For example, several unrelated
changes may bleed together into “super commits” so as to clear bureaucratic
hurdles only once. Next thing you know, it’s harder than ever to identify the
exact cause of any one issue: you have more changes in each commit, fewer
comments than ever, and a high probability that no thorough review was
_really_ conducted.

Large blobs make the review process completely fall apart. In the past I dug
up too many cases of people allowing 50-file steamrollers into a repository
when the entire “review” was literally two words (“looks good”). While this
probably happened due to time constraints, it makes the entire thing
pointless. With a simpler commit process, engineers might submit smaller
change requests at a time and people asked to review “7 files” might do a
thorough job instead of balking at requests to review “50 files”.

~~~
KKKKkkkk1
One of the things that happens as a result of anal code reviews is that at
some point you learn to commit as little as you can get away with (and this
includes things like comments and tests).

~~~
dom0
Avoiding comments or diagnostics/logging, because these two will be picked on
_every single time_.

"Don't have it debug log 'Foo'd 5 bars', make it 'Foodulated five bar things'"

------
ipsin
In Case of Emergency Break Glass

There really needs to be a procedure for violating procedure, so that when
higher-ups say "do this immediately", those same higher-ups can own the
consequences of violating procedure, and so that they know the business
implications of the rollback plan, etc.

If the bug is "people are getting laid off" or "our customer database is being
siphoned by an SQL injection", it probably should not wait.

~~~
Falkon1313
The process sounds fine, no need to violate it. The problem was that the IT
Admin said "I'll put this on the fast track" but didn't, and no one else saw
the urgency. It was just dumped at the back of the queue behind a lot of other
things and mistakenly labeled an 'Enhancement'.

Proper handling would have had a dev working on it within the hour. The
response to the code review would have been "other changes are out of scope,
open another ticket for enhancements". If the developer didn't already have
access to the test environment, at least IT would know that they're authorized
for it. (Critical fixes shouldn't be assigned to a new hire who doesn't know
the code base and doesn't have access to commit code yet.) The ticket would be
edited (Test Case: view WorkOrdersHours report, Expected Result: ~10% higher
total) and tester would be told to reread it and go ahead. And of course
documenting the test runs is the tester's responsibility (they're the one
doing the test runs), not the developer's, as is notifying the IT Admin and
Operations Manager when the testing is done and it's ready for sign off.

Exact same process, same quality checking along the way, timeframe reduced
from 6 days to 4 hours or less; likely less than 1 if the team works together
well and their tools are decent. A good process can handle emergencies without
sacrificing quality, as long as the people work together and the priority is
communicated.

------
perilunar
I once waited 6 weeks for a backend team to fix a prematurely closed div tag
that was screwing up the layout of a news site. The templates they were given
were good; they screwed up the implementation. In desperation I had to hack
the layout with JS (which I was able to update at will).

~~~
rewrew
That kills me to hear that. I'm in charge of Web where I am and we make 95
percent of our changes instantly. There is no reason to make people wait; it
honestly takes longer to project manage most changes than to just make them,
so we just make them, but of course Web sites/front end is a different beast
than application or other code.

------
xarope
I don't get this, whatever happened to a hot fix in minutes, and then a proper
fix 6 days later? Have we forgotten how to do this? Together with some other
reasonable comments, I can't fault the process in the story, but I just can't
get why they didn't allow a hot fix?

Then it would be a different title; It takes 5 minutes to approve the change
to 1 line of code and rolled into production, and then 6 days to get an
official patch - done in accordance with proper IT policy.

~~~
user5994461
What makes you think that they have CI and CD? It's entirely possible that
just building & shipping is a 1 full day procedure.

------
nsoldiac
I'm surprised no one has raised the problem of senior managers and VPs
underestimating and oversimplifying the complexity and LOE of work. It causes
unrealistic timelines for others who do have to adhere to a release process
(which is never perfect but certainly necessary). "Yes CEO, of course we can
get that in by tomorrow, no problem."

When the work misses the impossible deadline the feedback usually points to an
underperforming organization and rarely to unrealistic estimation.

------
dosethree
Agree with some other comments here:

Good: Code review? Great. Boy scout rules enforced in legacy code? awesome.
The QE analysis seems great!

Bad: Failure to expedite? Fail. And without that there's no story.

Also at a high level, the fact that it is only 1 line of code is irrelevant.
It's not closely correlated to the riskiness of the change.

------
patmcguire
Favorite line:

"Ed: Fuck that shit."

"Shirley: That may very well be true."

------
dsego
Policies are organizational scar tissue. [https://m.signalvnoise.com/dont-
scar-on-the-first-cut-e9afd2...](https://m.signalvnoise.com/dont-scar-on-the-
first-cut-e9afd256afdf#.f1od1i2ax)

------
HeavyStorm
I work on a consulting company. We have extremely aggressive deadlines, and
work with less than capable people on both sides of the table.

This means sometimes that takes a usual programmer a few minutes to change a
line of code and promote it to UAT.

Forget the number of bugs that happens after this. Yeah, the number is VERY
high, but is subjective. Think about the overall state of the program. When
all that matters is velocity, people takes all forms of shortcuts. OO is out
of the window. Hardcode? What about things that should've been parameters and
are not only hard-coded, but spread along many files?

TL;DR OP case seems contrived, or at least the other extreme. Processes should
be followed. If the factory stopped because of that single line - or any
other, someone would ask: all this to save 6 man days?

------
shakencrew
This has been discussed on Hacker News before
([https://news.ycombinator.com/item?id=3989136](https://news.ycombinator.com/item?id=3989136)).
The cited article was published in 2012 at a different domain that is no
longer alive. Thanks to the Wayback Machine, we have copies. Here's one:
[https://web.archive.org/web/20120623032243/http://edweissman...](https://web.archive.org/web/20120623032243/http://edweissman.com/it-
takes-6-days-to-change-1-line-of-code)

------
siliconc0w
Change control needs to be structured around risk. I.e how much $ are you
losing a second given an outage. Also, not all changes have the same risk.
Your change control process should be malleable to these realities. If a
change is low risk and the reward is high then it shouldn't need the same
controls as a high risk change. Also, there are more ways to mitigate risk
other than additional steps in the release process.

TLDR absolutists make poor release engineers.

------
joeld42
This is why it's important to have a "hotfix" process where you can make a
change fast and then still follow up with all the QA and coding standards.

------
yason
The bad thing with bureaucracy and slow processes is that eventually you'll
find yourself in a situation where you either 1) start working on something
else while waiting on some process to complete and get your changes merged,
and then you gradually forget to play your turns in the process, delaying it
even further, or 2) waste days sitting on the important change, pushing it
forward, being active and responsive, and basically just avoiding any other
non-trivial task because of #1 happens.

So you need to decide whether you want to get real work done or if you want to
get the one task done, for which coding took a matter of minutes or hours, but
the process eats up days.

------
suls
To change the frame of reference a bit, how would this story work out at
Google?

I just have an outsider perspective on this, but I really think that monorepos
& automation that comes with it will have an amplifying effect in the years to
come ..

------
mnarayan01
Requiring all your constants to be configurable seems like a great way to
bypass (potentially critical) code review and QA. I can totally see changing
something like "number of months" causing extremely non-obvious issues (e.g.
integer overflow in something 20 stack levels away)...once you make it a
configurable constant though, changing it won't be QA'd at all.

------
partycoder
If technical debt accumulates, you slow down until the point you need a lot of
time to understand the consequences of changing code.

Pretty much like the end game in Jenga.

One good approach is SOLID: [https://en.wikipedia.org/wiki/SOLID_(object-
oriented_design)](https://en.wikipedia.org/wiki/SOLID_\(object-
oriented_design\))

------
cdevs
It's hilarious to read but part of the events that lead up to this are
compliance , audit trails, security, investors and customers want to know
every 5 minutes some dev isn't jumping into vim and saving new code on
production and taking things down with a syntax error.

------
eykanal
The only difference between this and a large modern tech company is that the
latter has this process automated. If anything, large tech companies have
_more_ respect for the need for process, as they understand how critical it is
for everything to run smoothly.

------
patmcguire
It seems like buried in all the other policies in one meta-policy: nothing
gets done until someone with sufficient authority overrides all the rules.

How often does the president have to do this? Has anything ever been done the
correct way?

~~~
spydum
Reality is in a large org, process can usually be sidestepped with emergency
change requests and executive sponsor. Which is basically what happened here,
it just took extra days because it wasn't started that way.

------
sthatipamala
Are there really factories whose policies and parameters are encoded in
software vs. under the guidance of humans?

That in itself is amazing to me considering that the tech company I work at,
lots of things are just done by fiat.

------
wwarneck
Since this was a critical issue I think "David" should've brought key
stakeholders (or leads) together and set expectations for who was doing what
parts and when it would be complete.

------
keithnz
there's a saying, "if it hurts, do it more often" (
[https://www.martinfowler.com/bliki/FrequencyReducesDifficult...](https://www.martinfowler.com/bliki/FrequencyReducesDifficulty.html)
)

these guys should do all their PRs as one line (or a few lines ) changes and
put it through the system and improve the system till they can get things
deployed within an hour ( or some reasonable time period )

------
awqrre
They should also limit the number of lines that are allowed to be
changed/added/removed every 6 days... and maybe it should be a law for popular
products.

------
manigandham
Why did they hard code a variable in the first place? Hundreds of comments
about technical debt when it's just shitty engineering.

------
daveheq
This sounds like bureaucratic nonsense waiting for signatures and approvals.
Maybe the task flow needs to improve.

------
timwaagh
these processes are beyond fixing. if you absorbed into a corporate monolith
that values 'best practise' you'll just have to put up with it. however this
really would not happen at any shop that is concerned with saving money.

------
joslin01
I don't think I would eat 24 excedrin over something like this.

------
manarth
This sounds like perfect is the enemy of good.

------
nickthemagicman
Thats a dream orocess right there right?

The solution imo is to tell the higher ups to fast track it if peoples jobs
arecat stake.

------
Davidbrcz
This is true beauty

------
smegel
Become a $$$/hour contractor and learn to love the bureaucracy.

Then wait for those all-too-often moments where a manager's ass is on the line
if feature X doesn't ship by tomorrow and watch all the rules and regulations
get chucked wholesale out the window. Then chuckle.

~~~
eplanit
Exactly. The mantra I was taught when I first began consulting, when faced
with frustrations like these was (adapted from the old school Hawaii Five-O):
"Bill 'em, Danno!" I've been at it for 20+ years and it truly works.

------
edblarney
Surely there could be some optimizations, but there are no gross failures
here.

At every step there were reasonable policies it seems.

It's not wise to frame this issue as '1 line of code' because '1 line of code'
could ruin a company.

It's frustrating ... but it's not that bad.

------
Wellshit
Well, I've seen 5 line patch take 6 months (and counting), so...

~~~
weavie
I've had 6 months of work thrown out because management changed its mind over
which approach to take. So 0 lines in 6 months..

------
bbcbasic
TRWTF is that it is considered a problem that it took 6 days. Sounds like
there was no harm done and 6 days was OK.

------
renownedmedia
[2015]

