
It Takes 6 Days to Change 1 Line of Code - jacobr
http://edweissman.com/it-takes-6-days-to-change-1-line-of-code
======
bengl3rt
Honestly, for code that controls millions of dollars worth of business and
could seriously screw over a lot of people if a subtle flaw was introduced, I
don't see a problem with doing it this way.

Might be overkill for one line of code, but where do you set the threshold? If
you say any change longer than four lines is subject to this process, then
people will work on trickling in new stuff in increments of four lines at a
time.

Also, I can do a lot of damage in one line.

~~~
benjaminwootton
Reliability and quality are obviously important, but you have to question
whether all these steps are actually helping you to achieve that.

All of these items in the story add overhead and reduce agility, but
potentially do _very little_ to improve quality:

    
    
      - Mandatory paper trail via fields on the issue tracker
      - The need for documented internal sign offs 
      - Standards and policies with no clear reason for the existence
      - 'Micro' Code reviews
      - Excessive permissions and security processes internally
      - No pragmatism with regards to testing the change
    

This kind of stuff is a constant and substantial overhead on getting anything
done. You need to look at these processes very critically and be very sure of
the benefits in terms of quality before introducing it.

~~~
ajross
Exactly. The point bengl3rt was making, I assume, is that cavalier avoidance
of process is a bad thing because it allows mistakes to happen that would be
caught. That much is true.

But the assumption that's wrong is that _all_ processes avoid mistakes.
Clearly they don't.

And some of the examples here are just plain cargo cult misapplications of
good ideas. The point behind code review is to catch design flaws in _new_
code. You never want to demand refactoring of existing code in order to patch
features, that _hurts_ , it doesn't help.

~~~
theoj
This quote is appropriate here:

Skilled people without a process will always find a way to get things done.
Skill begets process. But process doesn’t beget skill. Following a recipe
won’t make you a great chef – it just means you can make a competent
bolognese. Great chefs don’t need cookery books. They know their medium and
their ingredients so well that they can find excellent combinations as they
go. The recipe becomes a natural by-product of their work.

From [http://the-pastry-box-project.net/cennydd-
bowles/2012-march-...](http://the-pastry-box-project.net/cennydd-
bowles/2012-march-28/)

~~~
vacri
Er... great chefs _do_ need cookery books. They may not refer to them as
often, but you won't find many chefs out there without a collection of cookery
books. They still use recipes for things they're not familiar with.

Besides, skill may beget _product_ , but it doesn't necessarily beget
_process_.

~~~
theoj
>> Besides, skill may beget product, but it doesn't necessarily beget process.

Oh boy, talk about not seeing the forest for the trees. For skill to beget a
successful product, you must follow a process to get from nothing to product.
So in the course of creating a successful product you have automatically
created a process -- the process to build the product. That process may be
used one time, or multiple times -- but it's still a process.

~~~
vacri
Not seeing the forest for the trees? Hell, if we're going to use that loose a
definition, _everything_ is a process, regardless of skill level. Even if you
end up without a product, to get the steaming pile of crap you abandoned, you
went through a process.

------
dools
I seriously don't see the problem. The author seems to be assuming that taking
6 days to have a change made is objectively bad.

Does the company run? Does it make money? Do the systems work and stuff?

Well, if whatever you're doing now to keep the systems working and the company
running and making money obviously works so I'd say it's objectively good.

The experience may feel bad subjectively but how you feel is of trivial
importance when compared with the health of the company as a whole.

It's all about risk right? Like if the company wants to spend $50,000 to
change 1 line of code, and it's not going under, then that's obviously the
right decision. It's just the same as spending millions of dollars per year on
insurance and then complaining when the place doesn't burn to the ground.

The "move fast and break things" attitude of Facebook works for Facebook -
it's a different company.

Small startups can break whatever the hell they want because they're not
making any money yet.

An established company with a functional system that makes money has to take
measures to protect it, because that functional system is where _all_ the
value is in the company.

~~~
zwischenzug
I don't see the problem either.

The article is extremely self-centred, and doesn't deconstruct why this is bad
for their business.

The programmer's job is to help the business achieve its ends. If she feels
stymied by process she needs to discuss it with the business. If she's
criticised for taking too long she needs to lay out where the time went.

Imagine if lawyers wrote articles about how it took months to convict someone
that's obviously guilty?

~~~
IgorPartola
The business wanted to s/3/4/ on a single line. The reality is that President,
CEO, CIO, etc. don't really care about whether this parameter is configurable,
if there are specific test plans, if the test team is happy with the variable
name, etc. They don't want to do a layoff. So they asked the developer to
perform the s/3/4/.

The developer did so, reasonably. Then it sounds like the self-righteous
testing and security people stepped in and basically said "well, this might be
delayed because it doesn't follow our standards and we don't have all the
sign-offs. This might mean layoffs, lost profits and so on, but damn it we
have to follow our policies!"

Sounds like the IT dept. is not enabling the business. They are, instead,
hijacking it, substituting their own priorities for those of the whole
enterprise.

~~~
roc
IME process doesn't spring fully-formed from the head of Zeus.

You get test plans when somebody goes cowboy on live code and breaks
tangentially related processes. You get security reviews when somebody goes
cowboy on live code, everything looks great, and then a month down the line
you lose days of work when someone changes a password. You get mandates to
create parameter files when someone gets burned having to run through this
entire process over and over again when business rule changes back and forth.

The reality is that once this change is done the proper way and the business
parameter is moved to a file, the company will _never lose time over this
issue again_. It will be configurable just as fast and responsive as we all
feel it should be, without any concerns about breaking things down the line.

Is it frustrating? Sure. But that doesn't mean it's _wrong_.

Going cowboy on live code, even for something as trivial as changing a 3 to a
4, can and has created large problems.

~~~
IgorPartola
Agreed. Process comes from a need. My point is that it sounded like the test
and sec. people were not trying to speed it up. They seemed to offer little
help or alternatives. Perhaps the communication process broke down. Did Ed
know the importance of this change? Test and sec. certainly did not seem to.

~~~
roc
In the enterprise, it's _always_ an emergency. If it isn't, it gets ignored,
because the pipeline for getting 'non emergencies' done is constantly pre-
empted for emergencies.

That said, every good process has the ability to handle exceptions. If it was
truly the emergency it sounds like, he should have been able to make the
change to get it up and running ASAP and then made the change _properly_
through the normal process.

But the question of whether an "emergency" truly is, is a complicated one.

------
Dove
If you want a change request processed quickly, you should get someone in
upper management to help you herd it through the system. Literally, take your
VIP with you, and walk up to the desk of each admin in turn. Say, "Don't wait
for Tuesday's meeting--approve this right now." And then walk to the next
admin and do it again. And then walk to the test guys and say, "Drop
everything and test this now." And then gather up the six essential
stakeholders for the CCB meeting and say, "Impromptu CCB meeting, right now in
the hallway instead of next month. Approve this."

I've gotten software changes that usually take two months to trickle through
the system through in a day. You have to spend all day doing it, though, and
so does your VIP.

But if it's really _that_ important . . .

~~~
ojbyrne
That works once. Then the next time you go to each desk, the person there has
already checked with their boss, and says "my boss says you need to go through
the correct channels and if you don't like it, you need to talk to him."

~~~
Dove
That's why you have a VIP with you. If he's not her boss's boss, the whole
thing is a lot less likely to work.

------
ww520
People don't realize processes, reviews, standards, and audits are there
because of painful (costly) incidents and breakdowns in the past. Changing one
line of code in the production app is fast and easy. Crashing the system or
screwing up the app logic can cause huge disruption to the operation.

After every single crisis, managers got together and vowed to never let this
happen again, and thus processes and reviews are born. After couple years,
things will get so stifled to the point of what OP was experiencing.

~~~
benjaminwootton
I liked this post on the topic -
[http://37signals.com/svn/archives2/dont_scar_on_the_first_cu...](http://37signals.com/svn/archives2/dont_scar_on_the_first_cut.php)

~~~
vacri
" _embed the learning in the organizational memory as a story instead of a
policy_ "

That memory is hard to maintain with procedural changes in modern times, as
employees move around quite frequently.

------
Dove
That "legacy code must be brought up to current standards whenever it's
touched" policy doesn't seem very wise. I'm all for practical refactoring, but
it's a major decision to be treated with care when it comes to legacy code.
You don't always want to invite it. I certainly wouldn't want to enforce it.

I mean, it's one thing when your oldest legacy code is two years old, and it's
all one program. It's something else entirely when it's, for example, written
in Fortran and running on hardware you can't buy anymore. Sometimes you don't
want to rock the boat any more than you absolutely have to.

~~~
protomyth
Yes, for an emergency change (change or there will be layoffs), this is a
rather dangerous policy. The goal of any emergency fix should be "do as little
as possible to solve problem".

I get the feeling that too many people in the chain did not treat this as an
emergency. It seems very business as usual. When a company is talking layoffs,
it is an emergency.

~~~
Dove
Yeah, I agree. The first time he told someone "Philip wants this done quickly"
and they said, "Sorry, it's policy," the next move -- at least my next move --
would have been to walk to Philip's office and say, "Will you come tell this
lady that this change is more important than that policy?"

I mean, they don't have a way to know it's _really_ an emergency. Every
engineer thinks everything's an emergency. There's nothing like the guy in
charge standing there while they work to communicate THIS IS ACTUALLY AN
EMERGENCY.

------
gacba
And that is how creativity dies. Not all at once with a swift, decisive blow.
More like the bites of 10,000 mosquitoes coming 1 at a time, about every 5
minutes so you don't notice that they've stolen a significant amount of your
blood until it's too late.

~~~
brazzy
No, this is how creativity is shut out after one too many incidents where she
wandered in, decided to be helpful and, due to imperfect understanding how
things work, caused massive loss of productivity, money and reputations.

~~~
CamperBob2
No, creativity is a process to be leveraged and managed like any other. This
story is an account of life at a company where the process simply does not
support the needs of the business. There are no excuses.

------
yason
Excess code review rules and company policies are, in effect, a defense
against the stupid. They're in place to limit or slow down the damage
potentially done by stupid employees.

However, they're also a defense against the smart.

If there's a barrier erected to weed out stupid changes the barrier works to
weed out smart changes as well. The barrier can't distinguish smart changes
from stupid changes, and the people operating the barrier can't easily do that
either since their asses are on the line in case they miss a bad change. As
you can see, even the word of a higher manager can't change that since the
defenses are spread out so wide: there's always some additional policy that
requires more and more authorization from more and more people.

Good systems always support skunkworks approaches where good programmers have
an unofficial half-sanctioned way to subvert the policies when needed.

------
gaius
Ironically, work to prevent layoffs neatly identified all the people who
should be laid off immediately.

------
badclient
This shit happens at my job and leaves me feeling depressed and at times
questioning my own skills because _I_ know and my bosses know that on my part,
I just had to make a tiny, tiny change. But when you have a dozen such
experiences, I feel like something is wrong with me because when the dust
settles, this was a one line code change that took me a week to push to
production. It also makes my bosses question my estimates. Of course, when I
goto the other extreme and give a ridiculous estimate, that also pisses them
off and makes them feel like I don't really know what I am talking about.
Saying "I cannot provide an estimate" also does not cut it for the boss.

In the longterm, I can feel this killing a part of me. I have always been a
start-up guy working for myself. Since getting my first full-time job, stuff
that I would expect the start-up me to take an hour to do ends up taking days
and weeks, even. I am aware of the (mostly legitimate) excuses to not take too
much self-blame but at the end, it still feels shitty because I _know_ , the
start-up me knows, this should not have been a one week project.

Ah the perils of having a non-tech boss and coworkers and being the only tech
guy. Reading stories like I experience at work is comforting in an odd way.

Here's how my typical story goes...

Boss: So can we make this little analytics thing your #1 priority for
tomorrow?

Me: Yeah, as long as we don't have anything else for projectX(my #1 priority)

Boss: Well, do you?

Me: Not right now. But it can pop up any minute Jenny decides she needs
something done.

\--NEXT DAY--

Jenny: Hey this isn't working in ProjectX. Could you look into it?

Me: Sure. _there goes an hour_

Boss: I know you're busy but this is very urgent. Can we grab you for few mins
for this meeting?

Me: Sure _goes another 30 mins_

...

Jenny: That thing you fixed. Can we undo it and push to production because I
realized blabla?

Me: _thinking wtf?_ Ok sure.

...

Amanda: Hey can you check if these 2 users signed up from same IP?

Me: You can check that in admin

Amanda: Where?

Me: The admin panel you emailed me info to when I joined the company..

Amanda: Oh right. Can you fwd me the email I sent, I can't find it. My gmail's
not working right.

Me: Ok.

...

 _Before I know it, I have spent every day of my job troubleshooting shit to
never really get in the flow of doing my own project assigned by my boss._

Boss(after a month): How's that project coming? I know you said it's a one day
thing.

Me: Yeah but the next day we changed the specs in the meeting. Also, projectX
is getting a huge rehaul.

Boss: But still...you said it's a week after that

Me: Yeah assuming I didn't have work for projectX

Boss: But you'll always have work for projectX

Me: Yeah that's exactly what I told you too

Boss: So where does this leave us? Can you finish it in next week? I don't
think there is much work for projectX

Me: But...

Boss: You're the startup guy. We want you to push us to release stuff quick.
When you give us deadlines and are so off, it makes us lose faith.

 _Leave work annoyed, put in a quiet weekend with no coworkers around and wrap
up said project_

~~~
ajross
This doesn't sound like a process problem at all, actually. Really it's the
reverse: your work environment is a mess, no one has the ability to do their
own job except to bug you to do things they need.

And that's a disaster too, but it would be _helped_ by adding process, not
hurt.

Also, dude: learn to say no. Learn to prioritize for yourself instead of
expecting your slag pile of a management structure to do it for you. i.e. ask:
"What things must I _get done_ to make sure I'm successful long term?" (which
is entirely different from "What is the highest priority thing I have been
asked to do?") and make sure you prioritize your work to get those things done
(which is an entirely different thing from "work on those things"). And if
your workplace environment really doesn't permit that, move on.

~~~
badclient
_And if your workplace environment really doesn't permit that, move on._

Doing just that :)

It's basically a startup with 10 of us where I am the only in house developer.

I've tried many things like asking folks to Skype me instead of walking over
to my desk or calling my name. I've gone to great lengths to get em to
understand why skyping to get my attention is different than other ways. None
of these guidelines really lasted more than ana few hours.

~~~
DannoHung
On this subject: Whenever someone chats me in the following manner, I want to
kill them. Straight up. Just fucking take an ice pick and jab it right through
their skull:

    
    
        Other Dude: Hi.
        Me: Hello

_A minute elapses_

    
    
        Other Dude: I'm having a problem
        Me: How can I help

_A minute elapses_

    
    
        Other Dude: ACTUAL PROBLEM THAT THEY NEEDED HELP WITH
        Me: Oh, here is your resolution which took all of 15 seconds to give to you.
    

WHAT THE FUCK MAN?! WHAT THE FLYING FUCKITY FUCK?! YOU JUST BROKE MY FLOW AND
STOLE 2 GOD DAMNED MINUTES! WHAT IS WRONG WITH YOU?! IT'S FUCKING IM, YOU'RE
NOT GOD DAMNED STANDING NEXT TO ME! AND WHY THE HELL DOES IT TAKE YOU A MINUTE
TO TYPE EACH THING ANYWAY?! DON'T YOU KNOW HOW TO TOUCH TYPE?!

I mean, CHRIST, if this happened once in a blue moon, fine, whatever, but it
seems like EVERY motherfucker uses IM the same way. EAT DICKS.

...

I might have went a little overboard on this.

~~~
redcircle
This is why I stopped using all chat clients.

~~~
dspillett
That is why I _insist_ on communication via chat clients when I'm busy. I'm
not immediately interrupted as I would be if someone walks up or shouts out as
I can ignore the little flash of the chat window on the task tray for a few
minutes, sometimes tens of minutes, while I bring my train of thought to a
more orderly stop.

If people start with "Hi, I have a problem" and don't state what the problem
is I just respond with "drop me the details and I'll have a quick think" and
get back to my work. Even if they respond immediately to that it might take me
ten or fifteen minutes to get look at their message again - that is their time
wasted, not mine.

Some problems require a more urgent response so I'll accept a walk-up or a
shout for those, and some problems are best illustrated by showing them than
describing them, I just have to trust people to know what they are and what is
less urgent. And of course a message will come in at a time when I am
eminently interruptible, in which case people get an immediate response
anyway.

I find it works except for people who insist on not following the process, but
they soon learn I'm far more helpful if they don't irritate me!

~~~
masklinn
> If people start with "Hi, I have a problem" and don't state what the problem
> is I just respond with "drop me the details and I'll have a quick think" and
> get back to my work.

If people start with "Hi, I have a problem" and don't state what the problem
is _I close the chat window_.

My work IM's status clearly states "if you have a question, ask it, don't ask
if you can ask", time wasters simply get ignored.

> Some problems require a more urgent response so I'll accept a walk-up or a
> shout for those

Absolutely.

~~~
dspillett
_> If people start with "Hi, I have a problem" and don't state what the
problem is I close the chat window._

In the past I've found that leads to people reporting "I asked him several
times over a few hours and got no response" which, if the person being
complained to does not know the situation, could look bad for me.

If I feel like being more terse, I just give a one-word response: "Details?".

~~~
masklinn
> In the past I've found that leads to people reporting "I asked him several
> times over a few hours and got no response" which, if the person being
> complained to does not know the situation, could look bad for me.

As I noted, my conditions are clearly spelled out as my IM status text, if
they can't bother following simple instructions I can't bother with them.

------
mangler
No, don't quit! Embrace! That's 14 hours (if I remember correctly) you could
have spent working on your own projects instead of being pissed off on HN. The
world is full of this, and it's well accepted and nobody questions it - make
the most of it.

~~~
zackzackzack
They wouldn't be his projects. They would be the company's.

------
boyter
I just went through the same thing. Except I was smart (or so I thought) and
had what I needed changed in a config file.

PHB decided that it still counted as a "code" change because someone needed to
log onto the box and change the config. Literally it was removing 3 characters
from it. The only reason I even had it in the config was to avoid it being a
code change to begin with. I argued back that in the time I would take to get
through the whole process I could write a screen to do the same thing.

PHB said that would be fine as from that point on its a config not a code
change. So off through the whole process, where I present the change to people
who don't care, or understand the change and ask for them to approve it. I get
that they need awareness, but seriously couldn't a short email saying X is
going to happen at time Y suffice?

Im tempted to embed a Python/Ruby interpreter into my next project with a web
interface to modify the code. Then I can do these minor changes without going
through the 7 day process.

~~~
brazzy
the difference between config and code is largely illusionary, so PHB is
right: both can break things and need to be tested.

~~~
boyter
I would argue that in this case where it was designed like that from the
beginning IE detailed as how the system works then there should be no issues.

I did forget to mention it did go though a round of testing, which took less
then 5 minutes to be verified as correct. It literally was 6 days of process
wrapped around 10 minutes work.

------
iirvine
'Fuck that shit.'

maintenance programmer words to live by.

------
zackzackzack
Why is the author still working there?

------
peacemaker
It's exactly this situation that made me leave a large company after only 5
weeks. I'd come from a smaller, more agile environment and having to go
through multiple layers of process and bullshit just to get the simplest thing
done was just too much for me. I know 100% that if I'd have stayed there I
would have ended up hating my job and affecting my passion for writing code.

The main situation I remember is a code review of a few lines I'd written kept
on being rejected by some of the more process-indoctrinated programmers purely
because I didn't use the correct type of verb in the first sentence describing
my change! I mean, when _that_ is the shit that matters and not solutions to
problems, I'm out.

------
bonaldi
"It Takes 6 Days to Change A Fundamental Business Practice" sounds a lot more
reasonable.

 _Should_ you be able to extend the backlog of a factory by an entire month
just by changing a variable, pushing a build and going for lunch? No.

~~~
mkopinsky
The problem, imo, is not that it took 6 days of reviews to change a major
variable. The problem is that the reviews were entirely immaterial to the
actual change taking place. Whether variables are hardcoded, have an audit
trail, and use proper naming conventions will have no impact on ACTUALLY
mitigating the business risk of making a giant change.

------
parasubvert
(rewriting this a bit to make it clearer).

It's clear that a mission critical change needs oversight and risk mitigation.
But think of the waste of this process from the Lean perspective: Waiting,
Handoffs, Checking, Inspecting, Obtaining approvals, Reviewing, Filing, and
Rework.

IT groups need shepherds to be able to guide these things through the system
of checks and balances. If you look at this scenario, Ed, the programmer, was
the orchestrator. The end-to-end process was ad hoc, and not designed
purposely. There were actually several processes at play, designed by
different departments (IT demand, support, delivery, QA, etc.), with different
value systems, and the interaction between them isn't well defined towards the
end value of "shipping software changes that work". The division of labour
across review, testing, etc., without incentivizing end customer results, and
minimizing waste, has devolved into a bunch of school marms that redline all
documents or code given to them without actually working to help with the end
goal.

The "this test plan isn't good enough" for example is a near and dear to me.
Why can't QA dedicate a resource for a brief period to help make it better? Or
at the very least, give guidance on the what they want to see for approval?
Usually this (and ornery change management boards) wind up being the primary
source of senior management overrides - the QA group doesn't improve quality,
it just slows down change thus maintaining the current (functional)
mediocrity.

IF the process was shepherded by a manager to actively involve QA, Code
Review, Change Management, etc. earlier, Ed might be less frustrated, and it
would have been done in 3 days. ;)

------
serverascode
This happens waaaaaaay more than anyone will admit.

~~~
epochwolf
Is it sad that I think 6 days is pretty fast?

~~~
protomyth
I've worked places where 6 days would be a miracle. For me, the worst has
always been when the database query plan suddenly switched to something with a
full table scan (I'm looking at you Sybase) or the always fun full sort merge
(nice going Ingres). It always seemed to come down to forcing indexes which
tended to get deployed fairly quickly.

------
CamperBob2
Look on the bright side: if large companies weren't this screwed up, small
businesses and startups wouldn't have a chance.

------
protomyth
I wonder if the place is covered by something like ISO 9000?

------
hcarvalhoalves
The _real_ WTF is that at the end, all it takes is the president giving a
direct order to put the code in production immediately. That is, all the
policies and processes just add noise to the interface between strategic
decisions and operational tasks. That's how big companies fail.

------
cnbeuiwx
This happens at my job too and I want to kill myself when it does. I get so
frustrated. All these work protocols and slow useless software that prevents
me from being effective at work.

I usually find a way to make things work by going outside the protocols and
just fixing the problems. Its not the problem that is hard, its the stupid
damn protocols and official ways of doing things that makes it very hard to
get something done.

Decisions being made by idiots who have no clue how to allow people to work in
a efficient manner, or how much money they lose by people sitting around doing
nothing, waiting for some useless step in a useless process.

Seriously, I can feel the rage coming just thinking about it while writing
this! What the fuck happened to this industry...

------
fierarul
Heh, not the same corporate snafu, but my most expensive work was done to
change 1 char.

Spent about 3 weeks with a new codebase to find that one number had to be
increased. Funny thing is, there was a comment there and somebody had
increased it before when they had a similar problem.

~~~
rmc
Time to change the character: 10 seconds

Time to find out what charater to change: 3 weeks

~~~
cpeterso
I've found that bugs that are very hard to debug often have the smallest
fixes. Once you've discovered what tidbit of information you were missing, the
fix is trivial. I guess this is a needle/haystack problem.

------
chris_wot
Better that than they push the line of code and everything stops working for a
fortnight!

------
cluda01
What I identified with the most is the requirements creep that happens in
QA/validation

------
brudgers
This is a case where programmers are overhead not production and the "system"
is sized to handle peak loads.

Another way of looking at it, "It Takes 6 Days to Prevent Layoffs."

That's pretty damn efficient from a business standpoint.

And pretty damn important from a human one.

------
S_A_P
While I can both relate and empathize with the story here- the initial
"crisis" seems forced and arbitrary. Once a company becomes sufficiently large
to mandate a promotion process like the one described in this story, the
management would probably know that there is no "immediate" changes that can
be made unless they want to bypass those processes. Further, if it took 6 days
to change a line of code to meet everyone's standards and satisfy
requirements, imagine how long it would take to push through a layoff. :)

------
tcskeptic
I understand the advice to quit because of the agonizing process involved, but
worse than that for me is the idea that this parameter is hard coded, and even
with this change in place that limitation still remains. NO WONDER the change
process is so painful, the software design is clearly brain dead. I bet all
kinds of unexpected shit happens when changes are rolled out. This is a code
smell like a rotting corpse.

~~~
elemeno
Moving parameters somewhere else doesn't change the problem at all, only moves
it to a different file. Altering a parameter that effects a large chunk of the
system still needs the exact same amount of testing to be done, all you save
is a compile.

For what it's worth. I'd also say that anyone who thinks that hardcoded
parameters are always bad, probably hasn't worked on anything large or
complicated enough yet. There are plenty of cases where you've got parameters
which could theoretically change, but the likelihood is small enough that it
doesn't make much sense to move it out to somewhere else - not coincidently,
these tend also to be parameters which effect large chunks of the system and
should never be changed without a solid business reason and a full QA cycle.

------
twelvechairs
As good a 'system' as you might have, it is still no match for having a middle
manager with a clue (and some freedom to move within the system)...

Sure - this should have still been tested (and thoroughly!), but putting this
'on the queue' when it needs doing now? Making the programmer deal with other
issues (rename the variable) while doing it? Not making sure that a testing
machine was available? No....

------
jiggy2011
I have to opposite problem, almost no process at all.

I'm not sure which is worse, having to wait 6 days to do a change or having
people request (often trivial) changes to be deployed the same day they
specified them and having people going round editing .php files directly on
the production server.

------
ovechtrick
THANK YOU.

This couldn't have come at a better time. Describes my current situation
perfectly.

I changed 1 line of code today. It's going to require me to call into 2 review
meetings to have it approved. These meetings happen once a week. 2 weeks to
get this code to production. UNBELIEVABLE.

I think it's time for a new job.

------
EternalFury
I can relate to that. Maybe 70% of the defect corrections I make in our highly
complex and optimized code base are less than 10 lines worth of code.

It can take between between 24 hours and a week to figure out what those few
lines should be, though. And that is extremely frustrating.

------
chaostheory
> Pissed off hours spent on Hacker News: 14.

Seriously I've already mentioned this for your last post, but imo it's really
time to leave your job for something else Ed. Unnecessary stress lessens the
quality of your life, as well as lowering its total lifespan.

------
deepGem
Ditto for most of the IT services companies in Bangalore. Oh just add "Project
Manager X chasing the required parties" between every line in that article and
perhaps add another 3-4 day delay as most of the teams don't work in the same
time zone.

------
Sukotto
I thought the last line would read "Weeks of notice given: 2"

Why are you putting up with that crap Ed?

------
rubynerd
This is the model which my Computing coursework bases around, you get
something like 6 marks for code and 8 marks total for a "test plan"

The best thing? I think I've lost the majority of marks for using RSpec
instead of 100 screenshots

------
tluyben2
This happens at both big and small companies; I worked with a company of 30
people who had this 'in place' already because the CTO was had 'experience in
leading very large projects'. It was kind of annoying.

------
hedgie
I'm very lucky to work in a place where they understand the limitations of
this policy.

Most of the product we work on (not a CRUD app) is finished or at least at a
fairly mature state. But the area I work on is not. For a while they harped on
us about following the process. But due to their lack of interest on what we
were doing and some managers protecting us, we ended up able to make rapid
changes in response to test failures, often several times a day.

It was incredible. They would seek investigations through change requests on
bugs found in testing. Immediately after they were entered - before the change
request even went to the overseeing group of project engineers and subject
matter experts - we would tell them what caused the problem and write that we
had committed a fix to the mainline. This circumvented days of waiting for a
response that would have blown our schedule apart.

It paid off big. Today for the first time our tests all started passing.
They've been failures for years and no one expected anything close.

There's a time for process. When a lot can break or you have a
finished/safety-critical product you want to lock it down. But in initial
development of new features, you have to throw it out and do what's right.

For a while they were demanding change control with CRs over software that had
never been written or run successfully. Why even bother writing the CR if it
never worked in the first place? That means the software wasn't done in the
first place, not in a state requiring change control!

It was unusually rapid iteration between the test and development teams, who
should sit next to each other and immerse themselves in making the software
work. Only by rapidly uncovering and overturning problems like this can you
ever get something close enough to stable to lock it down this way.

Development and test respected each other, understood what had to be done, and
by working together discussing behavior and the test environment were able to
determine what the software should do and how best to test this functionality.

There were times where waiting for the process would have taken us 5 days to
get approval to fix a bug. By talking with our project engineer and going
ahead with the testers - who realized it was not the time to stick to formal
process - we were able to succeed.

I'm very lucky. The testers were cool and understood the state of the software
and were willing to shout across the cube and explain what was failing. We
worked together and managed to succeed by knowing when a process was important
and when the managers would be happier to go around it.

The problems encountered here are mainly about people. We had testers and
others who trusted and worked with us. When you don't have that - like the
author of the article - it takes some social engineers and lawyer like
knowledge and parsing of company policy to get around the problems.

------
avelis
I find it troubling that they had to make a change at the point of layoffs to
create a solution to prevent it. Couldn't they have discovered this trend
before it got to that point?

------
389401a
This is how individuals working with open source software beat a large
corporation where by all accounts they should not stand a chance. Bureaucracy
is the Achilles Heel.

------
chrismealy
Is this a story or a real thing that happened?

------
rpikeca
This is an unfortunate reality. People really do not care about all the work
you have just the things they want done.

------
geekus_maximus
I smell opportunity. Companies that are as archaic and inflexible as this
deserve to be outdone.

------
Karunamon
This reads eerily like my job, just s/code change/system change request

Love the my job, hate the bureaucracy.

------
nnethercote
"you are responsible for fixing preexisting errors that violate new company
policy"

That's insane.

~~~
zszugyi
Seems like the "boy scout rule" got out of control.

------
sp332
Why aren't you appealing directly to Philip more often? Give him a list of
what you need and he could have what he wants with 3 emails. Also I don't
understand what "access" you need to Marge? You can't email her or something?

edit: d'oh

~~~
jacobr
Homer and Marge are most likely some kind of testing systems.

------
nerdfiles
"Also, dude: learn to say no." This is horrifyingly bad advice, and in fact I
doubt it even acknowledges the problem. It takes time to say "no". Are you
telling us to learn "the path of least resistance" in saying "no"? Is this
legitimate advice?

At a previous development gig, I would have to learn how to say "no" to at
least 5 different types of people. And quite honestly, and candidly, I think
this advice is absolute bullshit. I genuinely believe it wholesale ignores all
the junk-news that's bubbled up around Zuckerberg's attire at meetings, the
joke about Ohanian's suit on Bloomberg TV, etc., the culture of corporatism.
If you say "no" enough, you end up like Zuckerberg: An Emperor with no Suit.
(Or to put it less abstrusely: "This guy won't play ball.") And if you say it
with the force and cavalier quality I think you're thinking: you get fired.

This advice prompts me as if a manager, if it's taken without the gusto of the
frustration we're all feeling. And I don't want to be a manager.

Let me put it this way. Are you saying there's a verbal silverbullet that
shoots down CEOs, Team Leads, Content Authors, Co-Developers, Designers, IAs,
Managers, Testers, Team Members, etc.?

I suppose many others have already said this: _easier said than done_.

------
drivebyacct2
This thread reminds me of two lessons that I'm thankful I've learned young:

1\. Processes exist for a reason. They're often annoying and sometimes
overkill and unnecessary, but they're often there as a result of a past
problem.

2\. The whole top thread about communicating and literally ignoring people
reminds me of my roommate who bragged that he was a good leader, meanwhile
telling me that he would simply ignore his teammates because they annoy him.
This is why "geeks" get the stereotype that they can't interact with people
and it's why programmers with business and communication skills are so
valuable and sought after.

------
mjwalshe
6 days you LUCKY LUCKY Bastard - i have user stories over 6 months old that
haven't been actioned yet and had to wait 2 months to get a gwt verification
file installed

------
faucet
is it only me or there is a (just one) line of code gone wrong on ed's comment
handling server?

{"post_id":131434793,"html":"\u003Cdiv class='p_response_container'
style='display:none;'\u003E\n\u003Cheader class='clearfix'\u003E\n\u003Ca
href=\"[http://edweissman.com/it-takes-6-days-to-change-1-line-of-
co...](http://edweissman.com/it-takes-6-days-to-change-1-line-of-code\\)
class=\"p_view_all_link\"\u003EView All 16\u003C/a\u003E\n\u003Ch1\u003EMost
Recent Responses\u003C/h1\u003E\n\u003C/header\u003E\n\u003Cdiv
class='p_responses_list clearfix' data-posterous-responses-
container\u003E\n\u003Carticle class='p_comment p_response'
style='display:none;'\u003E\n\u003Cdiv class='p_info'\u003E\n\u003Cspan
class='p_icon'\u003E\u003C/span\u003E\n\u003Ctime pubdate='1337295601'\u003E14
minutes ago\u003C/time\u003E\n\u003Ca href=\"<http://tomakefast.com\\u003EPJ>
Brunet\u003C/a\u003E responded:\n\u003C/div\u003E\n\u003Cdiv
class='p_comment_body'\u003E\n\u003Cdiv
class='p_profile_photo'\u003E\n\u003Cimg alt=\"\"
src=\"/images/profile/missing-user-35.png?1337287723\"
/\u003E\n\u003C/div\u003E\n\u003Cdiv class='p_comment_text'\u003E\nHilarious,
I love it.\n\u003C/div\u003E\n\u003C/div\u003E\n\u003Cul
class='p_action_links'\u003E\n\u003C/ul\u003E\n\n\u003C/article\u003E\n\n\u003Carticle
class='p_comment p_response' style='display:none;'\u003E\n\u003Cdiv
class='p_info'\u003E\n\u003Cspan
class='p_icon'\u003E\u003C/span\u003E\n\u003Ctime pubdate='1337296138'\u003E5
minutes ago\u003C/time\u003E\nkkudi responded:\n\u003C/div\u003E\n\u003Cdiv
class='p_comment_body'\u003E\n\u003Cdiv
class='p_profile_photo'\u003E\n\u003Cimg alt=\"\"
src=\"/images/profile/missing-user-35.png?1337287723\"
/\u003E\n\u003C/div\u003E\n\u003Cdiv class='p_comment_text'\u003E\nI agree and
disagree at the same time. It does take a ridiculous amount of time to put
something in production, unfortunately, but the scenario you described above
is probably an exceptional case.\n\u003C/div\u003E\n\u003C/div\u003E\n\u003Cul
class='p_action_links'\u003E\n\u003C/ul\u003E\n\n\u003C/article\u003E\n\n\u003Carticle
class='p_comment p_response' style='display:none;'\u003E\n\u003Cdiv
class='p_info'\u003E\n\u003Cspan
class='p_icon'\u003E\u003C/span\u003E\n\u003Ctime pubdate='1337296154'\u003E5
minutes ago\u003C/time\u003E\nbob responded:\n\u003C/div\u003E\n\u003Cdiv
class='p_comment_body'\u003E\n\u003Cdiv
class='p_profile_photo'\u003E\n\u003Cimg alt=\"\"
src=\"/images/profile/missing-user-35.png?1337287723\"
/\u003E\n\u003C/div\u003E\n\u003Cdiv class='p_comment_text'\u003E\nDoesn't
inventory cost $ and space? Maybe CFO should be involved
too.\n\u003C/div\u003E\n\u003C/div\u003E\n\u003Cul
class='p_action_links'\u003E\n\u003C/ul\u003E\n\n\u003C/article\u003E\n\n\u003C/div\u003E\n\u003Cdiv
class='p_comment_form p_logged_in'\u003E\n\u003Ch1\u003ELeave a
Comment\u003C/h1\u003E\n\u003Cform action=\"/responses/create\"
class=\"new_comment clearfix\" id=\"new_comment\"
method=\"post\"\u003E\u003Cdiv
style=\"margin:0;padding:0;display:inline\"\u003E\u003Cinput
name=\"authenticity_token\" type=\"hidden\"
value=\"F6F7vP+9I0xtnokVneiKN6klXpPRM8ziSWl/D/arZxI=\"
/\u003E\u003C/div\u003E\n\u003Cdiv class='p_not_authorized
p_comment_section'\u003E\n\u003Cdiv
class='p_additional_fields'\u003E\n\u003Clabel
for=\"comment_name\"\u003EName:\u003C/label\u003E\n\u003Cinput
id=\"comment_name\" maxlength=\"80\" name=\"comment[name]\" size=\"40\"
type=\"text\" /\u003E\n\u003Clabel class=\"p_comment_email\"
for=\"comment_comment_email\"\u003ELeave this field blank to
comment.\u003C/label\u003E\n\u003Cinput class=\"p_comment_email\"
id=\"comment_comment_email\" maxlength=\"80\" name=\"comment[comment_email]\"
size=\"40\" type=\"text\" /\u003E\n\u003Clabel
for=\"comment_toast\"\u003EEmail:\u003C/label\u003E\n\u003Cinput
id=\"comment_toast\" maxlength=\"80\" name=\"comment[toast]\" size=\"40\"
type=\"text\" /\u003E\n\u003Clabel
for=\"comment_url\"\u003EHomepage:\u003C/label\u003E\n\u003Cinput
id=\"comment_url\" maxlength=\"80\" name=\"comment[url]\" size=\"40\"
type=\"text\" /\u003E\n\u003C/div\u003E\n\u003Caside
class='p_login_options'\u003E\n\u003Cdiv
class='p_asterisk'\u003E\u003C/div\u003E\n\u003Ch1\u003EWant to skip this
stuff?\u003C/h1\u003E\n\u003Cp\u003ELogin with any of the
following:\u003C/p\u003E\n\u003Cdiv class='p_login_buttons'\u003E\n\u003Ca
href=\"[http://posterous.com/login?flow=newcomment\u0026amp;jumpto=h...](http://posterous.com/login?flow=newcomment\\u0026amp;jumpto=http%3A%2F%2Fedweissman.com%2Fit-
takes-6-days-to-change-1-line-of-code%23comment\\)
class=\"p_posterous_login\"\u003ERegister or login to
Posterous\u003C/a\u003E\n\u003Ca href=\"#\" class=\"p_twitter_login\" data-
posterous-jumpto-url=\"[http://edweissman.com/it-takes-6-days-to-
change-1-line-of-co...](http://edweissman.com/it-takes-6-days-to-
change-1-line-of-code#comment\\) data-posterous-post-id=\"131434793\" data-
posterous-redirect-
url=\"[http://posterous.com/oauth/init_oauth_and_redirect/?ssod=edw...](http://posterous.com/oauth/init_oauth_and_redirect/?ssod=edweissman.com\\u0026amp;oauth_provider_type=\\)
data-posterous-twitter-login-button=\"true\"\u003E\u003Cimg alt=\"Sign in with
Twitter\" src=\"/images/site/sign_in_with_twitter.png?1337287723\"
/\u003E\u003C/a\u003E\n\u003C/div\u003E\n\n\u003C/aside\u003E\n\u003C/div\u003E\n\u003Cdiv
class='p_comment_section'\u003E\n\u003Clabel
for=\"comment_body\"\u003EComment:\u003C/label\u003E\n\u003Cdiv
class='p_comment_area'\u003E\n\u003Ctextarea cols=\"40\" id=\"comment_body\"
name=\"comment[body]\" onChange=\"if (this.scrollHeight \u0026gt;
this.clientHeight \u0026amp;\u0026amp; !window.opera){this.rows += 1;}\"
onKeyPress=\"if (this.scrollHeight \u0026gt; this.clientHeight
\u0026amp;\u0026amp; !window.opera){this.rows += 1;}\"
rows=\"5\"\u003E\u003C/textarea\u003E\n\u003C/div\u003E\n\u003C/div\u003E\n\u003Cdiv
class='p_comment_section'\u003E\n\u003Cinput id=\"comment_post_id\"
name=\"comment[post_id]\" type=\"hidden\" value=\"131434793\"
/\u003E\n\u003Cdiv class='p_submit'\u003E\u003Cbutton button=\"true\" class=\"
action_button\"\u003E\u003Cspan class=\"\"\u003EPost this
Comment\u003C/span\u003E\u003C/button\u003E\u003C/div\u003E\n\u003C/div\u003E\n\u003C/form\u003E\n\u003C/div\u003E\n\n\u003C/div\u003E\n","responses":[{"name":"PJ
Brunet","body_html":"Hilarious, I love it.","created_at":"2012/05/17 16:00:01
-0700","body":"Hilarious, I love
it.","post_id":131434793,"id":11028186,"display_name":"PJ
Brunet","comment_type":"comment","display_url":"[http://tomakefast.com,allowed:true,display_photo:/images/pro...](http://tomakefast.com,allowed:true,display_photo:/images/profile/missing-
user-35.png)},{"name":"kkudi","body_html":"I agree and disagree at the same
time. It does take a ridiculous amount of time to put something in production,
unfortunately, but the scenario you described above is probably an exceptional
case.","created_at":"2012/05/17 16:08:58 -0700","body":"I agree and disagree
at the same time. It does take a ridiculous amount of time to put something in
production, unfortunately, but the scenario you described above is probably an
exceptional
case.","post_id":131434793,"id":11028197,"display_name":"kkudi","comment_type":"comment","display_url":null,"allowed":true,"display_photo":"/images/profile/missing-
user-35.png"},{"name":"bob","body_html":"Doesn't inventory cost $ and space?
Maybe CFO should be involved too.","created_at":"2012/05/17 16:09:14
-0700","body":"Doesn't inventory cost $ and space? Maybe CFO should be
involved
too.","post_id":131434793,"id":11028199,"display_name":"bob","comment_type":"comment","display_url":null,"allowed":true,"display_photo":"/images/profile/missing-
user-35.png"}]}

~~~
vgnet
Hey, thanks for wrecking page display for me! :)

~~~
faucet
it was just one line! :-)

------
munkydung
What year did this occur? Do you get a lot of paper cuts from feeding the
punch card reader?

------
reinhardt
Fuck. That. Shit. Seriously, I'm not exactly thrilled in my current startup
role but stories like this give me a new appreciation for it.

