
My Response to “Are we overcomplicating software development?” - feyn
https://neilonsoftware.com/2017/01/18/my-response-to-are-we-over-complicating-software-development/
======
agentultra
"Process," is the answer to the question, "how do we get a team of talented
engineers to design and implement software together?"

The answer to what should that process be? I'd look to the SWEBOK[0] for an
introduction to the state of the art.

As for the question at stake here... let me use the words of someone much
smarter than myself:

> _Simplicity is a great virtue but it requires hard work to achieve it and
> education to appreciate it. And to make matters worse: complexity sells
> better._

> _Dijkstra (1984) On the nature of Computing Science (EWD896)_

If you don't appreciate the true difficulty of programming you're bound to
create problems and spend most of the rest of your time debugging them later
on. If your chief motivation is to sell a software product or service powered
by software then it behooves you to create complexity. It seems more
impressive when you can't explain why some component or feature doesn't work
in a single sentence. You just want to control that complexity enough so that
you can maintain the illusion of order.

If you want to achieve simplicity you have to work at it. It's hard and
doesn't come for free. You really have to think. There's no way around that.

[0]
[https://www.computer.org/web/swebok/index;jsessionid=306e197...](https://www.computer.org/web/swebok/index;jsessionid=306e197849715725b82e977c6b54)

~~~
feyn
(I'm Neil, the author)

I can't find a "like" button anywhere, so I'll just say "Like".

------
mindcrime
(in regards to the original post that started this)

I could rant for hours about how so many misuse the term "Agile" and the
misunderstanding the idea(s) behind being agile. But I'm at a point where I
almost don't care anymore. The use of "Agile" as though it describes a
specific, prescriptive methodology is so ubiquitous that it's almost
impossible to talk about the subject.

Let me just say this... go read the Agile Manifesto before issuing any
criticism of "Agile" and repeat this 10 times - "Agile is NOT a methodology".

Scrum is a methodology. Crystal is a methodology. RUP is a methodology. XP is
a methodology. OpenUP is a methodology. TSP is a methodology. One or more of
those methodologies may be "agile", but "Agile" itself is NOT a methodology.

~~~
ktRolster
Note that Agile values "people over process," so having too heavy a focus on
process violates the Agile manifesto by itself.

~~~
bluGill
You conclusion is wrong. Agile finds value in process, as well. Read the last
line until this sinks in:

    
    
        That is, while there is value in the items on

the right, we value the items on the left more.

There is a lot of value in the right process, the process however needs to be
a servant to the needs of the people. You can be agile with long complex
processes so long as you pay attention to how the processes affect the people
and ensure that every time a process is getting in the way of someone it is a
good thing.

I think we would all agree that process needs to stop the "cowboy coder" who
otherwise would check in code with many syntax errors just before leaving for
a long vacation. This example is nearly a straw-man it is so trivial - but if
anyone doesn't have personal processes to prevent it the team will need to
create a process of some sort. This could be a simple don't do that again
talk, a HR process to fire the guy, a automatic backout process, a formal
checkout for pushing, or whatever - the choice is up to your team, and depends
on team factors like if this is a one time mistake or a repeated offense.

Going to the other extreme, medical software that could kill should have long
complex processes. Agile is perfectly fine with this.

Agile is taking issue with some [big companies] that have created such long
cumbersome processes in ages past that no longer serve the needs. Sometimes
they created a complex process because of a one time mistake that is unlikely
to happen again. Or maybe the process made sense in light of the tax laws that
were changed in 1986. Some processes actually are required and useful, but
because of all the cumbersome ones they are ignored until it is too late.

~~~
ktRolster
I don't know if you realized this, but all your examples are of how process
controls people. Can you even explain what it means to make people more
important than processes?

~~~
bluGill
Process is about controlling people, so of course my examples are controlling
people. The trick is they are process controlling people in areas where they
need to be controlled. Contrast this to processes that control people in ways
they don't need to be controlled.

Is someone checking that are login to your computer at exactly 8am every day?
That a process that controls people but isn't needed for most programmers. The
process makes perfect sense in context of assembly line workers: if even one
person is late the entire line cannot start.

One thing that I didn't say but should have is that process should not be the
first thing to fall back on. You might have to deal with the guy who pushes
broken code all the time, but there are other solutions other than process
that you should explore.

~~~
ktRolster
Most of your post is still about processes, justifying why they are good. I
don't think you get what it means, still: "people over process."

~~~
bluGill
looking back, I accept your criticism. I'm an engineer, ultimately I
understand things like processes better than people.

I stand by agile saying people over process, but I have no clue how to
actually explain it.

------
Analemma_
The point about "Agile the philosophy" being better than "Agile the practice"
seems like a red herring to me, since so few get "Agile the philosophy" right.
It's the same way I feel about things like HATEOAS: maybe it's good as a
Platonic ideal, but if no one is reaching that ideal, then how useful is it
really? There need to be suggestions that we can _actually_ put into practice.

~~~
zpallin
Exactly my problem with Agile. The philosophy is immense and hard to follow.
Everyone sees it differently. The practice of it, thus, is always unique, and
never consistent. We need something better.

~~~
carsongross
Agile is never wrong: it is only the humans that have failed it.

More consulting, of course, is required.

~~~
dmichulke
That second line is the only thing that distinguishes Agile from communism ;)

------
nroach
Why does this need its own thread? (Other than ego-stroking the OP) Seems like
any response could be in the main thread or linked from there rather than
splitting the discussion.

~~~
jMyles
Are we running out of threads? :-)

Threads are cheap and this came to my attention more strongly this way than if
it had just been a comment.

Every piece of prose posted here is, in some way, a response to something.

I'm not sure it's "ego-stroking," strictly speaking. What's wrong with giving
the author recognition?

~~~
feyn
(me = author) FWIW, I can totally understand how it could come across as ego
stroking. In this particular case I wasn't trying to, but I have been guilty
of it in my life. As for recognition, trust me - give it a week and no one
will remember my article ever existed. I honestly just wanted to answer ian0's
question thoughtfully, and made the best call I could think of.

------
chrisbolt
Response to this:
[https://news.ycombinator.com/item?id=13426896](https://news.ycombinator.com/item?id=13426896)

------
ChicagoDave
Cloud DevOps is complex, but we're just moving on-prem complexities and
hardware complexities to the cloud, so it's a fair trade.

Microservices, when done properly, are significantly less expensive than
traditional service layers.

Continuous Integration is one of the greatest improvements in software
development.

~~~
feyn
1) Yes, plus security concerns...that were really always there because clearly
all those enterprise firewalls were not stopping anyone. Just ask Yahoo.

2) "When done properly" \- therein lies the rub.

3) Absolutely agree, but I have seen teams screw it up time and time again.
Never really figured out why that is.

~~~
goalieca
I suppose #3 might fail because of the "frog in boiling water".

If you throw a frog in boiling water it will freak out. But if you throw a
frog in cold water and slowly bring it to a boil, it will never notice.

I actually like to submit completed modules when they are ready. I find
constant pushing makes things a lot more complicated.

edit: fix terrible spelling

~~~
feyn
Re: #3, perhaps so. I have seen many a good CI process go to hell over time. I
could argue no one would have a CI process if in the beginning it wasn't
making life easier. It would seem no CI process survives contact with the dev
team.

Re: Modules, generally I do as well, but what is a "module"? A function? A
class? A library? A plugin? An extension? A listener? A handler? A subscriber?
A publisher? A DAO? A DTO? An in-memory service? A networked service? A
microservice?

The philosophy that I subscribe to is continuous integration is not the same
as "everyone check into the same branch anytime they want as often as they
want" but also that it is not "everyone work in their own branch until time
has run out and we have to force all of our stuff to work together." That
middle ground can be very tough to find, but I find that if the CI system and
the architecture are not designed to work together, devs will always thrash
around between those two extremes.

------
sopooneo
I agree with a lot of this, but I think there is a single correct answer for
the best way to first learn about the "Agile Philosophy":
[http://agilemanifesto.org/](http://agilemanifesto.org/)

~~~
feyn
(I'm the author) Well...yes...and that's where (I hope) most people will
start. What happens next, I've found, is that people's own confirmation bias'
kick in, and they take the manifesto and make it describe (in their own mind)
what they are already doing and/or want to do.

The manifesto (to me) is how people who understood the nature and goals of
Agile summarized their sprawling, complex, interconnected - and valid -
thoughts. At the risk of being hyperbolic, I think of it like E = MC^2. Yes,
that is a good summary of Einstein's theories, but if all you ever know of
them is this summary you'll miss all the implications of what happens when you
put it into practice. Probably a terrible example, but the best I could think
of.

------
BerislavLopac
I upvoted it at "Microservices are a concept, not an implementation."

~~~
twic
I don't really understand what it means.

~~~
feyn
This is a good place to start:

[https://martinfowler.com/articles/microservices.html](https://martinfowler.com/articles/microservices.html)

The opening sentence sets the tone well for the article well:

"'Microservices' \- yet another new term on the crowded streets of software
architecture."

A colleague of mine once said that his work (he was a business strategist) was
"slowly coming into focus." We're pretty far along today in our understanding
of microservices, but at the time the article was written (early 2014)
microservices were "slowly coming into focus". Reading what people where
thinking when the industry was trying to define them should help in
understanding what they are today.

------
zpallin
I agree with ian0's perspective, but I am compelled by Neil. I think we can
learn a lot of patience about the state of software development from this
article.

~~~
feyn
I appreciate that - thank you.

------
bluetwo
Yes, Very much so. Could not agree with the author more.

~~~
feyn
Careful - I know that guy (I am that guy). He hardly knows what he's talking
about half the time, and for the other half he makes it up as he goes along.

Seriously though, glad you liked it.

------
gandutraveler
OP vs this response is great example of gaps in "actual usage" vs concepts

------
keithnz
I'm not sure this article is really saying too much at all. I started with
extreme programming back in 99 (after following the proponents and all the
heated discussions on the most brilliant OTUG email list)

Many many times I have seen this kind of question, "Aren't we just making this
too complicated?"

and the response being "It's not complicated, if you misunderstand X Y or Z
you are probably going wrong"

My observation is, many things of the things we leverage in the software world
are answers to problems. Some of those answers seem so good, and work
brilliantly when combined with other answers we package a set of answers as a
"process / methodology" and advocate it.

But sometimes for some people the answers don't really match the problem, or
need adapting to the problems at hand. Now, without good insight into the
original problems, then YES it will be "too complicated". Any time you do
things for problems you don't really have it is

A phrase that was thrown around in the early days of XP was "Brain Engaged",
to try and highlight that you don't blindly do things, you need to be aware
why you are doing something and adapt as necessary.

However! If someone says "hey this new new way of approaching design is
awesome", and it piques your interest, You _should_ blindly learn/try that
something new and debug it till you get an understanding of why it was
considered awesome. At that point you can brain engage it as needed.

So back to the original question, yes, we do make things too complicated and
we shouldn't do things that don't make sense. It may not make sense because we
have the knowledge and experience to know it doesn't, or it doesn't make sense
because we are too inexperienced. Both are valid reasons to stop and start
asking what is going wrong? why isn't this working? In the world of "Lean"
this is the idea of "Stop the line", sort things out, start moving forward
again, even if slowly at first.

All that doesn't mean you should keep things really basic without changing, it
is very important to know how to adopt ideas for doing things better and that
actually contribute to things we care about.

Also important to try not to become a cynic, eg, "tried microservices, they
suck, OO sucks, imperative sucks, VB sucks, NoSQl sucks". When we become
cynical of something it becomes very hard to be brain engaged in regards to
them. Also the opposite is true, becoming enamoured with something can limit
our brain engagement also like "Functional program all the things!, unit test
all the things!, scale all the things! deep learn all the things! distribute
all the things! Reactive all the things! things! the all Forth"

~~~
feyn
"I'm not sure this article is really saying too much at all" I believe to be
accurate, and I'm the author. I wasn't so much as making a statement or a
point as addressing the questions in the best best way I knew how.

If "The Dude" wrote a response to Ian0's original question, I would think it
would have a similar sentiment to mine, though far more concise: "Yeah
man...but, you know - no man."

~~~
jkmcf
Value is always found in the honest exchange of ideas, and I found the link to
SWEBOKv3. I'm happy for your post bringing it to my attention!

