
The Hard Thing About Software Development - bcl
https://www.linkedin.com/pulse/hard-thing-software-development-jesse-watson
======
haburka
I agree completely. Rarely the hard problems in coding are thinking of a
clever algorithm, or solving a particularly nasty scaling problem. Usually
I've struggled the most with understanding and building what the customer
needs. A spec can be interpreted many different ways and will always have to
be polished after its completed. Someone who doesn't know what the customer
expects will almost certainly create an implementation that has these micro
problems that make the product miss it's mark.

It makes me think that we're training the wrong people in college by making CS
a very difficult, math heavy field which often causes the more human skilled
people to drop out. Programming doesn't have to be anymore math heavy than
building a house yet we force undergrads to implement algorithms on paper? The
amount of wasted potential talent due to college is staggering.

~~~
sidlls
> It makes me think that we're training the wrong people in college by making
> CS a very difficult, math heavy field which often causes the more human
> skilled people to drop out.

I disagree. I think instead there is a category error being made: that CS is
an appropriate degree (on its own) to become a software engineer. It's like
suggesting a BS in Physics qualifies somebody to work as an engineer building
a satellite. It doesn't, but that doesn't mean "physics is too math heavy." In
fact, engineering a satellite requires almost as much basic mathematics
education as a BS in physics requires (some exceptions might include the
specialized mathematics required for upper-level theoretical physics concepts
that _may_ not apply at an engineering firm).

~~~
ramzyo
> I think instead there is a category error being made: that CS is an
> appropriate degree (on its own) to become a software engineer.

Completely agree here. I often find myself wondering why 'Software
Engineering' isn't the degree required to be a Software Engineer and further
doesn't really exist as a major, whereas 'Mechanical Engineering', 'Electrical
Engineering', 'Civil Engineering', 'Chemical Engineering', etc. are the
degrees associated with those professional titles. To your point, I don't
think it's a simple matter of nomenclature (i.e. that CS and Software
Engineering are synonymous). Not a CS major myself, but amongst my friends who
did their undergrad in the US I don't think they had any classes that covered
requirements gathering, putting together schedules, etc. Any CS majors here
have a class/classes that covered those topics?

~~~
naasking
> I often find myself wondering why 'Software Engineering' isn't the degree
> required to be a Software Engineer and further doesn't really exist as a
> major

Because we don't yet understand software engineering. There has been
insufficient empirical study of what yields maintainable/performant/what-have-
you code, of what sorts of abstractions are "good" for maintenance/reuse/etc.,
at what's needed to reliably predict a program's resource requirements, and
many software developers balk at design tools that restrict their style to
detect certain errors as early as possible, like type systems.

Software development is not yet an engineering discipline for all of these
reasons, and more.

~~~
marssaxman
We will never understand software engineering in that way, because we are in
the business of automating ourselves out of work. As soon as we understand
part of the process well enough that it _could_ become an engineering
discipline, we simply build some new tools and let the robots handle that part
of the job, moving the humans along to wherever today's frontier of
uncertainty happens to be. We will never be engineers in the traditional
sense, because that would be a waste of human brainpower.

~~~
naasking
You overestimate AI. Incompleteness is everywhere in CS. Overcoming these
limitations is not trivial at all.

Besides, software hasn't automated any other engineering discipline, and those
are much more straightforward because they're more mature and the principles
understood.

~~~
marssaxman
I'm not talking about AI. I am making the claim that software engineering will
never be "mature" in the sense we ascribe to other engineering disciplines,
precisely because we will never completely understand what we are doing; once
we do understand the principles involved, we build some new libraries or
languages or other tools which automate that part of the process, and we move
on to thinking about other things we don't understand yet.

------
phkahler
Favorite line "... the software development process is exploratory by nature."

I always say the customer/client does not have requirements, they have
problems. You will not discover all the requirements until you start solving
some of the problems and providing solutions. Only then will they say "oh
but...." and drop more requirements on you that they didn't think of up front.

Back to that quote. It's not that software development is exploratory in
itself. It's that the development is intertwined with an exploration of the
problem being solved.

~~~
mason55
> _and drop more requirements on you that they didn 't think of up front_

I think one of the important qualities of an architect is to anticipate what
these requirements are going to be and define solutions to them ahead of time.

I have this conversation all the time with our client-facing team.

 _Me_ : "What is supposed to happen if this data changes?"

 _Colleague_ : "Well the customer didn't give us a requirement for that so we
don't have to worry about it"

 _Me_ : Screams inside

------
xvaier
As a lead developer building a platform dealing with the intricacies of union
agreements and labor restrictions, this summarizes exactly the thought process
that my team has gone through in the last year.

We started with a simple problem that plagues HR departments in every
conceivable industry with unions, finding substitute personnel and erroneously
assumed that it was a simple fix. Over the past year and a half we have
accumulated a great deal of knowledge after interacting with as many people as
possible and have finally released a version that meets our original criteria
(and much more). It was obviously not a simple fix.

If I have one thing to tell anyone who is looking for business ideas to try
out their new programming skills on, I strongly suggest taking the time to
learn as much as possible about the people to whom you want to provide a
solution, then recruiting one of them to help you build it, lest you become
another project that solves a non-issue beautifully.

~~~
christophilus
Yes. The most successful projects I have worked on, we (the dev team, not
project managers, not designers, etc) went on-site often. We observed our
customers. We had lunch with them. We stood behind them as they worked. We
asked them questions. We were on _very_ friendly terms with them.

We built systems that literally had people say, "Oh, thank God!" when demoed.
I haven't seen any other development methodology that matches it. You really
have to understand a problem at a deep level in order to reason well about it.

~~~
mfoy_
I'd love if that methodology were more common. The "traditional" method
usually involves companies insisting we provide waterfall style proposals and
they don't want to invest in up-front "mini-project" of having people analyze
the problem _first_.

~~~
Terr_
And heaven-forbid any developer actually calls up a customer or subject-matter
expert without routing absolutely everything through one or more layers of
middlemen and bureaucracy.

~~~
mfoy_
"Just send me your questions and I'll pass them along..." Because no one has
_ever_ had a question answered and had a follow-up question based on the
answer before... /s

~~~
Terr_
True story: Myself and a bunch of internal customers are literally less than
50 meters away in the same office. The blessed middleman is in a different
timezone 8 hours ahead.

So instead of a quick conversation 30 minutes with helpful whiteboard doodles,
I have to compose my questions and decision-tree into an e-mail and wait at
least for a day or two for a reply that may or may not be useful.

~~~
mfoy_
Are there repercussions to just walking over and asking them then forwarding
the middleman a summary / recap?

~~~
Terr_
There was a "Hey, don't do that, X needs to be in the loop" complaint. (Didn't
see it myself, but technical program manager who was also in on the meetings
told me.)

Later that remote person left the company, and the position was left unfilled
for a year and some projects got mothballed.

------
dublinclontarf
"Being able to program is not the problem. Understanding the problem is the
problem" \- one of my lecturers in University.

------
exelius
Agree 100%. In fact, this is how I built my career -- I knew the tech ok, but
realized that I was never going to have the type of influence I needed unless
I got into the product side as well.

In the consulting world, we call this job "enterprise architecture". It does,
in fact, pay very well: it requires someone with both a sharp business mind
and comprehensive technical skills, and those are very difficult to find in
one person. I personally am more of a "jack of all trades" type; but you can
be a successful architect by focusing on specific technologies as well.

I honestly find that it's easier to take someone who's a hacker type, and
teach them the business. You look at the business itself as a large, complex
system and model your application development around that. But you also have
to be a good enough technologist yourself so that you can tell your dev team
when their designs don't match up to the business problem (this is a common
problem when requirements are not clearly communicated).

A good architect is the person who understands both the business context and
the technology implementation. You don't have to be in-the-weeds building the
product, but often you do have to build quick POCs to prove out an approach
before handing off the designs to development - so being able to code is a
necessity IMO.

------
yawz
_> The nature of the beast is that software requirements rarely change_

Put like this, it's hard to agree with the above statement.

Following the spirit of the article, I assume the author means that the
problem domain is pretty stable. But I've been in this for more than 20 years,
and I know that requirements always change. Not only our understanding, but
also the customer/user's understanding of their needs and priorities change.

 _(Edit: typo)_

~~~
naasking
> Not only our understanding, but also the customer/user's understanding of
> their needs and priorities change.

And not only does understanding change, but the _business requirements_
actually change too depending on market conditions.

~~~
cratermoon
I think the notion of "requirements" the author refers to is what the business
actually does. When writing say health-care management software for say, a
hospital chain, they're not going to suddenly decide, no matter how much the
small-r requirements change, that they instead want their software to be able
to do capacity management for lumber mills.

The business domain constrains the field over which the requirements can
change, and the sort of deep context which the author mentions will also range
over that field.

~~~
naasking
> When writing say health-care management software for say, a hospital chain,
> they're not going to suddenly decide, no matter how much the small-r
> requirements change, that they instead want their software to be able to do
> capacity management for lumber mills.

But they might start with wanting billing software, and then change their
minds and ask for software to do triage and scheduling operating rooms. That's
almost as drastic a change in scope as your big-r from health to lumber mills,
so I don't really agree with the distinction you're trying to make.

------
chaostheory
Yeah programmers / developers / software engineers act as an interface for
other people for computers. It's not surprising that these qualities of that
'interface' affects price:

\- intuitiveness i.e how easy is it to communicate with this person (language
fluency, etc...)

\- quality i.e. how well do this person understand not only the requirements
but also the actual goals

\- 'latency' i.e. how convenient and how fast can you communicate with this
person (time zone, can you both see facial expressions, hear changes in voice,
etc...)

~~~
davidjnelson
This is insightful. These are possible remotely though:

* intuitiveness i.e how easy is it to communicate with this person (language fluency, etc...)

Native English speakers have an advantage here.

* quality i.e. how well do this person understand not only the requirements but also the actual goals

Experience, empathy, critical thinking, intelligence. Not necessarily common
or easy but on site vs remote doesn't affect this really.

* 'latency' i.e. how convenient and how fast can you communicate with this person (time zone,

Hire people from your country or even your time zone.

* can you both see facial expressions, hear changes in voice, etc...)

Use video chat constantly.

Remote work is a skill like any other. It makes sense most employers that
offer it require 5+ years doing it previously. The article author makes a good
point. A great way to get this is to work at a company with many remote
employees and start on site before transitioning to full time remote.

Maybe some people are just cut out for remote too. I remember at the beginning
of my career running a business where I talked to the CEO of a mid sized
company regularly about his needs, and always delivered. He was thrilled and
amazed. It was just good listening, communication, programming skill and hard,
applied work. Nothing fancy.

The weird thing about the Bay Area is if you want to live on 5 acres in a home
built in the last five years somewhere quiet and pretty that is 20 minutes
from the office in traffic, you're basically looking at Woodside. On the low
end, those houses start at around 3 million. Good luck paying that mortgage on
the income of even two software engineers.

Whereas you could buy the same house somewhere else in California for 300k.

So even though it is indeed pleasant to have coworkers to talk to in person
for social needs, the compensation to housing cost math just flat out doesn't
come close to working unless you are willing to make some serious housing
quality sacrifices.

Edit: typo.

------
indogooner
This is a post which touches on a subset of "hard" parts while downplaying
other hard parts. I agree that a developer with a deep domain knowledge is
precious. In fact, I have seen quite a few developers rising up the hierarchy
despite being mediocre on technical skills. The code written by them is big
ball of mud but they are good at communicating with stakeholders and
understand the business well(And also good at drawing boxes) . This does not
mean that there is no cost to it. It is just hidden from others. The memory
leak issue, the several bugs introduced due to multiple if-elses encoding the
business rules, swallowing exceptions or failing to set right properties for
client timeouts. These can exist despite the "deep context" and most of the
time junior developers are to be blamed for this because the so-called
architect is busy all day in meeting with management.

Also the experience may vary. In my admittedly not so long career(less than a
decade) I have seen teams where business rules are the major source of
complexity while there are other teams which have less business rules (example
the databases/data warehouse/build systems team). Admittedly there are less
teams of second type in the world so the general perception is that the
hardest part is communication and understanding of business context.

Coming to domain knowledge, even the Mainframe and COBOL chaps make a lot of
money while smart open-source contributors freelancing don't. Money is not the
only way to judge the hardest problem about software development.

~~~
walrus1066
The way I see it, my job isn't to just blindly implement what the business
thinks they want. It's a two way thing. If a set of business rules are too
complex and convoluted, I explain that this incurs a cost, in increased
likelihood of bugs, and slowdown of future development. We then think about
how we can simplify the rules, or reduce complexity elsewhere. In one such
case, we agreed with the business to remove an entire feature, to make the new
feature more robust and easier to develop. I deleted 1/3 of a codebase as a
result, ultimately the overall complexity of the system was substantially
reduced after the new feature. It was immensely satisfying.

~~~
indogooner
This is where a good product team helps. Understanding the threshold at which
technical debt would undermine a business feature. Glad that it worked out
well in your case

------
walrus1066
It is exactly this which makes interviewing so flawed in the field.

Solving well defined algorithm puzzles has zero bearing on the skills
described by the author.

------
Kadin
If you found the article interesting or insightful, I would encourage you
strongly to take a look at the book "Peopleware" by Lister and DeMarco. The
first edition was published in 1978; the edition I have is from the early 90s,
and I think there are newer versions since (at the usual astronomical cost of
books that get sold as required reading for college courses), but you can get
whatever is cheap for a used copy. Most of it hasn't changed much.

That, ironically, is part of the authors' point: software engineering hasn't
changed that much. They were saying that in the late 70s, looking back on the
past two decades (all the way back to the late 50s, when software was
typically written in assembler or machine language and had to be rewritten if
you bought a new computer!), but it hasn't become much less true. The
technology changes, sure, but the failure of software development projects has
never been mostly the result of technology. It's always been on the human side
-- failure to understand the requirements, failure to meet them, failure to
estimate the effort appropriately, failure to work with the customer, failure
of the customer to understand what they were paying for, etc. There's a long
list of things that go wrong, and I suspect anyone who has been in software
for a while has seen many of them.

It was eye-opening for me, the first time I read it, to realize that people
had been dealing with the same issues I was dealing with, for longer than I'd
been alive. (And a bit depressing, too, that we seemingly haven't gotten much
better.) Languages and project-management methodologies come and go, and the
tech skills and understanding are certainly necessary, but they are not
sufficient. The business knowledge and human factors seem to be the
difference, or at least the largest controllable variable that leads to a
difference, in a successful or failed outcome.

------
yawz
_> Sadly, offices are perhaps the closest modern equivalent to a "village"
that we have left._

Ha! We need a "tribal leadership" discussion forking off this point.

------
ensiferum
Why does some independent contractor make bank while others are fighting for
pennies? Because it's not about "what you know but _who_ you know."

------
markbnj
The author could have just left the "Remote" part out. This piece is really
about hiring independents for gigs vs. building and educating your own team.

~~~
mfoy_
I thought something quite similar, but upon further reflection the "remote"
buildup and conclusion sets the narrative frame for the rest of the article.

Rather than just a "wisdom dump" it becomes more of a story with a purpose.

~~~
scierama
Yes, the entire article is an attack on remote working and this suits Amazon
well since they keep trying to get people to relocate to Seattle.

~~~
mfoy_
It is not an attack on remote working. It is an attack on the naive assumption
that there is worthwhile value in being handed a stack of detailed
requirements and implementing them with no back-and-forth.

------
petraeus
Once you are experienced enough you'll realize two things, that all
programming languages let you solve problems and that programming in itself is
not the problem, but rather applying it to solve business requirements is the
true problem to solve.

------
methodin
This is an insanely useful article to read before a major project kick off.
Kudos!

------
FLUX-YOU
Here's the problem:

\- Here's a pool of knowledge about software development: hardware, operating
systems, memory, disks, file formats, databases, networks, protocols,
languages, debuggers, design patterns, security, accessibility, UI/UX,
distributed systems, paradigms, typical algorithms & data structures, and CS
problems

\- There's a pool of knowledge about whatever industry you get into as a
developer: user demands, existing workflows, existing infrastructure, previous
decisions, legal regulation & compliance, physical laws, profitability, and
practical limits.

Your software development skills should reach a point where you don't write
"Bad Code" \-- anything that's wrong like loading a entire database table that
eats memory when you can read individual rows, storing passwords in cleartext,
or not doing anything for accessibility (this is not design pattern, space/tab
debates). These have been done hundreds of times by new and 'experienced'
people.

It takes time to get to this point. More time than anyone likes to admit
because the pool of knowledge grows and shrinks daily, but has undoubtedly had
a net expansion since computers were a thing.

It takes time to get deep knowledge about whatever industry you get into. This
is different for every industry. There's a practical minimum that you need to
work on solutions or do maintenance on software within this industry. This is
to avoid "Bad Code" which will hurt you, other people, or your business.

You can gain industry knowledge by just being given problems and being shown.
This is probably how most of us know our industries from the get-go. A
minority of us came from those industries and transitioned to programming
later, so we already had a base level of knowledge of our problems.

If I've got the definition of Deep Context right from this article, it means
to get to that point, you have to spend a good amount of time within the
industry. It's not something you can gain completely by reading out of a book.

If you're to gain deep context within an industry, you have to devote some
time away from software. You can't do both at the same instant (but certainly
within a day). When you study an industry, there's an opportunity cost to not
learning something new about software and vice versa.

When you add more requirements to a single job, it increases the time we have
to spend before we're employable. Not every industry changes as fast as
software does, but some certainly do, possibly catalyzed by software.

If you increase the time requirements, it's going to reduce the available pool
of engineers as long as all of the engineers are honest and don't apply for
jobs or remote contracts until they're ready.

If you don't want the time requirements to increase, you have pay the
opportunity costs from one of your pools of knowledge.

So really, we need a much better "good enough" for employing developers and
career development, including teaching software and industry knowledge.
Because eventually the time requirements are going to become steeper and
steeper. It can't go up forever.

~~~
MrPatan
There is hope.

How about instead of becoming the expert at whichever business domain the
software is for, we become experts at helping business domain experts find and
express the business rules that need to be implemented?

You can fit a decent amount of that skill and the technical knowledge you
mentioned in one skull, and it still lets you be quite effective in more than
one domain.

That's what I'm going for anyway. Wish me luck.

~~~
FLUX-YOU
I think the deep context is what the author is using to solve that issue. Devs
are fluent enough and have enough context that business domain experts don't
have to spend an inordinate amount of time going into detail. The level of
detail cascading happens in the developer's head as opposed to conversation.

It's really an argument for who's going to spend the time and appears simple:

\- Devs learning software and industry knowledge

\- Business experts learning software knowledge (incl. technical writing) and
industry knowledge

Product Managers with some technical knowledge and writing skills are best at
being a middle layer between raw customer requests and development specs in my
experience. PMs and customers struggle when they don't have a good vocabulary
to use to describe features that they want. That's when a dev has to translate
or teach the person. Then again, that's asking a PM to learn industry
knowledge and technical knowledge and product management knowledge. This is
especially true if you have a good QA pipeline.

I've seen analysts and PMs that didn't have a good UI/UX vocabulary or weren't
exposed to different UI/UX's, and usually their requests were the most vague
and resulted in the most unspoken details.

I've also had PMs that knew how to write a good technical spec down to quick
UI mock-ups and error handling. They also had technical writers to pose
questions about some of the details.

Pretending I could be as good as the latter is foolish, and if I could, my
salary should have been combined for doing 3 jobs well. I think one-man-army,
$250k/yr full-time positions are rare though. We seem to be inching closer to
it though, maybe without the salary.

------
cryptos
This article sounds a lot like a big argument for Domain Driven Design without
naming it.

------
Flimm
Is there a way to read this article without logging in?

------
Tutankamon
The hardest thing about software development is having people in charge of you
that know jack shit about software.

~~~
softawre
Funny, I've always thought the hardest thing about being an engineering
manager is that by default all software engineers think they automatically
know way more about everything than their management.

~~~
creeble
Maybe they just know way more about software development than the engineering
manager if said engineering manager has never done actual software
development.

Of course both viewpoints are valid, I'm just really feeling the commenter's
point today because of my particular circumstance.

------
s73ver
I thought the hard thing in software development was naming things and off by
one errors?

~~~
johnny_reilly
Close. It's cache invalidation and naming things. But I feel ya

~~~
s73ver
To make the joke work, I could only pick one. I thought naming things was the
funnier option.

~~~
creeble
The joke is "There are only two hard problems in software: Naming things,
counting, and naming things."

