
The Developer is Dead, Long Live the Developer - roidrage
http://www.paperplanes.de/2014/4/17/the-developer-is-dead.html
======
hibikir
In my experience, the teams that run the best are full of generalizing
specialists. So you have three people that are quite handy with the DB: One is
the best of the lot, and is mostly doing DB work. The others are good at it,
but they bring unique strengths in other areas. In the same way, you have a
guy that is pretty strong at Unix administration, a language lawyer, some
people that are big into UIs and such.

So while people will probably dedicate 80% of a week to a given kind of task,
they can, and do, play multiple roles, depending on how much their specific
expertise is needed that week.

For instance, I am the number one choice on UIs, but I also play support
managing Postgres, because I did a whole lot of that in a past life, and we
don't need a full time DBA. I am also leveling up in our scheduling system,
all Akka actors.

At my previous job, the team had all the Database experience it needed, but
domain knowledge was weak, so I spent much of my work working with users
trying to figure out what they needed, and why they were unhappy with the
product. They hired me for my UI strength, but reality said that I would be
more useful doing customer facing activities half the time, so I did.

------
venomsnake
The developers should know how to do ops. And vice versa.

We are moving into weird heterogeneous bugs territory. A lot of stuff that has
wasted my time last year was caused by interaction of technology stacks and
not by pure logical errors. Or error localized in even one package.

I had problem with premature socket close on the response with just the wrong
setup for all of nginx, haproxy, slim and php 5.4 ... lets pure ops or pure
dev figure that out.

~~~
festivusr
The ops people you work with are also developers? That sounds great but has
never been the case at anywhere I've been.

~~~
venomsnake
They are not developers - they have no rights to write commit and promote code
but they are good enough at reading it to not be helpless when something
breaks down and they need to take a peek what has changed recently in the
codebase and stuff like that.

I on the other hand have no admin rights (only read access) but I know enough
of out network and ops structure to be able to make an educated guess when
something is behaving strange.

So if it hits the fan they could probably patch an emergency bug and I could
fire up some VM instance. It is not about wearing a lot of hats, but being
able quickly to put on a new one if needed.

------
agentultra
I argue that the shared responsibility mentality is distracting and diminishes
the power of an individual. It's hard enough being a good programmer let alone
one that also knows how to configure networks, firewalls, and operating
systems; one who knows how to manage security policies, LDAP organization
charts and ACLs, etc. I still believe that the jack-of-all-trades is a useful
individual but I don't believe an entire organization should be formed around
their ideals.

As in all things in life -- the truth is somewhere in the middle. If you build
significantly sophisticated systems you will need specialists at some point. I
don't recommend wasting their talents on getting reamed out for forgetting to
configure the firewall properly.

~~~
mfisher87
I don't think that's what the author is arguing for. More like, "You build it,
you run it" (quote via article). Instead of handing your code off to someone
else to be 100% responsible for testing, for release, for production support,
the author argues that it is better to have developers share responsibility in
those roles. It's very similar to what I'm doing now.

We don't hand off code to be installed, we give a release plan, an
installation plan, and the code. We are on call for first runs, and there's
always a subject matter expert developer on call to support overnight issues
if the support team needs help. If you have this responsibility, you write
_more responsible code._ If I may support something overnight, I'm not going
to unthoughtfully make it hard to support. We share testing and documentation
responsibility as well (although documentation is an issue), so again testers
and support are doing their job in a vacuum because they don't have the
developer's insight to the program. In fact, devs do most of the testing. It's
not always easy to write test cases for a system you didn't write or design.
I've kind of always thought that it's common sense to support your own product
in test/production as needed (though not the first point of contact). You
wrote it, you're the expert.

Reading this article surprised me because I thought it was the same
everywhere. Having no responsibility for testing or prod seems like asking for
bad code. If you're _just_ writing code, you're not going to focus on what
it's going to be like to support it, install it, maintain it. You'll have
other, more concrete (to you) things to focus on. Coding standards may become
more important. But are standards really as effective if they're not informed
by / don't care about responsibility for a production product?

~~~
agentultra
Perhaps it's that funny definition of "devops" blurring the lines.

I think it's perfectly reasonable for a developer to be aware of excessive
memory fragmentation, leaking FD's, and anything else related to the process
their program directs. It's well within my domain to debug such issues.

However the "devops" role is also pushing responsibility for configuring
firewalls, LDAP ACLs, securing the OS, managing init daemons, and the entirety
of the infrastructure on developers. Since it's all code now there can be more
people involved and the silos can come down. "You build it, you run it," as
you say.

What I'm suggesting is that for significantly sophisticated applications it
won't be tenable to spread the responsibility on everyone. It's not reasonable
to suggest that everyone be familiar with the implementation of a simulated-
annealing algorithm in the continuous-domain constraint search solver. I think
a support person would be well within reason to defer questions, bugs, and
requests with that particular part to the developer responsible for it. And
yet we're arguing that it's perfectly acceptable for that developer to be
familiar with the intricacies of managing a data center of compute clusters? I
don't know about you but I don't have an infinite amount of time on this Earth
to take on such responsibilities.

I do think that there's a middle-ground to be exploited. If our expert system
is ballooning in memory usage and crashing our servers it's reasonable for the
developer to profile the memory usage under simulated loads and data sets and
look for memory fragmentation or leaks, etc. I think it's a little too much to
ask one person to also be intimately familiar with managing compute clusters,
RAID failures, security policies, and package management. They, like many of
us, may have a good, general idea of how it works but there exist specialists
for a reason: it's hard to go beyond a certain degree of complexity without
focusing. It's the reason why we work in teams!

~~~
mfisher87
>It's not reasonable to suggest that everyone be familiar with the
implementation of a simulated-annealing algorithm in the continuous-domain
constraint search solver. [...] we're arguing that it's perfectly acceptable
for that developer to be familiar with the intricacies of managing a data
center of compute clusters? I don't know about you but I don't have an
infinite amount of time on this Earth to take on such responsibilities.

No one is arguing for that, as I said. Where in the article do you see
something like that? No mention of firewalls, data centers, developers doing
their own server administration. This seems to be a more important takeaway
from the article:

> Ultimately, DevOps is about empathy, with everyone on your team and with
> your customers.

No one is suggesting "everyone does everything" at all... you're sort of
creating an argument here.

------
devonkim
This reminds me of going back towards a 90s job title trend - the webmaster.
While that applied very much for small Geocities sites, there seems to be an
awful lot of overlap and even business alignment toward minimizing operations
staff / budget such that today's so-called devops engineers would really have
been the webmasters then and in SaaS shops we're just a team of webmasters
that have deeper specialization / interest in specific areas.

Regardless, this pretty much only applies for SaaS shops I'd say. There's
still software delivered the old way that's viable.

------
Shivetya
I have never been in a shop where as a developer I was isolated from quality
assurance or production. Fixing production issues was part of the job and
insuring the fixes and original code got through QA was there too.

However I have also seen this branded as "soft skills". Something that seems
to be yet another way to excuse many people for their lack of skills and
reward them for doing other stuff. I have seen it used to keep really bad
developers, if not reward them. It also tends to be a favorite term of those
who cannot manage their team properly.

~~~
mlieberman85
There are companies out there where there is that isolation. Especially in the
financial sector. I worked as a vendor to several big banks and credit card
companies. Devs were expected to write code for functionality, that's about
it. If there was an issue with the production environment and it wasn't
related to a deployment, it was operations' problem. For example if my app has
major memory issues every 3 hours or so it was operations' responsibility to
restart that app when need be and ensure it stays up and running not the devs'
responsibility to fix the memory issues. This isn't always the case and often
what was mentioned above were temporary solutions intended to give devs more
time but there was definitely that disconnect where operations is responsible
maintaining the running state of the environment and the devs were primarily
concerned with writing code that works and provides the functionality it is
supposed to but not concerned with making the code run well.

~~~
poulsbohemian
These are great examples and response to the parent. Just this morning I was
on a call with a financial sector customer who made the statement "we don't
want developers to have that information / access to that information" when
referring to an operations dashboard. The silos created in corporate IT are
huge, with corporate IT organizationally structured in a kind of polar
opposite to the typical software venture.

------
lectrick
Programming these days seems to be more of an exercise in managing unexpected
complexity between various components or pieces of code. Here's a few tips:

1) Unit test. 95% of your test suite should be unit tests. Objects under test
should not require the entire codebase to be loaded in order to perform...
ideally they depend on nothing, or just 1 or 2 things (which in turn hopefully
depend on nothing, or just 1 or 2 things, ad nauseum).

2) Don't mutate passed-in arguments. In fact, mutate as little as possible.

3) Don't use objects with potentially unexpected, surprising or unknown
behavior that makes it difficult to reason around the code (QUICK- what
happens when you merge a ruby HashWithIndifferentAccess with a regular hash
containing both similarly-named symbol and string keys?)

4) Use function objects, that have zero side effects, wherever it makes sense.
(Or just use an entirely functional language.)

5) Separate components that talk to each other through I/O should use
something like the circuit-breaker pattern
[http://martinfowler.com/bliki/CircuitBreaker.html](http://martinfowler.com/bliki/CircuitBreaker.html)

6) Microbenchmark things. Individually, a series of tasks might all "look"
fast, but when run 10000 times in a row, might expose unreasonable resource
utilization.

------
auganov
I think I'm a bit skeptical of how different the responsibilities of a
'DevOps' are different from a traditional 'Developer' in terms of actual
complexity. The whole discussion seems to reduce technological change to just
a little tiny part of this 'transition' to DevOps. How can you even develop a
web SaaS application without having a basic knowledge of
sysadmin/networking/IT/etc? In my opinion technology changed and development
methodologies followed. I wish the author proposed an alternative to status
quo [that would resemble the past a bit more]. Can anyone? Of course adopting
PaaS and especially BaaS solutions can change it, but again, that would be a
technological change rather than just a methodology thing.

------
prawks
> Putting developers in charge of not just building an app, but also running
> it in production, benefits everyone in the company, and it benefits the
> developer too.

How do companies which do this keep the support workload of developers low
enough so that they have time for development? It's a great idea, because as
the linked interview with Wener Vogels proposes, it creates a developer that
has more frequent contact with customers. I suppose eventually you have to
shift resources around to start other projects, but then who supports what
they're leaving? Even in large teams, eventually expertise will dwindle until
you're adding new people to support an existing system.

~~~
dasil003
It's about striking the right balance.

I don't really agree with the original totem-pole idea that devs can do
anything QAs and ops can do. While it may be true that a good dev is capable
of muddling their way through any challenge because they know how to get
around the numerous blocks thrown up by technology, there are definite skills
to QA and Ops that developers may be lacking and which will lead to the dev
being a poor substitute.

That said, I think devs have skills which are applicable _anywhere_ in an
organization. This goes back to the virtue of a programmer being laziness.
Being able to see how to build systems and automate things is something that
is potentially useful _everywhere_. In the cases of ops and product support,
there is an especially large amount of value from having devs poking their
head in from time to time to pick off easy targets for improved automation,
reporting, logging, or what-have-you.

~~~
bmj
_It 's about striking the right balance._

Yes. My company has tried for a good many years to keep product development
partitioned from operations. We build stuff, through it over the wall, and let
the ops team deal with it. Certainly, we give support where necessary, but it
tends to be grudgingly given, and we never actually track time spend doing
that support. I've been pushing, along with several other developers, that we
at least need some sort of rotating product support team so we can work with
ops to resolve problems in the field without disrupting our own timelines.
We've moving in the right direction, but there's a lot of inertia to overcome.

------
Pinwheeler
I read both articles and felt they both made good points.

It is in a company's (and the industry's) best interest to pair down well-paid
developers and open up more entry-level positions, however, it's in the
consumer's best interest for developers to maintain high accountability and
"closeness" to the product.

I don't see these approaches as mutually exclusive I see how full stack
development and silos can both exist in the same environment like if the
Dentist in the counterpoint article's analogy takes over the hygienist's role
when the hygienist is out on vacation.

------
thothamon
It's always good to have a wide base of knowledge, and it's always good to
avoid ivory-tower silos. But at the same time, it's economically inefficient
to use a specialized developer to do tasks that could be done a lot cheaper by
others. It's like using jet airplane to haul a tractor-trailer. Maybe you can
do it, but eventually someone will exploit that inefficiency and out-compete
you.

------
tkyjonathan
Maybe this can relate to the other story in HN about building boring systems
that just work... If you are in-charge of keeping the app running, maybe you
wont want to mess around with cutting-edge technologies.

------
chris_mahan
To me, the rise of DevOps coincided with the commodization of server services,
where provisioning of services could be performed by the software itself.

It's another abstraction layer, and requires new skills.

------
tianyi-aisin
> What really matters is the willingness to change, to learn a new stack when
> necessary.

You've hit the nail on the head there !

------
roncohen
Had been looking for the source of that Werner quote for a while. Thanks
Mathias!

------
michaelochurch
Elephants in the room: unfunded mandates, disparate equity, control issues,
and _deadlines_.

Ideally, people should support their own work. If I build something and have
complete ownership of it, I'll make sure it runs. If I'm the one who's getting
the money and career progress, and I'm picking the deadlines, I'll be the
"3:00 am guy". It seems fair that it should be that way.

On the other hand, if I'm asked to build someone else's idea, to their
deadlines, making technical compromises that I wouldn't choose, then I'm not
going to be the "3:00 am guy" for shots someone else called. Fuck that shit,
life's too short. If you want that from me, either pay a consulting fee ($250+
per hour) or give me serious equity (like, founder levels). Otherwise, That's
Not My Job.

Salary jobs are for the stability of pay and work (i.e. _not_ being the 3:00am
Guy) and for career advancement (resume, respected titles, networking). That's
what I'm paying for when I accept a salary 1/10 of what I'm worth to the
business. If I'm expected to take on extra duties that don't advance my
career, and not getting that stability, then I'm not getting what I paid for.

I'll gladly take the negatives of ownership if I get to partake in the
positives (autonomy, self-direction, participation in the reward) but
expecting me to take only the first is just unreasonable.

