
The plumber programmer - ColinWright
http://www.johndcook.com/blog/2011/11/15/plumber-programmers/
======
nadam
Most of the enterprise developers' job is connecting components together. I
hate it. I hate modifying existing systems and hate connecting them together.
It is usually hard, but the kind of hard which is not really fulfilling for
me. It is about fighting with accidental complexity created by other people.
It is not why I started programming when I was 12 years old. But this is what
needs to be done: it is rational from business perspective. Of course at least
in my side project I create something new from scratch (a component-based UI
system painted on HTML5 Canvas).

~~~
fleitz
It's not that difficult when you work with the right tools.

Most of these problems boil down to a couple operations.

    
    
      map
      reduce
      filter
    

Most of the data comes from a couple places:

    
    
      Some kind of SQL system
      Text files
      Webservices (of the JSON/XML/HTML variety)
    

Most of the data goes back to the same places that it came from:

    
    
      Some kind of SQL system
      Text files
      Webservices (of the JSON/XML/HTML variety)
    

All you need to make are four types:

    
    
      inputs: () -> Some<T>
      outputs: Seq<T> -> ()
      pipes: Seq<T> -> Seq<Q>
      tees: Seq<T> -> () -> Seq<T> -> Seq<Q> (once partially   applied the tee becomes a pipe)
    

Using the usual sequence operators pretty much anything is possible with those
inputs, outputs, and transforms (pipes) and you avoid a lot of overhead making
data pipelines.

~~~
dustingetz

      > Using the usual sequence operators pretty much anything is possible
    

i would absolutely love if this is true. however, i don't know of anyone who
has constructed a convincing argument that it is true. even if it were true,
it is probably impractical to require all systems you interface with to be
composed in these terms. is it cheaper to work with a ball of mud (which
poisons everything it touches), or is it cheaper to rewrite the ball of mud in
the style you describe?

" _People build BIG BALLS OF MUD because they work. In many domains, they are
the only things that have been shown to work. Indeed, they work where loftier
approaches have yet to demonstrate that they can compete.

"It is not our purpose to condemn BIG BALLS OF MUD. Casual architecture is
natural during the early stages of a system’s evolution. The reader must
surely suspect, however, that our hope is that we can aspire to do better. By
recognizing the forces and pressures that lead to architectural malaise, and
how and when they might be confronted, we hope to set the stage for the
emergence of truly durable artifacts that can put architects in dominant
positions for years to come. The key is to ensure that the system, its
programmers, and, indeed the entire organization, learn about the domain, and
the architectural opportunities looming within it, as the system grows and
matures._"[1]

Maybe, you are correct. Probably. We all hope. But, practically, in the
'bigger' problem domains, you can put a lot of smart and experienced people on
a project, and it still comes out a ball of mud. Maybe, there just aren't yet
enough experts to go around.

[1] <http://www.laputan.org/mud/> (conclusion)

BTW, any references you know that can articulate your assertion without hand-
waving, I would love to read. I must read. I'm currently devouring everything
I can about FP and there's a lot of concrete stuff, even more stuff with
handwaving, and then some alarming articles about people who did their startup
in haskell and wouldn't do it again, even if for out-of-band factors.

~~~
nickik
Well I think its only cheaper at first and even if people know better they are
forced to use a stupid java framework (that abstracts away the network) even
if the know that it will end up badly.

A other problem is that people often get tought that you need <input some bad
framework>. You can't do distributed computing if you done use some kind of
framework. At least the places I know they would never tell you something like
"just send json from one node to the other if thats all you need".

At clojure conj there where some talks about this, but the videos are not out
jet. See these presentation on Concurrent Stream Processing
([https://github.com/relevance/clojure-
conj/blob/master/2011-s...](https://github.com/relevance/clojure-
conj/blob/master/2011-slides/david-mcneil-concurrent-stream-processing.pdf))
or this on Logs as Data ([https://github.com/relevance/clojure-
conj/blob/master/2011-s...](https://github.com/relevance/clojure-
conj/blob/master/2011-slides/mark-mcgranaghan-logs-as-data.pdf))

For a other example that works in quite simular ways look at Storm (in use at
twitter. Its all sequential abstractions. See this video by Nathan Marz (look
all videos you can find): <http://www.infoq.com/presentations/Storm>

For a more philospical perspective look at the videos by rich hickey:
<http://www.infoq.com/presentations/Simple-Made-Easy>

------
AndrewDucker
Definitely. An awful lot of programming is "Take data from place A (in format
1) and place B (in format 2) do magic thing J to it and put it in location Z
(in format 3)".

Process J is usually the simple bit - the hard bit is taking it from A and B,
converting (1) and (2) so that they can be used together, dealing with all the
different eventualities that can cause things to go wrong when fetching the
data, parsing the data into (3) and then writing to Z (while dealing with all
the things that can go wrong when writing to it).

~~~
praptak
_"Yeah, meanwhile Z is temporarily unavailable and by the time it's available
again, the data from A and B is no longer consistent, so you have to re-query
again, ouch, your pointers to last read data on A are no longer valid anymore
and yeah, we forgot to tell you this little funny thing about B... <the
dramatic music in the background pauses here>"_ \- this is what you should be
thinking when looking at a small arrow drawn on a whiteboard.

~~~
patio11
In my experience in Big Freaking Enterprise development, the issue is closer
to "this little funny thing about B" x 100, plus "group Q doesn't want this
project to succeed", "you have to be compliant with Regulation F which means,
most relevantly, that all good technical solutions to this problem are legally
forbidden", "your success with this will threaten the budget of the Z
department -- be prepared for a knife fight with the VP", "group P will veto
the project if it is delivered with property G and group R will veto the
project if it is delivered without property G (bonus points awarded for values
of P equal to R)" and "by the way, this doesn't advance an actual business
goal."

The technical details are usually pretty boring. Data goes in, data comes out,
if it weren't for all the bloody people involved it would be very civilized.

~~~
cageface
I spent over _two years_ trying to get the contents of a single dropbox on one
form changed. Coding time, about 10 minutes. I guess if you want to look on
the bright side you can think about how much political weight a single HTML
element can carry.

------
uniclaude
Very true.

Funny anecdote, last time I heard about "plumber programmers", the meaning was
very different and actually quite pejorative. The person I was talking to was
referring to the type of programmers that can only write applications by
assembling third party components such as ruby gems while lacking the
algorithmic skills to solve problems that haven't been solved before.

~~~
rick888
I've talked about this too. Sadly, this is what programming has started to
become: slapping third-party components together and calling yourself a
developer.

It's one of the reasons I tend to stay away from things like Django and RoR.
It makes it easy to develop applications quickly, but I many times have to
create hacks to make any changes that don't fit into the one-size fits all
libraries.

------
j_baker
I think it depends on context.

A well-architected application (they _do_ exist) where a lot of work was put
into designing the arrows and boxes will suffer from this effect less so than
one where the arrows and boxes were added on haphazardly. In fact, I would
argue that spending too much time worrying about the arrows is a sign that
something's likely wrong with your architecture.

Of course, that's not to say that architecture can cure this altogether. In
particular, enterprise software tends to focus on integration a lot, and
mostly because enterprises have a lot of boxes that need to be put together.

Startups will tend to do a lot less of this, but it's still important.

------
mullethunter
For the past 8 years I've been designing .NET enterprise integration solutions
and I love it. I love the problem solving that is required and working on
getting square pages to fit seamlessly into round holes. Recently I switched
companies and was working with jQuery and PHP doing front end things like UI
validation and was losing my mind. It's not the pencil and paper type of
challenge as integration design.

~~~
zheng
Wow, could you elaborate more? This sounds like a very interesting viewpoint,
and one that I've not seen before.

------
timclark
I always shudder when people say it is just a simple integration project. The
arrows are the hardest part.

------
rachelbythebay
I've used the "plumber" term to refer to my own deeds, if only because they
don't resemble the shiny and showy stuff which is out there. I don't make
games, and I don't generate graphics. My stuff tends to be the glue which
makes other things happen in the first place.

I'm far more likely to write the server underneath that huge multiplayer game
and deal with all of the systems stuff, in other words. The actual gameplay,
graphics, sound, and user interface? That's someone else's realm.

------
jonmc12
Reminded me of this section from the Rspec book -
<http://cl.ly/2x231s0N2v3C2P1S1z04>, <http://cl.ly/2R1Z1D450Y1n1n3L170b> \-
which applies to testing object-oriented code by mocking objects and putting
assertions on the interaction between objects (vs vis versa).

"Object-oriented systems are all about interfaces and interactions. An
object’s internal state is an implementation detail and not part of its
observable behavior. As such, it is more subject to change than the object’s
interface. We can therefore keep specs more flexible and less brittle by
avoiding reference to the internal state of an object."

This 'Plumber Programmer" article is meant to be more general than object-
oriented code, but I think the analogy applies. Building systems is about
interaction and communication - system state is just an illusion that gives us
a first approximation of how things talk to one another.

------
run4yourlives
I was hoping this was going to be about programmers that moonlight as plumbers
on the side.

------
gurtwo
At the telecom company I work for, this is called 'System Integration'. It
takes most of our time.

------
dexen
Tangentially related: the plumber program from UNIX R11^W^W Plan 9 makes
programs work together by piping text around.
<http://doc.cat-v.org/plan_9/4th_edition/papers/plumb>

Again, there is more value in the connections than in the individual programs
themselves.

------
anamax
Seymour Cray said that he was an overpaid plumber....

