
My Heroku values (2015) - sotojuan
https://brandur.org/heroku-values
======
karlmdavis
I don't think I've ever read about an engineering culture that I agree with
more than this one. It sounds shockingly humane and practical.

From the somewhat... wistful tone in places, I gather things have taken a bit
of a downward turn there. I wonder what pressures caused that? The
acquisition, perhaps?

~~~
craigkerstiens
Just a note this was penned several years after the acquisition.

------
kelnos
> _But although technical management wasn’t an idea that lasted, it was a very
> good place to be an engineer while it did._

I suppose this may just be something that you either like or don't like, but
my experience has largely been the opposite. I don't want a boss who can
override my technology decisions just because they're the boss. Even if you
only hire people who wouldn't, the fact that they _can_ creates something of a
chilling effect.

I think software teams should have a technical lead (who the team does _not_
report to) in order to make sure the team's technical direction is cohesive
and consistent. The manager (who the team reports to) should be responsible
for their reports' professional development, and should be more of a "team
coach" that facilitates interaction with other teams and other departments,
and might do things like own (but not have final/unilateral say on) the
delivery schedule and backlog.

If you have a manager who is responsible for a decent amount of technical
work, the important work of _actually managing_ \-- of which there is plenty
-- will suffer. I suspect this sort of leadership worked ok in the early days
of Heroku because the company was small enough and the early engineers
scrappy/independent enough that they didn't really need much in the way of
management.

Some of my worst interactions with managers has been when they believe they
should stick their nose into technical issues. Perhaps the problem there is
that they're not technical _enough_ , and that if they spent enough time
coding, they'd be more competent when participating in technical decisions...
But then I still maintain that 1) even just holding -- without playing -- the
"boss card" creates unwanted tension, and 2) they'd be put into a position to
be much worse at actually managing.

------
phamilton
Has the Heroku story improved at all for stateful services? I've always seen
clustering as essentially unsupported. Doing websockets with something like
Phoenix/Elixir means that you have to bring in Redis for pubsub between nodes,
which has all sorts of quirks and limitations.

~~~
derefr
> Doing websockets with something like Phoenix/Elixir means that you have to
> bring in Redis for pubsub between nodes

I haven't even tried this myself yet, but it's been stewing in the back of my
mind: you should be able to (efficiently) pass sent messages between your
nodes _using server-to-server websocket connections_.

You just need

• some supervised GenServers that establish and manage long-lived connections
to your peers using a websocket client library, e.g.
[https://github.com/jeremyong/websocket_client](https://github.com/jeremyong/websocket_client)
or [https://github.com/ninenines/gun](https://github.com/ninenines/gun)

• a "server peer RPC" Phoenix Channel, that authenticates server peers in its
connect/3 using a common pre-shared key (ala Erlang's own "cookie") or
somesuch

\---

Going a bit further: once you have these two components set up, you can
actually extend it out to being a fully-fledged transport layer for the Erlang
distribution protocol.

Look at the guide at
[http://erlang.org/doc/apps/ssl/ssl_distribution.html](http://erlang.org/doc/apps/ssl/ssl_distribution.html).
It mentions two reference implementations of a distribution-protocol
connection handler:

• inet_tcp_dist
([https://github.com/erlang/otp/blob/master/lib/kernel/src/ine...](https://github.com/erlang/otp/blob/master/lib/kernel/src/inet_tcp_dist.erl))

• inet_tls_dist
([https://github.com/erlang/otp/blob/master/lib/ssl/src/inet_t...](https://github.com/erlang/otp/blob/master/lib/ssl/src/inet_tls_dist.erl))

Between the two, the API an `inet_websocket_dist` would need to follow is
pretty clear.

With such a thing, you get cluster-presence and cookie-based auth "for free",
and automatic leader election for use with `global`. But most of what you want
is probably just rpc:multicall/3.

~~~
phamilton
sigh...

> Any sufficiently complicated concurrent program in another language contains
> an ad hoc informally-specified bug-ridden slow implementation of half of
> Erlang.

Except this time, it's already on BEAM...

------
bizon
_I might avoid some of Heroku’s current technological staples (e.g. Ruby)_

What instead of Ruby?

~~~
ed_balls
They rewrite a lot in Go.

~~~
sotojuan
The author, while not at Heroku anymore, primarily works with Go now.

------
twic
_In a similar vein, picking one true way forward from a collection of similar
options helps keep engineers productive as they move between components._

I am a strong believer in this. However, i believe it's a wildly unfashionable
idea in this day and age, where developer autonomy, 'best tool for the job',
and polyglossia are the in things. I was interested to see it in this
otherwise pretty hip and happening list.

 _The idea was that making coffee would be five to ten minute process, during
which there would be time to interact with colleagues who happened to drop by
the area. The system worked._

There's a similar meme in medical research institutes (at least, the ones i've
worked in), which was popularised by the Laboratory for Molecular Biology:

[http://www.insight.mrc.ac.uk/2013/06/06/the-great-coffee-
bre...](http://www.insight.mrc.ac.uk/2013/06/06/the-great-coffee-
breakthrough/)

 _If an engineer needed a new resource for a service being deployed,
prototype, or even one-off experiment, they were at liberty to provision it
and keep on working, even if that resource wasn’t free. Resources here might
include everything from dynos to run an app, to a Postgres backend, to a few
extra EC2 boxes for deployment to bare metal (relatively speaking)._

I've come across this "self service" idea as a selling point for internal
platforms and PaaS, but there seems to be a small but important difference -
with a platform, developers can self-service anything that is _part of the
platform_ , whereas it sounds like at Heroku, it was anything - _full stop_.
Although, as the author mentions, the fact that they were on AWS meant that
that was ultimately still on a platform. Anyway, that difference can matter if
there are things that aren't on the platform - if you're using Cloud Foundry,
for instance, then you won't be able to self-service persistent disks, network
configuration, or anything else at that level, which means that in practice,
there is considerable drag from interacting with other teams to get that stuff
provisioned. A radical commitment to self-service would be interesting to see.

 _But although technical management wasn’t an idea that lasted, it was a very
good place to be an engineer while it did._

One of the genius things i learned at Pivotal was that a 'team lead' and a
'manager' don't have to be the same person, and that neither of them has to be
a 'boss'. Every developer needs a 'team lead' (in Pivotal cant, 'anchor') to
lead the day-to-day operation of a development team, and a 'manager' to
channel feedback, guide career progression, ensure happiness, etc, but they
it's better if they're separate people, and it's better if they're drawn from
the ranks of the developer's peers (albeit the more senior ones), rather than
being some formally separate layer of the organisation. You should always have
technical leadership; you should usually have technical management too,
although it probably makes sense for the most senior technical-management
people to have non-technical managers themselves.

~~~
eikenberry
I don't think the "best tool for the job" necessitates needless polyglossia.
They obviously have a "best tool for the job" mentality to some extent as they
have 3 official/primary languages, not 1 like some places I've worked.

How many different languages/technologies are required for it to qualify as
polyglossia?

~~~
twic
One more than however many I personally know, of course :).

I suppose the only defensible definition of the term would be more than one
language, with some qualifier like "for any given context", because using C++
to serve your JavaScript app isn't polyglot. But then we just argue about how
big a context is - are API servers and systems of record the same context?

A more fruitful perspective might be to look at how the decision to adopt a
new language etc is made. If an individual developer can do it, it's a
potentially polyglot environment, whether that potential has been realised or
not. If it requires an organisation-wide process (whether CTO decree or
consensus at an all-hands meeting), it's unlikely to be a polyglot
environment.

This perspective is definitely a product of my own particular experiences,
though, and might not be generally relevant.

------
lstamour
Correction: (2015)

~~~
sotojuan
I swear I wrote 2015! Weird. Can a mod fix it?

~~~
sctb
Done.

~~~
toomuchtodo
You and dang work too hard around here, but its appreciated.

