
A Tale of Two Onboardings - joeyespo
https://convox.com/blog/a-tale-of-two-onboardings/
======
adrianmacneil
Excited to get the app running, and following the brief instructions of his
coworkers, Joe types:

docker-compose up

His rails bootcamp didn't teach them anything about docker. It turns out
docker-compose isn't installed. Not wanting to sound like an idiot, he googles
around, and spends 3 hours trying to understand what a "container" is. Many
tutorials online mention using boot2docker, but some new people seem to be
using docker-machine. He asks Susan which one to use. She sighs, and mentions
that the team is split between them.

Eventually he chooses docker-machine, and boots the app up. Now it's unclear
where the app is running. Eventually he figures out that he has to run
"docker-machine ip default" every time the docker-machine VM starts, so that
he knows the IP to visit the development site at something obscure like
[http://192.168.99.100:5000/](http://192.168.99.100:5000/).

Finally, he makes a change to the code, and refreshes the app page. Nothing
happens, because it turns out that his local filesystem is not synced with
docker, which is linked through to docker-machine, which is running inside a
VM on a virtualbox host.

Don't get me wrong, containers are great. But this sounds like more of an
argument for good "getting started" documentation, and helpful coworkers,
rather than proof that containers are any more simple to get up and running
than your average rails app.

~~~
angersock
_But this sounds like more of an argument for good "getting started"
documentation, and helpful coworkers, rather than proof that containers are
any more simple to get up and running than your average rails app._

This bears repeating: in the second scenario, the new hire had somebody tell
them what commands to run. Without that information, they would've been stuck
Googling their way to success too.

Also, I can't help but be amused at how annoying that homebrew nonsense is--
aptitude on Debian is just a hell of a lot simpler.

~~~
lyime
What's "non-sense" about it?

------
avitzurel
I completely disagree with the assumption that Docker will make life simpler
here.

First, question is, what are you optimizing? Are you optimizing on-boarding
for new engineers? How many new engineers do you get a month?

Let's say you get 2 new engineers a month. Getting the Docker up and running,
setting it up so new engineers can use it easily is also cost, that also takes
time.

I don't think that evens out until you really need to set up 10-12 new
engineers every month.

This post also skips the part where boot-2-docker fails to load and you need
to restart VirtualMachine and a lot of other problems with Docker that you
will encounter LONG after the initial `bundle install` finished and everything
is set up on a single machine.

While back we had the same problem and we used the same scripts we run in
production only on Vagrant and we distribute that image

[http://avi.io/blog/2014/12/26/ruby-and-rails-development-
env...](http://avi.io/blog/2014/12/26/ruby-and-rails-development-environment-
using-chef-and-vagrant/)

Docker has a lot of advantages, however, I have yet to see a single company
that stuck with the initial promise of a reproducible dev environment on
Docker (Please correct me if I'm wrong here).

Bumping into the same situation I would select one of the following

1\. Document with all the steps (Wiki) 2\. Chef (same as runs in production)
on top of Vagrant

I would live Docker completely out of this, it will introduce more complexity
over time than the initial setup process.

Nothing will replace good docs and great onboarding culture in my book

~~~
nraynaud
I would tend to optimize onboarding up to a point, mainly to try to impress
new engineers. But also maybe because if something hard happens, and you need
to bring an external consultant in emergency that could be useful. But mostly
to impress new hires.

------
calpaterson
What a shame. I hoped this would have something insightful to say about how
terrible it is to be a new joiner on a development team but ultimately it's
just proposing a technological solution to a people problem...

What does Janelle do when she makes her first change for her first story and
something doesn't work right?

~~~
nzoschke
Convox engineer here. I do agree this is ultimately a people problem.

However I think there is a large technology aspect to it also.

I often think about how Rails changed problems like this.

Before Rails we were all writing bespoke PHP websites. I remember it taking
ages to get up to speed with code bases on a new job.

After Rails we shared a lot of common language and understanding. I have a
good idea of how to run tests and where to put changes. I also have a universe
of knowledge to tap into on Stack Overflow.

In a world with more shared conventions beyond Rails, Janelle has more than
her direct team to help her with problems.

She has tools that help her pinpoint the problem. For example, docker-compose
offers really understandable error messages around missing environment
variables, vs an app silently breaking.

She also has a community that can also help her with questions. Just today I
found an excellent Docker / Django quickstart guide that has hidden away a ton
of minutia about getting a Django project running on my computer.

------
exelius
I guarantee Joe understands a lot more about his application than Janelle
does.

I'm all for hiding complexity behind abstraction layers, but it's a really
useful exercise for a developer to be given an app and told "figure out how to
make it work, but feel free to ask any questions you have". It causes you to
really think about how the application works, and the process of figuring it
out yourself is FAR more instructive than any documentation would ever be.

Not to say you shouldn't have documentation (because of course you should),
but when you have to figure something out yourself, it sticks in your memory a
lot better.

~~~
MrTonyD
Remind me never to work for you.

~~~
exelius
I trust my teams to manage themselves - and part of that is learning for
yourself. Engineers learn through problem solving.

~~~
MrTonyD
There is literally decades of research showing that it is possible to learn
things faster and better with appropriate teaching materials. That is why we
have Universities and schools to teach people - and don't just ask people to
figure out everything for themselves (ie. high fidelity simulations were found
to be less educational - since their high fidelity proved to be a distraction
from building the important conceptual models.)

So, while I can and have figured out complex systems on my own, it would
really be much more helpful to have someone or something to help create
correct mental models at the beginning (and again, there are decades of
educational research showing that there is a cognitive cost to building
incorrect models of systems. That cognitive cost may mean nothing to my
manager - but that is exactly why I wouldn't want to work for such a person.)

~~~
exelius
I'm not saying we teach someone theoretical physics by saying "dive in!" But
you're talking about things that are relatively static.

I'm not going to spend a sprint building perfect mental models of an
application because if your app is anything like the ones that I've built,
they change significantly over the course of a year. We rip out services and
and abstract them behind new API layers _all the time_. Legacy interfaces are
provided of course, but the way things work internally is constantly a work in
progress.

Now, just because it's a good idea for a developer to learn how the system
works _today_ when they first start doesn't mean it's a good idea to carry
that complexity forward to your production servers. So if the day comes when
someone containerizes these applications with an actual business benefit, then
let's containerize the fuck out of them. But I'm not taking on a bunch of tech
debt (that will incur more tech debt as the application changes over time)
just to make onboarding easier - when making onboarding push-button isn't
necessarily accomplishing the goal of onboarding because the developer is
going to have to learn those lessons sooner or later.

------
sdoering
No Docs, no onboarding. No one helping and showing the new hire around
environment-wise. What does this tell me about corporate-culture?

Yes. Not to work anywhere near there. Tech does not fix people or broken
culture.

~~~
vinceguidry
Seriously. Is it so hard to maintain a document that outlines the steps needed
to get a standard OSX or Ubuntu install to the point where you can hack on an
app?

I have to do in my personal projects often enough that I maintain such a
document purely for my own reference, because I forget steps even for my own
projects.

~~~
zaphar
More importantly, Why was the new dev given a laptop without the crucial
pieces already installed. Janelle shouldn't have had to install docker and Joe
shouldn't have had to install bundler or postgres or any of the other things.

Both companies had broken onboarding processes. Docker just happened to make
the effects of the broken process less annoying for Janelle.

~~~
vinceguidry
Personally, I feel like it's the app that's badly configured if it won't work
with Postgres having been installed with either homebrew or the far-better
PostgresApp. Backing services should just work with the default install on
development. If it needs special configuration, like enabling Hstore on
Postgres, it needs to be documented, but honestly I consider it a code smell.
That stuff should be auto-configured by the app.

------
bmulholland
Everything else aside, it really feels like this article purposely stretches
out the first example and compresses the second:

* Joe gets told to work on his own, and only briefly consults a co-worker once. Janelle sits down with a senior dev and works through it together. How much of Joe's work could have been solved by sitting down with a developer?

* Every single command Joe writes out is included in the article, while docker steps (potentially more than one) are summarised with a sentence, e.g. "she just needs Docker on her system."

* Every time Joe has a problem, he has to figure it out on his own, google it, etc. Janelle just gets things pointed out to her by Rebecca.

Seems like a pretty biased comparison, even if the point is valid.

------
geuis
I started working with a company a little over a month ago. They "kind of"
containerized the different parts of their stack. They don't use the
dockerized versions in production. I have been the only engineer actually
trying to make this work. Its been a nightmare.

Getting setup: They have about 15 parts of the application stack. When trying
to get everything up and running, each layer was anywhere from 15mb to 100mb
and so we're talking several gb of Docker layers just to get the basic stack
in place. Then the configurations were all over the place and nothing was
quite documented right.

Then it turns out there is not any clearly defined, standard way to do local
development out of Dockerized applications. So I end up having to run the
parts of the site I'm working on locally, and then had to configure them extra
to talk to the docker-hosted parts.

So now I'm working with this hodgepodge setup that mostly works, except when
it doesn't. On 2 Fridays in the last month, oddly 2 weeks apart, my Docker
environment has just eaten itself and I've been forced to rebuild it. The
first time some space requirement got hit and the images got corrupted. Then
last Friday virtualbox was having some freakout and started pegging at 100%
cpu usage before again, everything got corrupted. Now its Monday and I'm
trying to put this back together again a 3rd time.

I _want_ this to work but its just been the most frustrating onboarding
experience of any company I've worked at.

------
ska

        Joe asks if there are any new developer docs.  “No, unfortunately,” says .... 
    

This is the real problem. Arguing about mitigation strategies is less
important than planning for onboarding.

~~~
gamesbrainiac
This is exactly what I meant. Most companies don't care about providing good
documentation, and so when new people join the company and don't know what's
going on they blame the platform or the package manager or something else.

There aren't very many good alternatives to writing good documentation.

------
debacle
The straw man here is that you can use containers poorly and do automated
builds without containers.

In the end, the first 1-2 days of stumbling are a rounding error on a
programmers overall productivity and getting the environment set up is only a
small piece of that.

The real take away is that you need to develop in an environment that's as
similar to production as possible. Virtualization, containers or no, is great
for that.

~~~
nzoschke
My personal experience... The first 1-2 days are a rounding error. But it's
very common for all internal tooling to be undocumented and fragile. This is a
cost a team will pay every day in developer efficiency.

This is a people and team problem.

A sign of a team that cares about this is when the time to first deploy for an
engineer is fast and safe.

~~~
debacle
Definitely. I'm a huge proponent of easy builds. The six month ramp up time
for new hires doesn't exist in 2015.

------
bradhe
1\. Use foreman

2\. This doesn't cover the other problems you'll have down the line with the
development process under docker.

3\. If you're having that level of difficulty getting a rails app up-and-
running you can't be expected to be productive on your first day anyway.

~~~
nzoschke
Convox team-member here...

Convox is a project by the creator of Foreman, David Dollar.

In fact the entire Convox team is ex-Heroku folks building a new suite of
tools that make local development and deployment awesome.

------
bm1362
I'm not convinced there is a solution to the problem. We will always struggle
to bootstrap our environment, this is a symptom of trying to reproduce
production environments on different targets. In the past I've been on a team
that has tried:

* Invested a lot of work getting a set of services to run on our dev desktops (a 'onebox') and it was very painful. Lots of useless configuration, adding new services is hard and it doesn't scale well. Some external dependencies can't be run locally. * Using a 'integration test' stack that you could hit from your desktop, but it was stateful and shared by 10 developers -- eventually causing us to have to block each other. * We actually built a simulation to run our tests in, it simply replaced the networking layer of our external clients with a in-memory queue; it introduced more bugs because of implicit ordering and subtle differences from production.

Production is the only easy environment, because the configuration and code is
immutable there. Testing and development are too nuanced, stateful and often
require lots of mutation within individual binaries/services.

From all my experiences so far, just running a test stack of services was the
easiest. SSH tunnels were important too, for things like service discovery.

------
mpdehaan2
This is a case for automation or documentation, not specifically docker. Could
use _anything_ including a shell script to get this env setup properly.
Picking docker means they also have to understand how to manage developing
within a docker lifecycle.

Even a wiki page would be sufficient as in developer envs, some understanding
of what is being done and why is generally useful.

------
zinssmeister
If you anticipate a architecture with many micro-services, sure docker might
be a good win for you. But my own experience was a win-loss scenario so I'm
now back to keeping things traditional (for now). Docker made it easier to
start the environment with a single command but it failed in other areas and
it was a pain to maintain.

~~~
imissmyjuno
What areas did it fail in?

------
jakobegger
Reading this makes me greatful that I develop a desktop app.

Here's how I set up my development environment on a new machine:

1) Install XCode

2) Check out source code

3) Double click project file

4) click "Run"

I'm really happy I don't need to set up half a dozen servers before I can get
started. I always thought web dev was simpler than native development, but
modern web development is extremely complex.

------
mcherm
I'm not sure why others are unimpressed with this article -- I thought it had
one simple point to make. And my project has precisely this problem: people
spend lots of time getting their environments set up and it is not
reproducable.

Obviously, my project needs to be fixed. This article points out that Docker
is one way of doing that.

~~~
vinceguidry
Part of being a Ruby dev is learning how to quickly onboard. In the same way
that as a C++ dev you need to learn your way around the output of configure
and make, and be able to quickly figure out what error messages are telling
you, you need to have that same baked-in knowledge of 1) pull the repo, 2) get
the right version of Ruby installed 3) run bundle 4) install missing
dependencies as bundler complains about them 5) run rails server and install
backing services / configure environment variables as needed. 6) run tests and
get them all passing.

If you don't already have this knowledge than you're working with an
incomplete set of tools and need to acquire them. The good thing is, if you're
sharp, you'll pick up the general gist of things pretty fast. That's why I
think nobody's yet written a guide encapsulating this knowledge.

Docker is not the answer to the problem, it adds complexity rather than takes
it away. You still have to know the ordinary way things work in case Docker
breaks. I guess someone does need to write that guide, given how many people
have issues with it, and the hopefulness evident in their pleas to just move
to Docker already.

------
Smushman
The problems described in this article really do hit home. I offer that
anybody who sees the Joe onboarding as anything other than typical may not
have been in the field long enough.

These problems have been happening since the 1990's and very little if any
progress has been made in these areas.

If it is that prevalent, I also would feel safe to surmise that it has little
to do with the other detractors posted here (ie it's over-engineered, needs
documentation, it makes Joe a better engineer, etc.).

Coders like to code productively first and foremost, not spend their time
documenting and designing processes and procedures. I say don't fight their
instincts. Let them do that!

This is where DevOps comes in to play, and should be the focal point of
resolution for this and related issues; letting the coders do what they
already do best.

------
robbies
I expected this to be more of a story about actual onboarding interactions
instead of a product pitch, though I certainly understand the motivation for
the latter.

In every company I've started at, I try to surf through process documentation,
or if there isn't any 'newbie' docs, then I keep a little live-blog of my
experiences, with the idea that I can share those later. The cost of such a
blog is minimal, and it helps the new hire recall parts of the process they
might quickly lose track of. Additionally, with some editing work, the blog
serves the greater purpose of: 1) Informing the team about the process 2)
Preparing the next new-hire.

------
idlewords
This just sounds like a vastly overengineered product.

------
memracom
The important difference here is not which bits of technology are in use, but
that YouGram has some shared standards and all the team members are using the
same toolset.

Also, everyone has issues with documentation and one good way to limit the
damage is to write build scripts or vagrant/chef build files or Ansible
playbooks or Dockerfiles. The principles are the important thing.

------
aggieben
As someone who has been onboarded with docker like in the example, I can
testify that this article is absurd. Docker _increases_ complexity, not
decreases. So much so that I question its value in most circumstances, and I
will _never_ on-board someone by throwing a docker image at them.

------
hardwaresofton
While I might be missing the point of the article, and this may indeed be a
people problem rather than one solvable with software, maybe a higher-level
(one that is above rails, and above bundler, in terms of generic use) build
tool could be of help here?

I could imagine a command like `make fresh` performing these commands,
noticing that things were missing, and informing the new dev of which way to
go?

Of course, documentation is the easier/possibly better solution to this
problem, but It's a lot easier to standardize with less things to standardize
on, and rather than trying to get every team to write good starting
documentation, maybe making a mandate like "your repo must have support a make
target called 'fresh' that will take a user from 0-setup to a fully setup
environment, or print a useful error"

[EDIT] - Also, some thoughts:

-Is some of this pain actually good for Joe/Janelle? Could some of these issues be constrained as learning opportunities which might benefit them long-term (i.e. have a problem with bundler, find out about how bundler works)?

-It seems like many (good?) engineering organizations with lazy (the good kind) developers will have solved this problem -- whether it's from the new dev wanting to save future devs trouble, or from the older devs wanting to save themselves time with env questions.

-Docker is great, but I feel like getting the solution without experiencing the problem can hinder technical understanding. Docker compose is a breath of fresh air BECAUSE you have an easier way to boot, manage, and isolate related (in that they are all related to your app) processes. I would fear that varying levels of understanding would be allowed to just get by with `docker-compose up`

------
ori_b
TL;DR: One company ships with a fully configured virtual machine. The other
ships with a machine that needs to be set up.

The lesson seems to me that you should be provisioning your systems with an
environment that's ready to handle the code.

------
desireco42
And then there is Otto.

In more serious tone, what was the point of this post? I don't think
developers struggle as much with bundler as they might struggle with weird
codebases.

~~~
nzoschke
Convox engineer here.

This post dances around the point. There is lots of room for improvement in
getting new developers up and running with a code base.

Convox and Otto share similar goals of one command to start and application
and one command to deploy it.

The differ in that Convox is a thin AWS layer. Deploying to Convox configures
EC2 Container Service, ELBs, and Kinesis Stream to run your dockerized
application.

------
joshbuddy
Is it fair to presume the docker person is going to be waiting a lot more for
tests to run due to poor io performance?

~~~
nzoschke
That is not a fair presumption.

If you're on a Linux laptop a Dockerized app has effectively no performance
decrease.

If you're on a Mac, a VM and Docker does impact raw IO. My anecdotal
experience developing like this every day is that it does not cause noticeable
waiting.

There are other downsides to Docker-based development, around tool complexity
and stability. In my opinion this is no worse than tools like Homebrew or
rbenv.

There are also productivity upsides.

