
Our Development Workflow - martindale
http://engineering.zenpayroll.com/this-is-how-we-zenpayroll-our-development-workflow/
======
kyberias
I feel it's slightly concerning that the same text contains:

"There’s no room for error."

and

"Someone on the product team will usually play with the feature on our staging
environment trying their best to break things."

The first sentence seems to imply that they try to maintain the highest
quality possible. Yet the second sentence does not really fit that idea. The
article pretty much only lists the tools they use.

But on a scale from Random PHP site to NASA space shuttle software, where is
zenpayroll located?

~~~
adamnemecek
One presumably talks about the production environment, the other about staging
environment.

~~~
superuser2
If you truly have "no room for error" (i.e. you are shipping code to a
hundred-billion-dollar space vehicle), you might have:

\- Very strict coding guidelines

\- Rigorous code review

\- An extremely thorough test suite

\- Multi-stage approval through a QA process involving many pairs of eyes and
thorough, formally defined and scientifically rigorous test procedures

\- Static analysis and possibly even algorithms proved correct in Coq

Simply playing with it in staging for a while and determining that everything
looks good is NOT the kind of testing that you do when there's "no room for
error."

~~~
adamnemecek
No room for error probably means one thing to people working on a $10^11 space
vehicle and something else to webdevs. Does that surprise you?

~~~
BillinghamJ
As a statement, it is absolute.

~~~
markc
As a statement, it is meaningless.

------
ktzar
Testing _after_ merging with develop is a very, very bad practise. And
there're plenty of tools to help you don't do that, just kick a build to run
all your tests, which will add a tick or a comment to the branch's PR. And
never merge anything without that tick or comment.

~~~
yegor256a
Totally agree. Testing should be done BEFORE merging, like this article
explains: [http://www.yegor256.com/2014/07/21/read-only-master-
branch.h...](http://www.yegor256.com/2014/07/21/read-only-master-branch.html)

------
michaelgrafl
Point 2 (Branch off development) confuses me.

At my workplace we have branches for dev, staging, and production. We're
working on feature branches. Now if my coworker merges feature A into dev, and
I merge feature B into dev, and feature A is done and ready for release, but
feature B needs more work, and my coworker checks out a new feature branch C
from dev, he'll be unable to merge into production without merging in my
unfinished feature B.

We branch off of production so this won't happen. Are we doing it wrong?

~~~
rorykoehein
Couldn't you merge the features that are done to your staging branch and from
there merge to production? Your staging branch would be the 'release branch'
in this model: [http://nvie.com/posts/a-successful-git-branching-
model/](http://nvie.com/posts/a-successful-git-branching-model/)

~~~
michaelgrafl
Thanks for the link.

We merge feature into dev when we want to show a feature to a fellow
developer.

We merge feature into staging when a feature is ready to be tested by non-
developers and to be included in documentation.

We merge feature into production when it's ready for all users.

Has been working out well so far.

~~~
securingsincity
We have a similar process it allows for more releases. imo there are things
that require more testing then others, and they shouldn't hold up the process
of releasing code.

------
TorKlingberg
This looks like a very sensible workflow in general. The one thing that
worries me is long lived feature branches. This workflow is fine as long as
features never take more than a few weeks. If it takes longer you end up with
bugs that have been fixed on the development branch still appearing on various
feature branches. Also if you do a lot of refactoring on a feature branch
while development goes on in parallel there will be difficult merges. It is
especially problematic when work on a feature stops for some reason and an old
branch sits around for months.

In general code should be merged to the development branch as soon as it is
working, tested and reviewed, even if not all the functionality of the feature
is done.

~~~
rimantas
Did you miss the part about rebasing?

~~~
TorKlingberg
It looks like rebasing is only done after the feature is complete. Maybe that
was just a simplification.

~~~
rimantas
Why does it matter? All the fixes will already be there after rebase anyway.

~~~
pionar
The point is that if you're not continually merging development down into your
feature branch, then rebasing becomes overly difficult. Merging can introduce
bugs if not done correctly.

At my company, we eschewed feature branches about a year ago. We have two
branches - development and main, with a branch ("tag" in git speak, we use
TFS) created for each release of our product (we're a software company, not a
service company). Development happens in the development branch, and is merged
up into the main branch after code freeze.

This does not preclude us from releasing often. With judicious use of feature
toggles, it's simple to release with a feature that's not done.

Feature branches would just create a mess for us as there's necessarily many
communicating parts between teams, and the number of feature branches required
to keep them all in sync would be exponential.

------
jiraaya
1\. "Don’t optimize for the short-term" vs "Keep it simple and
straightforward".

Often the simplest thing may not work in the long run. But its better to err
on the side of keeping things simple.

2\. Code reviews + off mainline development is a potential disaster in waiting
unless the review process is fast. The fastest review process I have seen is
automated testing and pairing when someone is working on something that cannot
be caught easily by automated tests like synchronization.

~~~
mtdewcmu
>> 1\. "Don’t optimize for the short-term" vs "Keep it simple and
straightforward".

>> Often the simplest thing may not work in the long run. But its better to
err on the side of keeping things simple.

IMO, the best reason for simplicity is that simple things can be torn out and
replaced more easily than complex things. So the simplest thing isn't
necessarily right, but you haven't wasted much resources by choosing it, so
you've left your options open. You always want to have options.

~~~
jiraaya
Ah, that's what I meant by "erring on the side of keeping things simple". It
is okay if you make decisions that are not optimal in the long run, but are
simple since they can be torn apart.

------
_mikz
Nothing special. It is just normal workflow when you use common sense, no?
Except the development branch because master is new development branch.

------
eitally
Reading things like this make me long for the simplistic development
environment of a web startup. Almost none of this would fly in an "Enterprise
IT" environment, except in the very rare case where the team is legitimately
building (and owning) a product rather than creating business process
optimization crutches.

------
securingsincity
Dedicated QA team? Person? Feels like they are missing a crucial piece of non-
devs testing and accepting features. Only my opinion but in my experience
developers have a different definition of done then non developers so having
both QA and even some user acceptance testing goes a long way.

~~~
securingsincity
Also this article is a year old. Can we get a title change? and maybe someone
from ZenPayroll can chime in how their process has changed, if it has?

------
athenot
And THIS is how you write a job posting. Explaining the setup and perhaps some
of the culture lets potential candidates know right away if that's an
environment matching their style. (And it doubles as an interesting article
too.)

Disclaimer: satisfied ZenPayroll user here.

------
dkarapetyan
This is the first time I've ever seen a write up of a development/deployment
process that makes any sense. Good on ZenPayroll for having their fundamentals
straight.

------
jorgeleo
This was an infomercial in the form of a blog... dissapointing

------
elwell
> The name of this test server is Leeroy, who does all this with the help of
> Jenkins.

------
ricknew
No usability testing?

~~~
acveilleux
Presumably this happens at the spec stage. At least it's implied.

------
dlss
Error 1008 for me.

