
Continuous Delivery Sounds Great, but Will It Work Here? - msolujic
https://queue.acm.org/detail.cfm?id=3190610
======
jakevoytko
I work at a company mentioned in the article (Etsy). I've also worked in
environments with daily cuts; these feel similar in practice. Having the
ability to do continuous deployment is not the same as practicing continuous
deployment. You don't buy much if developers still yolo-push 6000 line commits
with fully baked features. You still need to foster a strong developer culture
that encourages small changes. This is true outside of continuous deployment.
But since you gain the option to deploy small and isolated changes to
production, the consequences are much different.

This has many implications. It needs to be OK to push back and ask engineers
to split large code commits into smaller units. Engineers need a strong
understanding of how to sequence changes: "I need to do the ORM changes, then
the API endpoint, then the controller change, and then I can do the frontend."
This means that senior engineers invest time in helping junior engineers
understand why this has to be done and why this particular sequence is safe.
Methods need to be devised to avoid holding up future pushes - for instance, a
culture of rolling back to safety at the first sign of trouble instead of
debugging in production.

Anecdotally, the best developers seem to always be pushing small changes, or
always have a few commits in the daily cut. Interpreting this is a bit of a
chicken and egg problem - is throughput higher because the changes are
smaller, or does high implementation volume just imply more changes? Or maybe
it's just a free form of advertisement: "I always see that X is pushing, so
they must be good"

~~~
klenwell
> Anecdotally, the best developers seem to always be pushing small changes

More formally, this is true, too. At least according to Donald G. Reinertsen,
the author of this book:

[https://www.goodreads.com/book/show/6278270-the-
principles-o...](https://www.goodreads.com/book/show/6278270-the-principles-
of-product-development-flow)

Batches and queues are primary themes of the book and Reinertsen waxes prosaic
on the extensive benefits of making batches smaller all around during product
development.

I've noticed the benefits mainly with user story size and pull request size
(which are generally correlated). Large batch sizes lead to longer delays
which lead to cut corners which lead to more defects which lead to further
delays. There are other underestimated penalties that Reinertsen discusses as
well.

Of course, too small and you're incurring disproportionate overhead with each
batch. The key, I suppose, is figuring out mama-bear batch sizes for your team
or organization and effectively paring down to them.

~~~
curun1r
There's a quote from Blaise Pascal that captures my feelings on small changes
fairly well:

 _" I have made this longer than usual because I have not had time to make it
shorter."_

Small changes mean front-loading a lot of the thought into breaking down the
problem. Large stories/changesets is often a sign of a vague requirement or
design up-front where the developer mostly figures out how to solve the
problem while writing the code. It's not really the size that's the problem,
it's the uncertainty. But the size is much easier to observe and use as a
forcing function for increased planning (i.e. large stories need to be broken
into smaller stories before a developer can start work.)

When you talk about the best developers being the ones that push small
changes, what I believe you're really saying is that the best developers are
the ones that are able to be given a large problem and break it down into its
constituent parts most easily and the ones that have the discipline to do that
in planning rather than while coding. Which, to me, is a lot less
controversial statement.

------
denzil_correa
One point missed in this entire calculation of "Continuous Delivery" of
systems is the estimated "Cost of a Mistake" [0]. There are many applications
where an small error in deployment could lead to a loss of a human life. Code
for safety critical systems requires formal checks before being deployed and
iterated [1]. There are indeed many many environments where continuous
delivery won't work for very good reasons.

[0]
[https://twitter.com/benhamner/status/832785333070606337](https://twitter.com/benhamner/status/832785333070606337)

[1]
[https://en.wikipedia.org/wiki/Static_program_analysis](https://en.wikipedia.org/wiki/Static_program_analysis)

~~~
icebraining
Continuous Delivery != Continuous Deployment. The first won't lead to any
cost. Also, why can't those formal checks be applied in a CD fashion?

~~~
fny
Treat every line of code you write as both a feature and a liability, then add
on a much, much higher cost to test/verify than cost to build. Like say you
need the feedback of 5 very busy people to get buy in for features and say you
also need to have 3 of those very busy people verify the output. Now add in a
third-party you need to pay to do security testing. So for any new feature you
add, you’re directly adding risk and increasing the cost of formal
verification.

With these constraints, even if you want to do CD, your CD will end up looking
exactly like a waterfall development model. It makes no financial sense
otherwise.

I work on predictive systems at a hospital, and this ends up being the case
for almost every deployment. The costs of making a change and verifying it or
adding features is so damn high that you need to think in 3-6 month deployment
cycles about everything.

~~~
SideburnsOfDoom
> say you also need to have 3 of those very busy people verify the output.

This assumes that this verification / testing has to be done by a person, and
cannot or should not rather be a person producing automation that can be run
repeatedly, as much as is possible.

Challenging this assumption is one of the points of Continuous Delivery.

yes, it's more expensive, slower and more painstaking when there are lives on
the line. But this applies regardless of it being automated or not. And
automation is more repeatable and scalable than human verification for most of
the tests that you want to make.

~~~
fny
We don't use human verification for the bulk of tests, just the last pass. In
the healthcare space, you absolutely need to have a few physicians review the
final results of a prediction model to make sure they "feel" consistent with
prior releases. This even applies for more trivial changes like switching from
a bar graph to a line graph. So for both legal and practical reasons, a few
physicians _will_ need to give a go ahead.

I'm sure you can imagine the legal reasons.

A lot of the practical reasons stem from fears that some unknown unknown might
trigger a failure that won't be caught by the quality metrics and upstream
tests. Anyone in software knows this happens a lot more often than we'd like
to admit. Given people will die if the software doesn't work and people will
be sued if people die, it would be both irresponsible and risky to depend
entirely on an automated test suite for the pre-deployment delivery. Then add
a requirement to have a third-party audit the code for HIPAA compliance and
security testing, and you can kiss your CD good bye.

Even if you want to act like you don't, you still have a milestone, a release
date, the evil poison that brings agile thinking to its knees.

------
mabbo
I once helped lead my team from a tedious manual process into full-CD. Took a
lot of pain to get there, and we had to build some new tools to automate all
of the steps we were doing, but we did it. Most of the pain was in realizing
we didn't have enough UI regression tests. Every 'incident' sparked a new set
of tests and over time the incidents stopped happening.

Our velocity increased, time-to-market dropped, operational workload fell. It
was great.

Sadly, after a few years of the joy of full-CD, more conservative leadership
took over our org. Now, all deployments of software must be accompanied by
paperwork (digital), step-by-step plans of how you're going to deploy this
software, approvals from senior managers, just general micromanagement. CD
isn't possible in such environments.

~~~
koolba
Fully automatic deploys are distinct from fully automated ones. I insist on
the latter but not everyone is comfortable with the former. Large / legacy /
lumbering institutions ( _think big banks_ ) in particular want a change
ticket, window or some other control before a release makes it's way to
production.

The middle ground I've found is to have fully automatic deploys in lower
environments and automate the entire process for production. But rather than
have releases flow there automatically, production is handled by pressing a
single "Deploy whatever is in UAT to prod" button (i.e. the "Big Red Easy
Button"). You then give access to that button to somebody else in the org (ex:
a UAT team, a release team, your manager) and everybody's happy. It also
conveniently gives you an excuse to never fill out production release
paperwork as " _It 's not my job, I'm not allowed to release to production!_"

~~~
hinkley
There’s a new group learning CD that mistakenly believes that automatic
deploys _are_ CD. Most of these came to the concept after it had supplanted
discussion about CI.

Automating a deploy is relatively straightforward but doesn’t provide much
value if that’s all you do. Local reproduction of build steps, metrics,
historical data, building out test infrastructure, all of this is important
for scaling up a team and keeping regressions low.

~~~
devonkim
I think there really needs to be a separate acronym from CD to better
distinguish deployment from delivery (similar to authn v. authz in security
nomenclature). Nobody with the right mindshare seems to have made this a
priority though because automated deployments are probably not cost-effective
in many organizations but delivering a change, however, is something that has
solid value everywhere. Coming up with a change ticket with the impact of a
change and what’s typically paperwork in many places absolutely has value and
will help people move faster with lower risk while making management feel like
they have control (because most continuous code based controls of delivery
mechanics tend to be more democratic in practices, not manager-oriented).
Maybe it could be CDD (continuous delivery description) vs CDE (continuous
delivery deployment)?

~~~
hinkley
I hold that on a well functioning team the mid level developers are delivering
most the the code. The senior people are dealing with clearing roadblocks and
cross cutting concerns like stability, security or performance. This is not a
blame the victim structure. It’s more like a pedestrian version of NASA’s
incident system.

To do this the process needs to be self service. You can find mid level people
pretty easily. Anyone who wants a promotion or become an SME should have tools
to do a lot of that work themselves instead of having to extract it from the
leads or follow the School of Hard Knocks.

Good automation supports this, bad automation makes this harder.

------
0xcde4c3db
I don't feel like the article really addresses "highly regulated
environments". To me, that means things like avionics, military equipment,
medical devices, and nuclear power plants (cf. the list of things buried in
most sufficiently-BigCo EULAs that you're not allowed to use the software
for). Are there good examples of CD-like practices being used in these
domains?

I'm also a bit disappointed at how generic a lot of the wording is. It seems
like you could do a search-and-replace of "continuous delivery" with almost
any other process improvement term (e.g. agile, kaizen, Six Sigma) and have
people nodding right along for 90% of it. Is there a concise explanation
somewhere that really makes clear how this is something more fundamental than
devops-flavored kaizen? Or am I expected to read the entire "canon" before I
understand it enough to evaluate it?

~~~
amarkov
Ensuring deployments are "predictable, routine affairs that can be performed
on demand at any time" isn't just a generic business process problem. It
imposes significant constraints on how code has to be written and tested.

------
js8
How does CD work with multiple vendors?

For example, I work in a company on a product that reads data from many
products of other companies. How do we tell our customers that we certified
the product with the certain version of the products that we read data from?
If vendors put change into the production continuously, how can we react to
those changes before the customer gets them?

------
Shoothe
Jez Humble's Lean Enterprise book was one of the best quality content I read.
Highly recommend!

