
AWS CloudFormation now supports blue/green deployments for Amazon ECS - unigiri
https://aws.amazon.com/about-aws/whats-new/2020/05/aws-cloudformation-now-supports-blue-green-deployments-for-amazon-ecs/
======
awsanswers
The cloudformation team is small bc the people who run it are not fun to work
with and not willing to bring on talent with their own opinions. It's a weird
irony, dug in and overall detrimental.

The only option teams that need CI/CD for their cloud resources & want to use
cloudformation for most of it is to have a side chain process for handling
resources that can't be expressed with Cloudformation. (not at all
insurmountable but shouldn't be necessary)

~~~
guitarbill
roughly how many is "small"? 10? 20?

~~~
awsanswers
Very small - and to be clear this is not supposed to happen at Amazon or AWS
it's "disagree and commit" vs "two pizza team".

It'll suss out eventually but not soon.

Start telling your teams to do what I said above and if anyone tells you "but
we want to use one tool" tell them they must grow out of that.

------
grubbypaw
I've never understood CloudFormation's lag time in supporting new service
functionality.

This launched nearly 3 years ago.
[https://aws.amazon.com/blogs/compute/bluegreen-
deployments-w...](https://aws.amazon.com/blogs/compute/bluegreen-deployments-
with-amazon-ecs/)

~~~
x3n0ph3n3
Because while "ready" for new features includes API support, it does not
include CloudFormation support. The problem is entirely AWS politics and
culture.

~~~
jcims
Who builds the cloudformation support? I could understand not baking it in on
an initial feature release but it should be there ’soon’.

Same with Config.

~~~
x3n0ph3n3
I've been told that each product team is supposed to add CloudFormation
support themselves. For some reason, it's not treated as a high priority and
often lands on the laps of the core CloudFormation team.

It may turn out that the tools that CloudFormation team provides don't make
integration easy, especially if the operation takes longer than 15 minutes
(meaning they can't implement support via a single lambda invocation as an
under-the-hood custom resource).

~~~
scarface74
You can also invoke a custom resource via SNS and then the SNS topic is
subscribed to an API endpoint.

But it’s hard to believe if the team responsible for the blue green deployment
functionality developed an API endpoint to do it, they couldn’t just hand it
to the CF team to call. At the end of the day that’s all CF does. Call APIs
based on the different lifestyle events as far as how it actually creates
resources.

~~~
jcims
>lifestyle events

Intentional or not, I'm going to start using this. :P

~~~
scarface74
Ughhh. And it’s outside of the edit window...

------
phamilton
It looks like this uses a new, not yet documented, top level attribute in a
template: Hook.

At least, I haven't found any documentation for it other than the template
they say to copy in the linked user guide for this feature. Definitely not
supported in CDK.

~~~
dkdk8283
What’s CDK in this context?

~~~
mcbain
[https://github.com/aws/aws-cdk/](https://github.com/aws/aws-cdk/)

------
ciguy
The ECS default deployment model is quite frankly a disaster. I've written
many Python/Go scripts over the years to wrangle it into a sensible form for
CI/CD.

Good to see that they're working on it, but I don't know why they don't fix
the underlying paradigm instead of making it Cloudformation exclusive.

~~~
vazamb
I would love to know what the problem is. We do dozen of deployments every
week with a ALB + ECS + Fargate setup. We upload a new container image, create
a new task and launch as many tasks as desired (so if we want 2 containers
running we launch 2, for a total of 4). ALB calls the /health endpoints on the
new containers and if they pass the healthchecks it drains connections to the
old containers and stops the tasks. This has worked seamlessly for a long time
now without any downtime during deployments.

EDIT: I should mentioned that we are using AWS CDK for all of this. All it
does is register a new task as the default task for a service and ECS/ALB does
the rest.

~~~
WatchDog
I find ECS particularly, ECS on EC2, to be really painful for small
deployments.

I just want a cluster that scales in and out to the amount of memory my tasks
need, I'm not very concerned about CPU. Until capacity providers it was more
or less impossible to do so without having a bunch of excess capacity
provisioned. Even with capacity providers, I can't seem to get a cluster to
scale in to 0 instances when no tasks are needed.

I have one ECS service that requires an EBS volume mount, which means that
when the task definition is updated, I need the service to stop, so that the
new task can mount the same volume. This deployment model is essentially
impossible without implementing a custom deployment strategy.

Fargate makes things a bit easier, and now that you can use EFS volumes with
it it might make more sense. But overall, everything in ECS just seems poorly
designed, clunky and hacky, like many AWS products.

~~~
friend-monoid
You can’t do EFS on Fargate in CloudFormation yet though.

~~~
scarface74
True. But you can do it with a custom resource.

[https://gist.github.com/guillaumesmo/4782e26500a3ac768888daa...](https://gist.github.com/guillaumesmo/4782e26500a3ac768888daab3c55b139)

------
laminatedsmore
The list of restrictions in the user guide seems painful - cannot be used in
the same template as nested stacks, cannot start a blue green deploy if other
infrastructure would be modified at the same time, cannot import properties
from other stacks, cannot export output properties. I was interested, but
can't imagine using it with these restrictions.

~~~
laminatedsmore
Thinking about this further, the benefit of these restrictions is in avoiding
conflicts where dependant infrastructure is updated in place before the blue
green has completed. Lack of inputs and outputs still seems killer though.

------
tbrock
I don’t think I’ve ever met someone that actually uses cloudformation unless a
template was provided them to setup something like a lambda.

Why would you when terraform exists?

~~~
devonkim
Terraform doesn’t support rollbacks (handy for application roll-outs) and if
your shop is heavily invested in AWS CF is a perfectly fine tool. I’ve
maintained and started up tens of thousands of lines of both TF and CF and
they both have their strengths and weaknesses.

~~~
tbrock
Sure it does? Revert your code and apply. It’s not atomic but I’m guessing
neither is CF.

I’m not sure I’d use either of these tools to roll out new code though.

~~~
devonkim
That’s more of a GitOps style intervention requiring a source change and the
workflow to revert a change could certainly be done but is by design not a
first class construct in Terraform providers (not every provider supports
every feature such as importing of resources). To respond to a Cloudwatch
(heck, Prometheus, Grafana, ELK, etc) alarm saying your error rate went up
because of a Route53 change it’s not out of the box with Terraform (would be a
custom providers or null resources probably). And as a CLI application
(granted, it’s run more like an RPC style architecture) there’s no obvious way
to signal different failure levels and to respond to different failure modes
of different resources.

CF roll-backs are on by default and will revert changes. Sometimes it can fail
and be a real pain, but it’s overall been more of a help for myself than harm.

------
tilolebo
I've been wondering: doesn't CDK depends on CloudFormation, as it basically
converts code to CF files? (similar to Troposphere but with more languages
supported).

If that's the case,I don't see how CDK stands a chance against Terraform or
Pulumi.

And I'm not talking about multi cloud support, but just about the fact that an
AWS-managed product, CloudFormation, keeps lagging behind for YEARS for such a
core feature of a core AWS service.

------
divbzero
I’m almost surprised this welcome feature was implemented at all, as ECS
development appears to have slowed in favor of EKS (Kubernetes).

Also too bad that it’s still CloudFormation.

