
AWS Copilot - bdcravens
https://aws.amazon.com/blogs/containers/introducing-aws-copilot/
======
noisy_boy
Not directly related to copilot but in context of ever increasing yml/json
configurations for cloud apps, am I the only one who feels overwhelmed with
the configuration options? I see tons of articles with just drop this json
here and that yml there and run this cli command - where is the reference for
these json/ymls? Does everyone fully know/understands the structure/options?
How do people keep track of the changes to that between upgrades? I feel
increasingly dumb the more I see these configs.

~~~
realharo
I feel like trying to use a "dumb" declarative config language for everything
is part of the problem.

The domain is not simple enough for that, so you end up with all kinds of
weird scripts around it all, or other templating/codegen solutions, plus all
kinds of ad-hoc conventions that are extremely easy to break and very annoying
to navigate.

Having a full blown statically typed language like TypeScript - where things
can reference each other strongly (not just by names and other strings),
naming conventions can be encoded in functions (or even enforced by types),
and structures on _every_ level (even your project-specific ones) have a well-
defined and checked schema - would help a lot in my opinion.

Maybe deno could find a use in this niche.

~~~
scrollaway
You're (accidentally, I think?) describing Pulumi in striking detail.

[https://www.pulumi.com/](https://www.pulumi.com/)

~~~
realharo
Kind of. From a brief look, it looks very framework-y and all-or-nothing
(though I could be wrong, they do have a page about gradual adoption at least
[https://www.pulumi.com/docs/guides/adopting/](https://www.pulumi.com/docs/guides/adopting/)).

I was picturing something more low level and less opinionated, though I can
only guess how such a thing would perform in practice.

Not even necessarily a specific product, just the adoption of workflows that
use already existing tools (e.g. the various API client libraries - for AWS,
Kubernetes, git, docker registry, etc.) to achieve this. I realize that that
could lead to reinventing the wheel, and eventually an in-house framework
anyway, but it feels a lot more comfortable to start there, as you can design
things closer to your exact needs.

Pulumi definitely looks interesting though.

------
justincormack
Note also that the Docker integration into Fargate was released at the same
time, see [https://aws.amazon.com/blogs/containers/aws-docker-
collabora...](https://aws.amazon.com/blogs/containers/aws-docker-collaborate-
simplify-developer-experience/) and [https://www.docker.com/blog/from-docker-
straight-to-aws/](https://www.docker.com/blog/from-docker-straight-to-aws/) \-
this provides a simpler way of using Fargate, especially if you are already
using Docker Compose. You can output cloudformation from this as well, but you
don't have to understand the internals of ECS to use it. This is in beta and
would love more feedback (here [https://github.com/docker/ecs-
plugin);](https://github.com/docker/ecs-plugin\);) I worked on this at Docker
with Amazon.

~~~
gavinray
My goodness, Docker Compose file to production stack?! My dreams have come
true.

The arguments for k8s I have is that the local and production setups are
identical.

It's a PITA to run k8s locally, but the alternative is that you use Docker
Compose local, and then deploy to k8s, which means two sets of configs. And
that's terrible.

Or, you use Docker Compose locally, and fragment all of the containers into
individual ECS/Fargate services which you write the deploy pipeline for
individually.

These were your best options, for one glaring reason: You can't use a docker-
compose.yaml in production. (I know, I know, Kompose, but it doesn't always
work quite right and you now have a k8s cluster).

Really excited to try this out, thanks for the work on it!

------
gavinray
I happened to catch this on the Github daily trending list a week or so back,
really cool.

For similar tools (though this one is ECS + Fargate), also see:

[https://somanymachines.com/fargate](https://somanymachines.com/fargate)

[https://github.com/awslabs/fargatecli](https://github.com/awslabs/fargatecli)

I'm pretty particular to Pulumi, and defining infra in typed programming
languages with full IDE tooling. AWS CDK is the AWS-centric equivalent, and
that's a great and under-represented tool as well IMO.

[https://www.pulumi.com/containers/](https://www.pulumi.com/containers/)

[https://docs.aws.amazon.com/cdk/latest/guide/home.html](https://docs.aws.amazon.com/cdk/latest/guide/home.html)

My issue with ECS/Fargate is that on the surface, it seems amazing. And so you
deploy your container, and that's relatively easy.

But now you have just a container on HTTP, on a bare IP. You probably want
HTTPS and a domain name attached.

So then you learn that you need to:

\- Create a Route53 hosted zone for the domain

\- Use AWS Cert Manager to provision certificates for the domain

\- Create a load balancer, and figure out how to attach it to the ECS/Fargate
service

\- Wire up the LB address to the record set and configure the termination for
HTTPS

And to do something like change an environment variable, you have to create an
entirely new "task definition", deploy the task revision, and then spin up the
new service and stop the old one. This takes about 5 not-particularly-easy to
find or navigate screens in the AWS console.

This is what it ended up looking like, for me to deploy a Fargate service:

[https://gist.github.com/GavinRay97/fbedfb18fe852bf625451a417...](https://gist.github.com/GavinRay97/fbedfb18fe852bf625451a4175695ad4#file-
hasura-fargate-https-alb-route53-cert-pulumi-ts)

It's great that these tools can help reduce the burden of doing this but man
is it a painful experience if you're not a wizard at DevOps.

Google Cloud Run is 1-2 bash commands to deploy a containerized app on HTTPS.

~~~
kohidavez
Copilot can also help set up HTTPS/ACM Certs/Subdomains:

[https://github.com/aws/copilot-
cli/wiki/Applications#additio...](https://github.com/aws/copilot-
cli/wiki/Applications#additional-app-configurations)

I love the cloudrun style service too - but one thing I think is cool about
running on ECS/Fargate is that once you outgrow the constraints of CloudRun
style services - you can drop down to the more granular infrastructure to make
tweaks and adjustments. Just my 2c :)

~~~
gavinray
> Copilot can also help set up HTTPS/ACM Certs/Subdomains

Oh yeah for sure! I was really excited about this for that reason when I found
it because of recent Fargate pains.

The irrational part of me just wishes that the "magic" from tools that AWS
publishes like Copilot, Cloud Development Kit, Fargate CLI, etc could abstract
it away in more of my interactions with AWS ;^)

> once you outgrow the constraints of CloudRun style services - you can drop
> down to the more granular infrastructure to make tweaks and adjustments.

Definitely, you're trading convenience for flexibility with Cloud Run vs
Fargate. Cloud Run also doesn't support websocket traffic yet ( RIP =/ ).

I'm a massive fanboy of Serverless Containers and think they're the best thing
since sliced bread. I run services on both AWS and GCP, and think Cloud Run
and Fargate are both solid.

My takeaway with AWS I guess is: Don't interact with provisioning/managing
infra raw, use the CDK or other tooling you publish haha.

~~~
phamilton
> Don't interact with provisioning/managing infra raw, use the CDK or other
> tooling you publish haha.

I've moved pretty much all my sandbox exploration to CDK. A one-shot destroy
when I'm done to clean everything up is great. I also appreciate the
documented types in Typescript. But beyond that, I've found that there's
really no easy way to manage permissions manually. Doing `
bucket.grantRead(service.taskDefinition.taskRole);` is so much simpler than
any of the policy generators in the aws console.

------
kohidavez
Hi!! I’m a developer on AWS Copilot! We’d love your feedback!

Homebrew: brew install aws/tap/copilot-cli

Docs: [https://aws.github.io/copilot-cli/](https://aws.github.io/copilot-cli/)

Feedback & Requests: [https://github.com/aws/copilot-
cli/issues/new](https://github.com/aws/copilot-cli/issues/new)

~~~
emmelaich
It's a pity you used "copilot", because that name is already used for Red
Hat's admin product.

~~~
alexktz
Do you mean cockpit?

~~~
emmelaich
Oops yes. But there is also Performance Co-Pilot (pcp)

------
drej
I probably missed it in the blogpost and github readme, but is there a good
way to "export" this into a CloudFormation/Terraform template? After all, this
seems to spawn a few things (networking, mostly), so I want to make sure I can
automate any further deployments.

Thanks!

~~~
jeremya
Copilot generates CloudFormation templates based on the Copilot-specific
config files. There’s a place where you can add your own additional resources
in CloudFormation syntax IIRC. It also has the ability to create a
CodePipeline for app deployment. That creation is also done by generating a
CloudFormation template and deploying it for the user. I’m not sure why it’s
not clearly documented that this is happening, but it is a selling point to me
because if I want to, I can “eject” from Copilot and just maintain/enhance the
CloudFormation templates.

~~~
Aeolun
I have a hard time believing anyone would consider ejecting to CloudFormation
a selling point.

~~~
llarsson
Could be nice to version control them. Some CLI stuff looks cool on a blog,
but do you want to remember what commands you typed, and in what order, to
bring up what production looked like two weeks ago manually?

------
vazamb
What are the advantages of this over CDK? It's trivially easy to spin up a
loadbalanced service [https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-
cdk_aws...](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ecs-
patterns.ApplicationLoadBalancedFargateService.html)

~~~
fulafel
CDK is terrible on so many levels, it's possible that this is less so.

~~~
fulafel
Expanding failures of CDK due to popular demand:

\- It introduces an edit-test-debug cycle from hell, it doesn't do what you
think from the mental model of a user-friendly CF template generator library
(doesn't statically ensure stuff it emits actually works even superficially)

\- It builds upon an already difficult and leaky abstraction (CloudFormation)
so you just get additional failure modes and cognitive burden

\- Glacially slow iteration, can easily take 10-30 mins to teardown/setup your
app due to CloudFormation.

\- fails to provide upsides from the costly tradeoff it makes to jump into
turing complete programming languages, like 3rd party cdk components or
iterating expressions at a repl

\- don't even get me started on the npm dependency hell (you'll randomly be
debugging mystic npm warnings already on the next day when installing a new
submodule of cdk deps, when version mismatches appear), the other-languages-
to-npm bridges, the "you can't use this without an IntelliSense style IDE" API
design, buggy cli tools, etc

To summarize it doesn't manage complexity or give you a good abstraction.
Probably because it tries to take on all of AWS functionality at once.

(Also the GP linked Fargate example is far from the functionality from the
copilot example, like enabling you to actually dev and deploy the app - ecr,
codebuild, codepipeline etc)

------
throw03172019
This reminds me of Convox. Very simple to get started and runs on ECS. Copilot
has less features as of now but looks like it’s going in a good direction.

(Happy convox customer)

------
twillin
This really seems like it could be a future replacement for Elastic Beanstalk.
Especially when they finalize the ability to provision storage infrastructure.

~~~
judge2020
Hopefully. Elastic Beanstalk is a product that "works" and is "maintained",
but when it fails it crashes hard. At least for the PHP environment, updating
the environment variable configuration can take 2+ minutes, and non-immutable
app deployments with 2 hosts can take over 10 minutes to pass health checks.
And finally, if it fails to do something (eg. a deploy command fails on the
instance), it will be stuck in the "error" state for 5-15 minutes where you
can't do anything to the environment and must wait for it to clear to fix your
bad deploy.

~~~
freeqaz
Amen I thought I was just doing something wrong! What is the best alternative
to Elastic Beanstalk on AWS? Is Kubernetes overkill?

------
jaxxstorm
I wrote a blog post about this that did the rounds last year about the topic I
believe Copilot is trying to solve:

[https://leebriggs.co.uk/blog/2019/04/13/the-fargate-
illusion...](https://leebriggs.co.uk/blog/2019/04/13/the-fargate-
illusion.html)

Summary: the marketing spiel about how easy fargate is to use and get up and
running isn't actually how the experience is. I'm hoping Copilot improves that
experience.

~~~
dunk010
Really great article. I've walked the same path and thought the same thoughts,
and always felt somehow that it was just me. Here's hoping that Copilot lowers
the barrier for all developers - especially ones who just have a small site or
hobby project.

------
ookblah
having tried everything from convox to fargatecli, deployfish ,etc. in the end
i just ended up using pulumi to set up the basic infrastructure components and
writing my own bash scripts to deploy tasks and services to ECS using the AWS
cli.

there was way too much magic going on and i wasn't confident i could easily
roll back or do modifications if the CLI didn't like something in particular.
deployments based on terraform or cloudformation were particular susceptible
to this.

------
wballard
This is pretty cool - very close to
[https://shipula.userbugreport.com/](https://shipula.userbugreport.com/) that
I've been cooking up to make my own life easier.

------
vadiml
The example yaml file contains cpu: 1024, but the text says deploy on 1 cpu?

~~~
omni
In ECS land, a single vCPU is described as 1024 "CPU units" that can then be
distributed more granularly. This would allow you, for example, to evenly
distribute a single CPU across 4 processes.

