
Docker Releases Plugin for Simplified Deployments into AWS - msolujic
https://www.infoq.com/news/2020/07/docker-ecs-plugin/
======
secondcoming
None of this is simple. We're drowning in YAML files. It's difficult to
maintain. Jinja + ansible + helm... it's a joke.

There must be a better way of doing all this.

~~~
bbarnett
The real problem is, people are deploying dozens of different coding
languages, any technology that whimsically passes by, and replacing simple,
streamlined monolith technology with 100 micro services.

All of this is endlessly pushed by AWS, Google, Docker, and anyone else with a
foot in the "Snag as much cash from DEVs" crowd.

Other old timers will explain how they ran thousands of hits/second on 6 or 7
bare metal servers, 15 years ago, without CDNs, using LAMP, ajax/js frontends,
and more. The key was code optimization, SQL queries without inane and poorly
written MVC SQL logic, and the list goes on.

I am relentlessly gobsmacked at how people are spending quite literally 100 to
1000 times the cash to host on AWS, using microservices. And often amused how
new devs just can't get it through their head, that yes, this is 100%
factually true.

Docker replaced something that was already done ; identical PROD and DEV
environments. All DEVs I had working with me, were issued VMs with 100%
replicated PROD. VMs auto-build using debootstrap + SVN checkout.

Rollbacks in prod? Handled by SVN rollbacks.

When I look at the insane complexity being displayed by containers on top of
VMs on top of baremetal, the MASSIVE loss in performance (yup, it's there..
especially for IO)... I just don't get it.

Many DEVs have all been sold a pack of complete and total lies.

I get called in again, and again by clients to reduce cost, optimize resource
usage. I literally bring 1000x performance boosts to the SQL layer with
minimal improvements.

Anyhow. Yes, this was a rant. Sorry it's a reply to you specifically.

~~~
treis
Docker images are just lighter weight VMs. Or to be more accurate, they
accomplish the same goals as full VMs in a different way.

If you're running Docker in a VM on a bare metal server you're doing it wrong.
You should be running Docker on a bare metal server.

You're also conflating different problems here. If someone is writing poor SQL
it doesn't matter if their deploying with a VM, Docker, or onto a bare metal
server.

~~~
jen20
> they accomplish the same goals as full VMs

Not if you're doing things that require certain kernel features. For example,
if I have an application that uses io_uring, it's _very_ pertinent as to which
kernel it runs on. A VM has that in scope, a Docker container does not.

~~~
secondcoming
Exactly. io_uring is a game-changer for linux.

------
irjustin
I am in the process of migrating my stack from Elastic Beanstalk multi-
container to Fargate so this looked like an interesting thing I could 'pick
up'.

This does potentially unify the container definitions between compose and
ContainerDefinitions in the task definitions, but for my self, that's not a
super helpful tool.

Much of the complexity of running Fargate is outside of Fargate, wiring
everything up so Route 53 => CloudFront + WAFv2 => ALB => TargetGroup =>
Fargate w/ security groups, subnets underpinning it all.

I can't recommend looking at this seriously as something to run in production.
CloudFormation/Terraform is still the best place to sink your time.

~~~
indemnity
I just spent this weekend setting this up to learn a bit of AWS for a toy
project. I thought I would "just" quickly drop a Rust API server image in an
ECS cluster.

By the end of the weekend, I had the architecture you describe.

\- Route53 alias A record -> ALB DNS name

\- LetsEncrypt cert in IAM

\- ALB listener doing SSL termination using the cert -> forwarding to target
group

\- ALB listener doing 80->443 redirect

\- Security group on ALB listener allowing only approved IP ranges in (not
ready for this thing to be public yet)

\- Security group on ECS service only allowing ALB to connect

\- ECS cluster using Fargate

\- RDS instance only allowing ECS service to connect

\- CloudWatch log group for the container logs

\- Subnets

\- Secrets Manager for pulling Docker images from private GitHub Packages repo

Did it all in Terraform, and then added GitHub Actions to the Terraform
repository to do _terraform validate_ on PR and _terraform plan && terraform
apply -auto-approve_ on merge.

Then, yesterday, hooked up GitHub Actions on the Rust API server repository
build a version tagged image and publish it to GitHub packages, create a PR in
the Terraform repository to update the ECS task definition for the new image,
and if it passes the PR checks, automatically merge it (which triggers the
Terraform plan/apply run).

It did seem complex the first time I did it, but looking back over both the
AWS and GitHub Actions configuration, I wouldn't change too much. I feel
fairly confident this is secure, and I understand most of the configuration
options and why they are there. Something that "simplified" it for me would
just become a straightjacket as I get more proficient with AWS.

IaC 101 I guess, but I was chuffed when the Rube Goldberg machine whirred away
after making a code commit to the Rust repo, and two minutes later my new code
was running on ECS :)

Considering writing up a blog post about it just to firm up my own
understanding as well...

~~~
whatsmyusername
What you're describing is what I sold the company on last year (save
Letsencrypt, that's weird but whatever).

We only use terraform for the initial burn in (VPC, 2x public/2x private
subnets, empty lb, bastion, and some subnet groups) but the rest is one to
one.

Fargate isn't the cheapest platform out there but it's great for "I don't have
any ops people" or "I have a fraction of several ops people not dedicated to
my product." It takes a lot of patching and maintenance out of the equation.

If you want to give yourself a huge resume item, hook AWS WAF into the load
balancer and play with it (you can alternatively hook it into Cloudfront if
you elect to implement that in front of your LB, though then you have to make
sure you protect what Cloudfront is talking to).

Easy task would be to geoip limit your application to the US, Canada, and
Mexico. You can verify this by running your site through uptrends and looking
at what cities get 403ed
[https://www.uptrends.com/tools/uptime](https://www.uptrends.com/tools/uptime)

------
jcims
This is far from the first time, but I have a weird dirty feeling when i see
infrastructure management tools directly reference specific service provider
IaaS/PaaS services.

It’s the most direct route vs formalizing a standard but, uglh.

~~~
scarface74
How do you have a “standard” without each provider having to wait on a
committee anytime they want any improvements?

Even the almighty Terraform which for some reason people still think makes it
easy not to experience “cloud lock-in” has cloud specific provisioners.

~~~
BjoernKW
In my opinion, being cloud-agnostic is today’s being database-agnostic.

In more than two decades of professional software development, I have only
once encountered a situation where an existing application was migrated to a
different RDBMS instead of just being rewritten. Using an ORM framework for
the sake of database independence seems hardly justified in that case.

I suppose that things will turn out much the same with cloud agnosticism.

Similar to the RDBMS case, being agnostic of the underlying technology
prevents you from using that technology’s more interesting features, to the
extent that you’re treating a database as a dumb data store or cloud hosting
as a mere space for hosting your files.

~~~
mrgordon
I mostly agree with you but we definitely reduced reliance on AWS specific
functionality after starting to offer on-premise software installations. Most
of them don’t gain you THAT much and then the software really isn’t very
portable.

~~~
ramraj07
I question this logic - there seem to be fundamental differences in even the
basic services between various cloud providers. For example you can resize a
compute-attached disk size live in GCP while you can't in AWS. Many
intricacies like this are annoying, but at the same time advantageous if you
know and use them. If you are primarily trying to just offer another layer on
top of these cloud services, (like snowflake) where cross-cloud compatibility
is part of your selling point, then it makes sense to abstract the cloud
layer. Otherwise, I feel like it's better to choose a reliable vendor and
stick to them, optimizing according to their strengths/weaknesses. Or go cross
cloud but for very specific technologies (iirc BuzzFeed did this).

~~~
jcims
Not the person you are replying too but some industries have regulatory
mandates for vendor diversity in cases like this.

In addition, when your bills get to millions per month, the provider supplies
quite a bit of TAMs and technical resources to your account. This can be
helpful, but they also get a good understanding of where you are and are not
in a position to dip out if things get sideways in billing. (Also, other
providers will throw 6-7 figure credits your way to earn your business, being
in a position to leverage them is a good thing.)

>Many intricacies like this are annoying, but at the same time advantageous if
you know and use them.

This is very true. Building services or features that depend upon these is a
good idea. Enshrining them deep within your assumptions and requirements about
how you operate cloud-based workloads can work against you.

~~~
scarface74
When your build gets to be “millions per month”, you are already locked in.
Any migrations is going to be a painful multi year progress. “Infrastructure
has weight”.

Have you been part of an integration with a health care system? They are so
tightly locked in to their existing EMR/EHR system it would make you cry.
Every third party vendor that comes along has to integrate with it.

Part of the work I did at the company that I mentioned where I was a dev lead
involved migrating our company from Workday. Their entire process was
integrated into it.

~~~
jcims
You're definitely locked in at some level, but if you are able and even
demonstrate the ability to swing 30% of your workload over to another provider
in a quarter, you're going to maintain some leverage.

Yes, have worked in healthcare on and off over the past 20 years and know
precisely of what you speak. That's a different situation IMHO, same with
ERP/HR/etc.

------
ev0xmusic
Disclaimer: I'm entirely biased. I am the CEO of Qovery.

I created [Qovery]([https://www.qovery.com](https://www.qovery.com)) to
address this problem of simplifying the Cloud. Allow streamlining the use of
the Cloud for any developer. Today traditional players stack layers to
simplify what they have created but without going out of their way of seeing
things.

Qovery makes application deployment very pragmatic for developers because we
put ourselves in their shoes. How does a developer think? He simply thinks
about his code and wants to focus on his mission, which is to address business
issues and not all the plumbing around it.

~~~
heliodor
Unrelated, why are you using parallax effects on your landing page? It works
really poorly when I scroll down fast.

~~~
sacredcows
Scroll down slower

~~~
heliodor
I tried but when I scroll down slower I have to wait for the content to slide
in from the left or from the right, or to assemble at random. Still no good.

------
jeremylevy
At my current startup, we're building something like this.

A "catalog" of architectures that you could use to create a complete cloud
architecture on your AWS, GCP or Azure account in less than one minute.

For example, you could create a docker-based architecture with CI/CD, auto-
scaling, zero downtime deployment, SSL, load-balancing, high availability and
MongoDB in less than one minute in your own AWS account.

Like Terraform with the user-friendliness of Heroku.

~~~
gonzo41
AWS in theory has a product called service catalog that fills this role. It's
not super fun. Or at least my interactions with havn't been so great. Good
luck.

~~~
jeremylevy
Thanks!

We've released an alpha version, with only one architecture, some days ago.

If you want to learn more, I've written a blog post:
[https://medium.com/revolv2/im-building-revolv-to-automate-
cl...](https://medium.com/revolv2/im-building-revolv-to-automate-cloud-
architecture-
creation-35a6e8b0b411?source=friends_link&sk=ef93b060f5094b920d5b8435ecc372f6)

~~~
qppo
I think your product would work great as an "a la carte" consultancy, it seems
well aligned with startups which may have some incentives to pick particular
pieces but need help to wire it together - and already need to allocate a
developer's salary to do it.

~~~
jeremylevy
Interesting, thanks!

When you say "'a la carte' consultancy", do you mean with a one-shot payment?

Do you think that the "user-friendly" features (deployment
monitoring/rollback, CI on all branches, health monitoring...), that you can't
have with Terraform only, are useful enough to ask for a recurring one?

~~~
qppo
What I mean is that early stage startups will have to bootstrap some cloud
infra, but their engineering teams are laser focused on creating value for
their users and validating product market fit. Every day that a technical
founder/early hire spends writing their cloud story is a day taken away from
their runway without (directly) creating value that could lead to revenue or
funding.

People will pay you to build out their cloud infrastructure one feature at a
time, or pick features off the cart as they want them (so maybe like an all
you can eat cloud buffet?). There's a bonus to that too, iterative consulting
is a great way to validate a tech stack for a problem domain before you figure
out how to turn it into a recurring revenue source.

AWS for all its flaws is actually pretty good for this already, which I think
is why it gets so popular with seed stage startups before they get locked in
(plus the credits that they give out like candy...). I don't have to think
about architecture, I google what aws cli command I need to set stuff up and
move on with my life.

~~~
jeremylevy
Thanks you very much! It's very interesting.

It's exactly what we've started to do: targeting startups in incubators.

DevOps people in early-stage startups are not very common, so it's seems safe
to say that a Devops-as-a-service may be useful for them.

------
dmarinus
I have the feeling that docker compose only supports a small subset of ECS.
ECS has stuff like Load Balancers, schedulers and a parameter store in very
specific ways. I don't think you'd want to use docker compose for that on AWS.

------
smetj
Just eat the frog and learn Terraform or Cloud formation.

~~~
oblio
> eat the frog

Eat the frog? Which language is that? The expression sounds really cool!

Most of the time in English I've encountered "bite the bullet".

------
lucraft
Hang on, I thought Docker _was_ a tool for simplifying deployments into AWS.

------
sali0
I am pretty new to this space, but isn't one of the main advantages of WASI
the fact that we can skip a lot of this containerization and simplify
deployment?

------
nwsm
This looks like spending extra effort to further couple your CD and
infrastructure to ECS.

------
miked85
I guess Docker gave up on Swarm?

~~~
chopraaa
Curious to know why you think so? ECS is a managed service. If Swarm was a
managed service on AWS, I'd agree with you here but that's not the case.

Swarm is still brilliant for running multi-node deployments.

~~~
mplewis
Mirantis purchased Docker Enterprise at a firesale price last year. They've
pledged support for two years, and they say they will be "continuing to invest
in active Swarm development," but the writing is probably on the wall. Their
release was more excited about K8s.

~~~
LeSaucy
Which is disappointing. Swarm is infinitely easier to setup and get running,
and it scales down / uses WAY fewer resources than k8s.

------
jhatemyjob
I never understood why people use Docker. QEMU exists

~~~
Traubenfuchs
Whether it was luck or hard facts, Docker became the standard and default
nearly to a point where juniors only knowing how to program a "Hello World!"
webservice will dockerize it. It's an ubiquitous technical skill like basic
SQL.

As someone who has very low interest in container technology itself, I have
zero reason to inform myself about QUEMU..

~~~
jhatemyjob
[https://qemu.org](https://qemu.org) is VM host software, like VirtualBox, but
FOSS. Instead of a Dockerfile I have a shell script spin up a VM and install
the dependencies for me.

------
sunilkumarc
Exciting news for Docker community. In our organisation we use ECS service to
orchestrate our services within AWS eco system.

On a different note, recently I was looking to learn AWS concepts through
online courses. After so much of research I finally found this e-book on
Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+
years. I found this e-book very helpful as a beginner.

This book covers most of the topics that you need to learn to get started:

If someone is interested, here are the links :) Single License:
[https://gumroad.com/a/238777459/MsVlG](https://gumroad.com/a/238777459/MsVlG)
Team License:
[https://gumroad.com/a/238777459/EpUED](https://gumroad.com/a/238777459/EpUED)

I highly recommend buying this e-book if you think AWS documentation is
overwhelming.

~~~
callamdelaney
You are literally trying to sell product in every single post you make.

