
Under the Hood of Amazon EC2 Container Service - werner
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
======
samstave
The biggest issue that I had with ECS is that you need to initially create EC2
instances to put into your ECS cluster, using the AMI such that they have the
ECS agent on them... BUT you have to prescale that cluster manually
thereafter.

In the task definition - it would be MUCH better if you could select the EC2
instance type you want, collect them in an ASG and have the task slicing scale
the ASG accordingly.

Right now - you have to manually determine the slice size for each container-
to-EC2 and manually scale the ASG.

Further, it was noted that ECS is actually NOT AZ aware and it will spread
load over EC2 instances in the pool -- but it wont also balance the tasks
across AZs....

So, its a fantastic version one... but these are some resiliency and scaling
features that should have already been included.

~~~
werner
Nope, we have built a new scheduler for you that will allow placement over
multiple AZ's, replace failed containers, allow them to connect to ELB's, etc.

~~~
2461001642
Thanks for responding! Can you say how you deal with the significant latencies
over multiple AZs?

Also, would you mind answering my other question about whether or not I can
run Marathon or Chronos against ECS since it runs Mesos under the hood?

~~~
werner
Can you elaborate on the intra- inter-* latencies your are seeing? for AZ
independent services this should single digits.

~~~
2461001642
When I'm balancing a single deployment across multiple AZs (e.g. US-East ->
US-West), the latencies between the containers seem far higher than just the
200-300ms predicted by speed of light. Am I doing something wrong?

------
justinsb
I think it is interesting that AWS seems to be moving to consistent data-
stores. Previously they were championing eventual consistency everywhere, even
when it made for painful products (SimpleDB) or painful APIs (retry loops when
using EC2 APIs).

~~~
larsmak
I don't know, S3 and DynamoDB are both eventual consistent. And keeping in
mind the CAP-theorem it makes sense. And I for one love SimpleDB - it's just
that, simple. And great for prototyping (really cheap) and small production-
loads. Often you just need a place to stick your data, scalability can be
achieved to adding a caching layer.

~~~
justinsb
They have actually been making S3 _more_ consistent over time: in the newer
regions you get e.g. read-your-writes for object creation. DynamoDB also
supports consistency, though still defaults to eventual consistency if you
prefer.

In my mind, there's definitely a trend towards consistency here. I'd love to
see an AWS blog post about the reasons behind this!

~~~
zorked
We should be glad. Eventual consistency is hard to reason about, particularly
when its tradeoffs have to do with other people's systems...

------
Rapzid
A glaring gap currently is security. Per-container IAM roles would go a long
way IMHO, but that still leaves "other" secret management which is a PITA.
Other options such as kubernetes lack the AWS/ELB integration; all seem to be
lacking a good security management model.

~~~
TheIronYuppie
FWIW, Kubernetes provides its own load balancer, which you can put behind ELB.

Other than that, Kubernetes works on AWS out of the box, with a one line
setup.

Full disclosure: I work at Google, on Kubernetes.

~~~
Rapzid
I'm not sure that's what I would call an integration... AWS provides for easy
host management and elastic scaling traditionally through the integration of
the ELB with autoscale groups, and now with life-cycle hooks. I'm not aware
that kubernetes integrates with this stuff in any way or provides a sufficient
alternative. Reading through the documentation I was not able to find
information about connection draining on rolling updates, taking hosts out of
service for maintenance/scaling/replacement, and so on. I am aware that
kubernetes will run on AWS now and there is a guide for setting it up.

However this really wasn't the point of my comment, which is that security for
application secrets(and AWS API access) is currently a sore spot. It would be
nice if kubernetes would adopt some of hashicorps stuff like consul,
templates, and vault. Maybe that's too far up the container stack though and a
popular bundling of technologies will appear.

------
kylemathews
How does this compare to Kubernetes?

~~~
carterehsmith
I guess the better question would be "how does Kubernetes compares to this"?
It's not like Kubernetes is a market leader or anything.

------
2461001642
I've heard that it just runs Mesos under the hood, how does this differ than
me running Mesos on my own? Can I run Chronos or Marathon against it?

