
Blox – Open Source Tools for Amazon ECS - samsgro
http://blox.github.io
======
addisonj
One of the more interesting announcements. ECS has seemed like a non starter
for any "serious" project for a while, but as someone who has been
implementing mesos for the last few months, its pretty interesting that they
would prioritize pluggable schedulers rather than getting better feature
parity with kubernetes on google cloud.

Pluggable schedulers is one of the best features that is missing from
kubernetes, swarm, and nomad. All the scheduling/lifecycle algorithms are
pretty naive when it comes to any stateful (DBs, kafka) or batch (hadoop,
spark) services. But even for typical webapps, pluggable schedulers give you a
nice place to hook into a lot of the lifecycle that makes things like pre-
warming or cleanly killing sockets at scale down a lot simpler.

So this is pretty cool... and makes ECS more attractive... but I wish it was
just mesos which is built around the idea of providing a framework for
building schedulers and would be way more portable.

~~~
dastbe
What other features are you missing from ECS?

~~~
Rapzid
Not the OP, but I'm missing in no particular order(and some of these may be on
offer now):

* Secret storage

* Pet sets(coming along in k8s)

* Leader election

* Persistent volumes

It looks like Blox can address some of this stuff, but kubernetes is providing
this stuff OOTB.. It's also locked to vendor-specific service offerings. This
isn't to say Kubernetes doesn't have room for improvement; its AWS
integrations could use some work(Application Load Balancer support, etc).

~~~
justinsb
Hi - I do a lot of work on Kubernetes on AWS. We definitely have room for
improvement - and not just in the glib "there's always room for improvement"
way.

But: We don't support ALB yet because we haven't actually found a compelling
use-case for it. Ingress on Kubernetes seems to do everything ALB can, but
without the limitations. At least that's what we've thought so far, so if you
have a use case do open an issue explaining it, and we can look at adding it
:-)

~~~
Rapzid
I'm not sure the last time the issue was looked at, but I could consolidate
some of the benefits that have brought up here:

* Websockets support

* Http/2 support

* Layer 7 routing + Route to specific ports(stop sending all traffic to all nodes and preserve source ip)

* Request tracing(recently added to ALB)

I hope this doesn't sound too negative, and I KNOW you do a TON of work on
this stuff, but there often seems a disconnect between the people working on
the kubernetes integrations and the feedback coming through from people trying
to run businesses on AWS. Which happens; it's tough to be in both roles if
your business isn't infrastructure.

~~~
aledbf
Please check the nginx ingress controller (you can use
[https://github.com/kubernetes/kops/blob/master/addons/ingres...](https://github.com/kubernetes/kops/blob/master/addons/ingress-
nginx/v1.4.0.yaml) in aws) The only missing piece is "Request tracing(recently
added to ALB)"

~~~
Rapzid
These are advantages over ELBv1. We have no desire to move away from AWS's
load balancing services at this time.

------
sync
In case it wasn't clear to anyone else, this is by the Amazon ECS team. An
introductory blog post is here:
[https://aws.amazon.com/blogs/compute/introducing-blox-
from-a...](https://aws.amazon.com/blogs/compute/introducing-blox-from-amazon-
ec2-container-service/)

------
fapjacks
I am really glad to see this kind of thing. I built an important project on
ECS at the beginning of this year, and it was an extremely frustrating uphill
battle, and I consider myself a seven or eight out of ten with containers and
container orchestration in general. I wanted so much to enjoy working with
ECS, especially because it was the first major Docker project that particular
client was working with, and I wanted to blow their minds. The choice to build
on ECS was the client's (not mine), and ECS made their introduction to
container orchestration lukewarm at best. Because of those obstacles, I have
been recommending _against_ using ECS. Seems like it's time for me to evaluate
ECS once more to see how far it's come along.

~~~
dastbe
I remember I had asked you forever ago about what issues you had, so I looked
it up and you responded! One thing I don't understand from your response is

 _Amazon also funnels you into using a single container type per EC2 instance.
It 's not impossible to use a single EC2 instance for multiple containers, but
if you desire to run multiple instances of one specific kind of container on
one node, ECS doesn't make the implementation easy for you at all._

Was this related to ELBs and host ports? ELB really isn't a great fit for
containers because you attach them to the instance on specific ports, which
stops you from running multiple copies of a task on a single instance and also
requires a lot of port janitoring between different tasks. ALBs attach via
instance:port pairs, so they can actually work as a "container load balancer".

~~~
fapjacks
Yes, but actually I just threw in an instance of the jwilder/nginx-proxy for
this since it was HTTP-based traffic and needs to be proxied by hostname. I
suspect this is similar to what you're describing with ALB, but this goes back
to another one of the big reasons I recommend against ECS: Terrible
documentation and ##aws on Freenode is not a helpful community. Pretty much
any other container orchestration application has a thriving community on
Freenode which is indispensable for problems like this. But ##aws is almost
exclusively populated by other frustrated users.

~~~
dastbe
Yep, sounds like ALBs would've helped. They were only launched in August 2016
so it sounds like they weren't available when you were doing your project.
They just conceptually map better to containers and the documentation has been
updated to recommend ALBs unless you need layer4 routing.

Documentation is always a moving target, but the feedback links at the bottom
of documents, in the console, and feedback to CS do make it through to service
teams who act upon it.

As for Freenode, AWS doesn't maintain an official presence there (same w/ AWS
reddit). I understand you're probably looking for something more
synchronous/interactive, but the AWS forums (
[https://forums.aws.amazon.com](https://forums.aws.amazon.com) ) are the
official place for getting public feedback from AWS.

------
Thaxll
Vendor locked on ECS, why would anyone chose that over Mesos / Kubernetes that
can run anywhere?

~~~
TomFrost
ECS is free in that you pay only for the EC2 nodes running your containers --
there's no need to host ECS or do scheduling on your own hardware to use it.
It's also Availability Zone-aware right out of the box, making sure the
distribution of container instances is optimized for durability. Finally, it's
fully managed. No one needs to maintain or upgrade your ECS implementation.

Granted, there's a lot of advantage to building on top of an infrastructure
that can be installed on any hardware from any provider. However, we're not
talking about rewriting your applications if you need to move away from ECS;
it's all still the same containers. Going from ECS to Mesos or Kubernetes when
needed is a matter of writing new config files.

It's a very attractive proposition for small teams on AWS who are trying to
spend minimal time on ops.

