
Rebuilding Our Infrastructure with Docker, ECS, and Terraform - mafro
https://segment.com/blog/rebuilding-our-infrastructure/
======
zenocon
I just setup something similar for one of our api services as a trial to see
how it would work out. My biggest gripe with ECS is that it currently does not
support dynamic port mapping between ELB and a container instance. For
example, if your service listens on port 8080, and ELB terminates port 443 and
forwards to a container, then you can only have one container instance per EC2
cluster node, b/c spinning up two containers on the same node results in a
port resource conflict.

This makes the whole solution fairly similar to just using auto-scale to spin
up a new EC2 instance per app.

I know this is a major gripe right now with ECS, and hopefully Amazon is
working on a solution.

Other than a few other frustrating gotchas, ECS has been pretty nice. I do
like the idea of putting central config in S3 and copying config files over to
cluster instances with a user data script (how they recommend you pull from
private docker repo).

[https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-
agent-config.html#ecs-config-s3)

~~~
whodatninja
you can get around the 1:1 ratio of docker container to host by introducing
service discovery. We are using ECS + docker with consul as our service
discovery solution. We have an edge gateway using Zuul, with our routing
filters hitting consul to get a service endpoint and port. Then forwarding the
request to that service. It has been nice sofar and allows us to run multiple
containers (same service or different service) on a host.

------
benjaminwootton
Self promotion, but I recently published an intro to ECS after doing an
extensive evaluation:

[http://www.infoq.com/articles/intro-aws-
ecs](http://www.infoq.com/articles/intro-aws-ecs).

ECS is solid. Happy to answer any questions on it.

~~~
bndw
Great article. It focuses on the details of ECS, as well as the high level
concepts of orchestration.

As someone relatively new to Docker, the latter was extremely helpful.

------
simonpantzare
I can see how having separate AWS accounts for dev/stage/prod makes things
easier. Such as not having to think about name conflicts on ECS clusters, SQS
queues, roles and whatnot. We use a single AWS account with multiple IAM users
and name our resources like Uploads-Dev, Uploads-Stage, Uploads-Prod, and so
on but I feel that it clutters the AWS console and makes it difficult to keep
IAM and roles (as well as their access policies) simple.

Maybe I have missed some developments in AWS world, but in the screenshot of
the AWS Console it says you are logged in as ops-admin @ segment. So you have
one AWS account with alias set to segment, right?

Do you use an identity provider to be able to log in as calvin and is ops-
account a user in your AWS account named segment? And is ops-account in the
segment AWS account then allowed to assume roles in your three other AWS
accounts for dev/stage/prod?

If you switch to ops-admin @ stage for example, and then go to the S3 page in
the console, will you get a listing of all buckets in the stage AWS account
even though your login session as calvin "belongs" to an other AWS account
(segment)?

Sorry if my questions are confusing. I am very interested in knowing what the
relationships among the identity provider, AWS accounts, and roles look like.

~~~
athrun
They are leveraging the Cross-Account access feature of the AWS Management
console, which has been released early this year:
[https://aws.amazon.com/blogs/aws/new-cross-account-access-
in...](https://aws.amazon.com/blogs/aws/new-cross-account-access-in-the-aws-
management-console/)

> If you switch to ops-admin @ stage for example, and then go to the S3 page
> in the console,

> will you get a listing of all buckets in the stage AWS account even though
> your login

> session as calvin "belongs" to an other AWS account (segment)?

It should work like that, yes. Your privileges will be "scoped" to the role
you are assuming in the "stage" account.

> I can see how having separate AWS accounts for dev/stage/prod makes things
> easier.

It's also a good way of maintaining agility when you have multiple teams
working in parallel on different projects (ex: 1 team = 1 AWS account).
Zalando has recently published a bunch of tools leveraging IAM to help manage
multiple AWS accounts (ex: federated SSH access) :
[https://stups.io/](https://stups.io/)

~~~
simonpantzare
I missed the announcement of cross-account access. This seems great, thanks!

STUPS looks interesting. What scares me a bit about these suites that provide
many abstractions on top of AWS is how they work in mixed environments where
some resources have been set up and are managed out-of-band. I understand that
the purpose of STUPS for example is to provide a higher-level interface to
AWS, and that having many AWS accounts avoids these mixed environments.

Perhaps it's just me suffering from analysis paralysis. I kind of want there
to be one or two leading suites of AWS PaaS tools to choose from, whereas the
market today seems fragmented with new tools popping up all the time. For the
moment I'm betting on HashiCorp. :)

------
dminor
Using DNS for service discovery is very convenient but it also makes it
difficult to change your setup quickly in the face of ops emergencies, since
it's so aggressively cached.

We're in the process of transitioning from our datacenter to AWS and we landed
on much the same set of technologies, minus Docker.

~~~
wakwanza
If done right can really be awesome , set up stack using hasicorp's
consul(consul.io) and theres the option to change the TTLs and cache limits.If
using the DNS interface the updates are relatively painless and nodes are
almost instantly available when provisioned.

~~~
dminor
We ended up using consul's K/V store for discovery.

------
affablebloke
[http://weave.works/guides/service-discovery-with-weave-
aws-e...](http://weave.works/guides/service-discovery-with-weave-aws-ecs.html)

------
late2part
Good for you setting up separate AWS accounts. Companies that do this avoid
many problems! Companies that assume their code will arbitrate the separation
intrinsically often go oooooops!

------
wh-uws
No matter what url I try to visit at
[https://segment.com](https://segment.com) I get an ssl encryption error.

I'm on android 5.0.1 in chrome

~~~
medius
It might be an issue with the certificate chain not installed in the correct
order on their servers. I've had this issue in the past. Mobile browsers
sometimes do not "fill in the gaps" for missing/out of order certificates, but
desktop browsers are much better at that. Not sure why.

~~~
mmcclellan
Just checked it here:

[https://www.digicert.com/help/](https://www.digicert.com/help/)

and chain appears to be correct.

------
sinzone
I see you use different machines for NGINX and auth. Do you plan to combine
all of them under one API gateway like Kong?

------
bndw
Is it possible to use ECS with something like Dogestry, rather than ECR or
Docker Hub?

------
joestr87
I think it died.

