
Show HN: The Segment AWS Stack - calvinfo
https://segment.com/blog/the-segment-aws-stack/
======
ivan_ah
Very nice. I didn't know about terraform, and seems like a very powerful
combo... Can anyone comment as to the differences in functionality between
ansible AWS playbooks and terraform?

The Segment Stack is a good showcase of best-practices for AWS ... only one
thing I didn't understand, why a NAT instance is needed in each subnet?
[https://github.com/segmentio/stack/blob/master/vpc/main.tf#L...](https://github.com/segmentio/stack/blob/master/vpc/main.tf#L54)
It seems a little wasteful. Couldn't you just allow traffic between the
subnets?

~~~
bhahn
> why a NAT instance is needed in each subnet?

I don't think it's strictly needed, but it's best practice because instances
in each AZ remain independent from failures in other AZs. Were the AZ with the
single NAT to go down, then instances in the other AZs wouldn't be able to
communicate outside the VPC (ie. to the rest of the internet)

There's also a side benefit of much lower latency using a NAT in the same AZ
vs going across AZ (unscientific benchmark is 0.1ms in same AZ vs 0.3ms across
AZ)

~~~
vacri
Note that this is only important if your use-case requires your servers to
have constant, direct 'phone out' access. If you're using ELBs or similar as
your link to the outside world for your use-case, a NAT isn't really necessary
for constant access. We use a single micro NAT in each VPC, which is only used
for system updates (no, I don't have a package cache yet...) and for when
we're manually in the server troubleshooting. If there's an outage, well,
there's not much we can do in that case, and the NAT isn't needed for
production use. And if we _really_ need that NAT back up, just spin up another
one and modify the VPC.

As you say, it's not strictly needed. It really depends on your use case. If
your use-case suffers for the NAT being down, then you need HA on it. If it
can wait, then no. With the new managed NAT in AWS, you may as well go with
that if you need HA - it's cheapest is roughly twice the price of a micro
anyway, and it's one less bit of clutter in your instance list.

~~~
coredog64
If, for whatever reason, you can't run VPC endpoints then you also want the
NAT to be able to reach S3 (and some other service endpoints)

------
avitzurel
This is great!

I've been working on an open source stack that is somewhat similar and serves
the same purpose. Allowing startups to bootstrap their stack on AWS.

[http://docs.the-startup-stack.com](http://docs.the-startup-stack.com)

Great work on open sourcing this. I will see what I can do to contribute.

~~~
avitzurel
Github organization is here: [https://github.com/the-startup-
stack](https://github.com/the-startup-stack)

I am developing everything "in the open" so feel free to contribute / ask
questions.

I spent the last 3 months collecting usage information from startups using
the-startup-stack and about to make another effort to commit all that
knowledge back into the project.

------
moondev
This is amazing. Kudos to the Segment team for putting in the effort to share
and open source this. And also kudos for leveraging Terraform rather than
CloudFormation. I may fork this to get it running with kubernetes on GCP

------
tekronis
One thing that I would have loved some more detail on is how are secrets and
credentials being handled?

~~~
conorgil145
Getting secrets and credentials correct seem to be the crux of most
architectures. I'd also love to hear more from the Segment team on their
approach with this setup!

------
beat
This is so timely for me! I was struggling to build a repeatable stack with
CloudFormation, started down the road of Terraform just last night. This will
help me skip a lot of learning curve. Thank you!

------
diegorbaquero
Thank you for sharing, simple and detailed. However I'd like to comment that
AWS is not cheap.

~~~
misframer
What would you consider to be cheaper than AWS?

~~~
beat
If all you want is stuff running on the equivalent of EC2 instances, you can
use DigitalOcean or some other competitor. However, AWS offers a degree of
depth and sophistication that just isn't available from the competition.
Trying to do something that is as security-clean as this VPC using DO or
Linode or something, from scratch, sounds like weeks of hell to me.

"Not Invented Here" is a big problem in this industry in general. We tend to
be too comfortable cobbling a solution together from stone knives and
bearskins, rather than using someone else's solution (and paying for it). If
you are running a business, though, you shouldn't be building things that
aren't what you sell, unless you really cannot otherwise buy them.

~~~
ethbro
_> If you are running a business, though, you shouldn't be building things
that aren't what you sell _[or that you can't build more efficiently] _,
unless you really cannot otherwise buy them._

I'd add the above. Otherwise AWS would have never been built.

~~~
beat
I'd add, though, that "efficiently" isn't just whether you can build cheaper
than buying. It's if the cost of building the functionality is cheaper than
the profit/growth you can generate with a _sellable product_ built using the
same amount of developer effort. That's a very, very different (and probably
more expensive) proposition. That's why executives should make the decisions,
not engineers!

Back in the dot-com days, I worked on a project to build some functionality
in-house that we could easily have bought off the shelf. The engineers argued
that we'd save the company a million dollars. But frankly, we just wanted to
do it because it was _badass_. And it turned out our solution would actually
have cost us more per-system than the commercial solution we sneered at
(hardware costs, not just development cost). Six man-months of engineering
when everyone knew we were racing the clock before the money ran out? Absolute
stupidity.

If I were CEO/CTO and caught wind of such a project, I'd tell people that if
they lifted a finger on it, they'd be fired. But that's a very different
perspective than I had back then. Risking the very existence of what could
have been a very big company in order to someday save a million dollars? Feh.
(Of course, no one stopped us, because the CTO was just head nerd, and the
money execs were busy fundraising rather than supervising)

~~~
ethbro
_> That's why executives should make the decisions, not engineers!_

I understand your point and it's valid, but opportunities like this are juicy
steaks for Oracle and IBM sales guys. I think the best possible outcome is to
involve engineers, solicit a reasonable internal cost bid, then invite
external contracts, then pick what makes sense.

"We can just buy it" is what funneled money that could better be spent
elsewhere into the coffers of legacy infrastructure companies for decades.

~~~
beat
As a for-example, we were implementing this stuff in the days before MySQL had
ACID transactions (back when they bragged about how much faster they were than
other databases, handwaving over the cost others had for implementing
transaction logs and rollback). The DBA and I tried to get Sybase in, but the
Sybase sales rep, unused to startups, quickly offended the CTO with vague
pricing. He was an open source purist and wanted MySQL.

I cannot begin to explain how much extra code we had to write to deal with the
lack of transactions in MySQL! Sybase would have saved us a ton of time and
risk.

------
garysieling
If you were to set this up and leave it running (say for a dev environment),
what would your monthly bill look like?

------
joshpadnick
So, my co-founder and I have basically been independently building a
commercially supported alternative to this excellent open source package. But
there are a few differences that make sense when offering this as a paid
service:

\- We wrote our own terraform testing framework to validate that every change
to our modules doesn't break functionality \- We actively update our modules
based on feedback from new client engagements \- We provide commercial support
for each module \- We combine our modules with consulting and training as
needed

And of course, there are many similarities \- We give 100% of the source code
to our clients \- Everything runs in the client's AWS account \- Everything is
self-documented, modularized, and can be combined/composed as the needs of
different teams require

I didn't mean for this to be a shameless plug; more just that I found it
interesting to compare the open source vs. commercial approach to solving this
same problem. Props to the Segment team for sharing this.

~~~
beat
There's a lot of potential for such a service as a consultancy - advice and
in-house customization included along with the software. As cool as the
Segment stack is, it doesn't let people off the hook for designing effectively
in the first place, or for taking a poor design and re-engineering it.

~~~
avitzurel
My initial approach with the-startup-stack[1] was exactly that.

Building an open source solution with a "pro" level all setup included in it.
I had a very hard time quantifying how much companies will pay for this and
whether they even will.

How I see it, you are either on Heroku (or other similar) and you don't care
about anything except `git push` or you have a full blown stack.

I know the middle ground between the two is where the stack is but I just
couldn't figure out how many companies are actually experiencing those
difficulties and how much are they willing to pay for the help.

Since it's not a startup for me, it's just an open source project I decided
not to worry about it, but would still be nice to get some input.

[1] [http://the-startup-stack.com](http://the-startup-stack.com)

~~~
conorgil145
Based on personal experience, I strongly suspect there is a market for that
type of middle ground. Also, the author of the top level comment in this
subthread said they are doing commercial consulting work, so that shows there
is some market for it. Given, he/she did not say how good business was,
where/how/if they find customers, if they are profitable, etc.

------
fireworks10
What tool did you use to make the flow chart, or is it custom?

[0] [https://segment.com/blog/the-segment-aws-
stack/images/main.p...](https://segment.com/blog/the-segment-aws-
stack/images/main.png)

~~~
alexkappa
Thats the `terraform graph` command.

[0]
[https://www.terraform.io/docs/commands/graph.html](https://www.terraform.io/docs/commands/graph.html)

~~~
avitzurel
`terraform graph` _was_ used in the post but it's not the one he linked to.

This seems custom to me, maybe Sketch or something.

~~~
fizx
Looks like [http://www.graphviz.org/](http://www.graphviz.org/)

------
blahi
I so much wish segment would offer their services on the Azure stack.

