
Terraforming 1Password - kawera
https://blog.agilebits.com/2018/01/25/terraforming-1password/
======
jbergknoff
Great post. It's always good to see more examples of people putting these
tools to work.

With that said, I consider getting the AMI id dynamically to be an anti-
pattern which undermines the principles of infrastructure-as-code.
Specifically, it introduces an implicit build variable "time of `terraform
apply`", which is not tracked in version control. Happily, because of
Terraform's design, this sort of thing mostly won't cause your infrastructure
to drift into unexpected states (e.g. production instance 1 running AMI X and
production instance 2 running AMI Y). Within an environment, things should be
consistent, but your staging environment may run AMI X while production is
running AMI Y, and you wouldn't know from looking at your Terraform
definitions.

I previously wrote about similar ideas in the context of pinning dependency
versions, where wildcards can and often do get you into bad states.
[https://jonathan.bergknoff.com/journal/always-pin-your-
versi...](https://jonathan.bergknoff.com/journal/always-pin-your-versions)

~~~
juliangamble
Interesting point. Here is the counter-argument.

With a Continuous Delivery Tool Like Go-CD (Not Continuous Integration) - you
solve this problem with connected pipelines (value-streams.)

In this case - the ami-id is resolved dynamically, but the particular version
comes from the pipeline, is entirely repeatable and traceable (and dare I say
- immutable).

The scenario being that the first pipeline builds the ami, then stores the ami
ID as a value/variable/text-file that is then passed on to the terraform apply
pipeline. You can repeat the terraform apply with the same ami-id. You can run
a new ami build, get a new terraform apply with the ami from that build.

With respect sir - I believe there is room for more nuance in your claim that
'dynamic ami-ids in terraform are an anti-pattern' \- this is a solveable
problem. This has been solved.

~~~
jasonlotito
We just store the AMI in a terraform file itself, and update it as needed. I
mean, if you are already updating a text file, might as well update a a text
file called ami.tf with the new AMI ID and store that in source control as
part of the build process. This ensures that the terraform plan is an easy to
read source of truth for what is supposed to be up in production.

------
geerlingguy
The code comparison between CloudFormation’s abysmal JSON formatting and
Terraform’s DSL is a bit disingenuous.

CloudFormation has supported YAML for at least a year or two now, and it’s
leagues more readable and compact, not to mention maintainable—you can even
add comments to your code with YAML (something that is impossible with the old
JSON format).

I’ve spent a lot of time working between the two, and while Terraform does
have a lot to offer and is often a very valid option, CloudFormation has its
virtues, especially the fact that almost all bleeding edge AWS features are
first available to be managed via CloudFormation (sometimes with weeks or
months of lead time), and many bugs can be more readily ironed out with AWS
support (assuming you have it).

Again, not saying don’t use Terraform, just that I don’t think the decision is
quite as black and white as this blog post seems to make it.

~~~
djhworld
Personally I find YAML for cloudformation worse than JSON because of the
whitespacing requirements. JSON isn't much better mind, especially once your
template reaches to hundreds of lines with nested objects.

Recently I've come around to using Troposphere [1] to write cloudformation
templates, it's actually very pleasant to use. You just write your
infrastructure in python, and it will generate you a template at the end. The
developers seem to respond quickly to changes AWS make to the CF templating
language too.

[1]
[https://github.com/cloudtools/troposphere](https://github.com/cloudtools/troposphere)

~~~
cle
I'm sorry, what? You don't like YAML because it uses whitespace as a delimiter
instead of curly braces? That is probably the least important feature that
YAML adds for CF templates.

YAML, despite its warts, is _much_ more readable and maintainable for CF
templates than JSON, particularly when you are doing non-trivial things and
need to use a lot of intrinsic functions and string manipulation. Or you want
to put comments in your template.

I've worked on huge JSON and huge YAML CF templates (I'm talking templates
that are thousands of lines long--in YAML). YAML is without a doubt easier to
maintain.

I can't recommend any CF libraries for generating templates either, unless you
want to wait around for new CloudFormation features to be implemented or
suffer from half-broken existing implementations (or waste time hunting down
bugs and submitting patches). _Sometimes_ it makes sense to use a template
engine like Jinja or ERB. But I'd stay away from libraries that generate CF
templates--they're mired with missing edge cases and they're mostly an
unnecessary dependency.

~~~
matt_wulfeck
One supports inline comments, the other doesn’t. That should be enough for
anyone to choose one over the other.

------
rgoodwintx
Side note, for anyone wondering like I was, the visualization tool is
Cloudcraft: [https://cloudcraft.co/](https://cloudcraft.co/) . I'd love to
have something similar for doing isometric views of any kind of diagramming.

~~~
smsm42
I am not sure I get the point of making it (fake-)3D. Except for all text to
be diagonal and me having to tilt my head to read it, and cute pseudo-3D
pictures of variously formed boxes, I don't see any advantage. It's still
essentially 2D - there's no third dimension I could usefully explore - it's
just presented in a visually cute but informationally cluttering way. Is there
any advantage to this form of presentation?

~~~
epistasis
It's funny, I came to the comments to specifically complain about that
graphic. There are significant disadvantages to the isometric view for that
data, and absolutely no advantage that I can see, other than it looks pretty
if you're not trying to get information out of it.

I think that Tufte needs to make a resurgence with this generation of
designers.

~~~
sneak
Take it up with BeOS? :D

~~~
epistasis
Hah, I think _icons_ are the place isometric views work really well, actually!
Lots of room for highly differentiated objects with an isometric view.
However, text and interconnections and maps are really not great at these
isometric angles. And I _do_ really like the icons used in the figure in the
original post, just not the whole.

------
philsnow
Here I am wishing that CloudFormation had one killer feature that would have
allowed us to use it at work: the ability to adopt existing resources into a
CF stack. When we were starting on the path of "hey maybe all our infra
shouldn't be pointy clicky", we chose between CloudFormation, Terraform, and
making something in-house. Out of those three, Terraform was the clear winner
for us at the time, but it has not been without issues.

Nobody else in this whole thread seems to be complaining about state
management. I think it's insane that Terraform encodes where a given resource
is in your filesystem / module hierarchy into the state JSON structure.
(Unless something has changed since I last looked,) if you want to move things
around in your .tf source, terraform can only apply that by tearing down the
old resource and recreating it.

For our large setup, in order to adopt Terraform, we've had to spend a ton of
time upfront thinking very hard about how all of our .tf sources are going to
look, and it's delayed our deployment by months.

~~~
raziel2p
Terraform allows you to make modifications to the state file yourself, both
with terraform commands like terraform state mv, or manually if you're brave
enough to edit the JSON. It requires confidence in using the tool, of course.
But it also encourages you to create your cloud resources in a way where it's
safe to let Terraform destroy and re-create most of it at any time.

~~~
bchallenor
`terraform state mv` is indeed the trick. It took me a while to understand it,
but this blog post helped. [1] It leads you through refactoring some resources
into a module.

The key takeaway for me was "we really only need to consider the nodes that
map to the physical resources of our infrastructure when we are planning our
state surgery. This means we can ignore all of the nodes that correspond to
data sources, variables, and providers."

So after a refactor, this is what I do now: (1) run plan to get the names of
everything terraform wants to delete and recreate; (2) pair all the resource
nodes manually and translate them to state mv commands; (3) re-run plan and
verify that terraform is now convinced there is nothing to do.

It would be nice if terraform could do this for me, of course, but I find that
it is generally possible to avoid delete and recreate if all I've done is a
refactoring.

[1] [https://ryaneschinger.com/blog/terraform-state-
move/](https://ryaneschinger.com/blog/terraform-state-move/)

------
curun1r
Their AWS bill seems like it would be a lot higher than it needs to be.
They're spinning up always-on staging, testing and development environments in
3 different regions. I know they've got autoscaling configured, so their
production environment should be significantly larger than the others, but it
still should be possible to be much more economical while accomplishing all of
those non-production workloads.

The whole point of infrastructure-as-code should be the ability to spin up
environments on-demand, do work, and then spin down. There's no reason to spin
up an always-on shared development environment when developers can easily spin
up their own environment when they need to do testing and kill it when they're
done. Most development tasks can be tested in a single AZ, let alone region.
Similarly, QA shouldn't need an always-on, 3-region setup.

~~~
roustem
You are right. Our AWS bill could certainly be lower and we will have to start
optimizing it at some point.

It is does require extra time/effort though. We deployed 1password.ca and
1password.eu just a few months ago and never really got to that.

The only thing we "optimize" at the moment is the smaller number and size of
EC2 instances in non-production environments.

~~~
curun1r
FWIW, I wasn't trying to be critical. Or, at least, I was trying to be
constructively critical. Getting to a Terraform setup is an excellent
start...lots of companies can't get that far. I guess I'd just encourage you
to view it as the beginning of a longer process that allows you to take full
advantage of the fact that AWS bills by the hour and lets you launch as much
infrastructure as you want.

Always-on non-production environments are, to my mind, a vestige of the time
where you had physical servers that needed to be provisioned by a person and
cost roughly the same amount when switched off. Or at least the time where OPS
built and maintained each AWS VM by hand. On-demand just offers so much more
flexibility...stuff like the example from my response to a sibling comment--
you should be able to type a single command and spin up an environment from a
pull request any time a code reviewer wants to do testing to ensure that what
code looks like it does is what it actually does.

The more you can leverage the work you've done to get to where you are, the
more you can drive down your AWS costs while giving greater flexibility and
isolation to your non-prod workload.

------
wgjordan
I use CloudFormation to manage similar AWS web-app infrastructure. I've been
continuously evaluating Terraform over the years (it is indeed maturing
quickly), but have still decided to stick with CloudFormation for now, and
would still continue to recommend the same for anyone managing an AWS-
exclusive (or mostly-AWS) deployment.

To respond to some specific items mentioned in this post:

\- "Terraform has a more straightforward and powerful language (HCL) that
makes it easier to write and review code."

It's easy to pipe configuration through your favorite 'straightforward and
powerful' templating language of choice to generate the stack template used by
CloudFormation. I definitely wouldn't use CloudFormation at all without an
extra preprocessing step of some sort.

\- "Terraform has another gem of a feature that we rely on: terraform plan. It
allows us to visualize the changes that will happen to the environment without
performing them."

CloudFormation has a similar feature called "Change Sets" (released in March
2016).

Finally, two more things to note:

\- CloudFormation supports "Custom Resources", which allow you to write
Lambda-function scripts to perform any custom operation you want (e.g.,
interact with third-party APIs, or support new AWS resources that don't yet
have an official CloudFormation resource implementation). There's not as
extensive a library of providers Terraform supports, but you can often find
open-source providers to fill the gaps on AWS resource until an official
implementation is released. And if not, you can just write a quick and dirty
implementation yourself. Terraform supports a similar feature in "Custom
Providers", but they are Go-only as opposed to the various supports Lambda-
language runtimes.

\- CloudFormation and all official AWS-resource implementations not only have
excellent public documentation but are officially supported by AWS Support,
which is a big deal if you're deploying something with any degree of
complexity or cost/risk.

As for Terraform's strengths: its extensive set of resource providers beyond
AWS ecosystem can't be beat. Also, it has great support for mapping existing
infrastructure resources, which is a feature still sorely lacking in
CloudFormation. (I'm surprised the author of this post didn't take advantage
of this for their migration to avoid downtime!)

~~~
0x62c1b43e
Could you share what preprocessors/templating languages you've used with
CloudFormation?

~~~
malcolmjuxt
At JUXT we use Clojure's EDN, bolstered with some tag literals courtesy of our
Aero library (described here:
[https://juxt.pro/blog/posts/aero.html](https://juxt.pro/blog/posts/aero.html)).
We use ClojureScript to compile to TF's JSON. EDN allows comments, ignores
commas and otherwise is a nicer JSON. Aero allows us to encode multiple
environments in a single document, and include ciphertext for secrets. We're
pretty happy with the overall result.

------
chatmasta
I’ve always wondered — from a security perspective, is this kind of an in-
depth engineering blogpost a good idea? You’re basically handing a map of your
internal infrastructure to any potential attacker who reads the blog.

Of course obscurity is not security blah blah blah. Still I can’t help feeling
that writeups like this could backfire down the line.

~~~
vr46
The internal infrastructure of AWS apps are fairly guessable anyway. There are
only a limited number of AWS resources and they fit together in predictable
ways, VPC - Gateway - ASG - EC2 etc etc.

I think it's really great that they've talked about this, it's quite rare to
hear about these kinds of internal migrations, and it's something I do a lot
with clients but it's not really glamorous enough to talk about.

~~~
KurtMueller
Hi vr46, where should I go / what resources should I consult if I want to
start learning more about AWS & setting up infrastructure? Thanks!

~~~
alexbilbie
Forgive the self promotion but I'm building a subscription video site -
[http://stackleap.com/](http://stackleap.com/) \- to teach people how to build
and deploy to AWS infrastructure.

The difference between StackLeap and sites like acloud.guru is that I'm
focussing on the day to day stuff rather than high level concepts you need to
know to pass the certification exams.

I'm hoping to launch a beta in the next few weeks.

~~~
insomniacity
I've been very impressed with CloudAcademy - it seems to issue AWS accounts
per course/per lab - it guides you through the steps and check whether you
completed them by checking the AWS account, and then tears it down at the end.

You might not be ready for that level of integration, but see if you can get a
demo or trial.

------
mterwill
Interesting blog post, but AgileBits should have communicated planned downtime
to their customers via email, which they did not. I’ve been a 1Password user
for years and recently switched to their hosted offering. I know a massive
infrastructure migration is rare, but that’s all the more reason to be
transparent ahead of time.

~~~
roustem
I am sorry if the downtime affected you in any way.

Sending several million emails could be a challenge considering that most of
our customers depend on 1Password apps and usually not affected by the
downtime.

We do have a status page and Twitter feed where we make announcements:

[https://status.1password.com](https://status.1password.com)

[https://twitter.com/1passwordstatus](https://twitter.com/1passwordstatus)

~~~
NelsonMinar
Have you considered building a notification system into the app?

------
fredsted
If you want to try out Terraform yourself for a simple Web server/DB
Server/Load balancer infrastructure, I made a tutorial for just that here:
[https://simonfredsted.com/1459](https://simonfredsted.com/1459)

------
juliosueiras
If you are using vim for terraform, then you should try this
[https://github.com/juliosueiras/vim-terraform-
completion](https://github.com/juliosueiras/vim-terraform-completion), hoping
to migrate to langserver by this year so any editor can use it

P.S. sorry for the self-promo

------
dev2ops
Awesome post! Always happy to see more committed infrastructure. We made this
move a while back and were really happy with the results. Check out Packer for
AMI builds and you're all set.

One suggestion re: TF files is to keep each service in a separate TF
file/state and keep all your state files on S3 that way if someone does an
apply it's always consistent. Keeping things separated means that you don't
have to worry about hitting other services when you do an apply

------
acd
There is a Terraforming github project which exports existing Amazon EC2
infrastructure and creates Terraform infrastructure tf code and tfstate.
Terraforming github repo does not seem to be related to 1password.

Terraforming github repo
[https://github.com/dtan4/terraforming](https://github.com/dtan4/terraforming)
[http://terraforming.dtan4.net/](http://terraforming.dtan4.net/)

------
GordonS
Wow, I found it fascinating to read such a detailed description of the
architecture of a running business! Is anyone else aware of similar blog posts
from other companies?

~~~
Zaheer
I've found [http://highscalability.com/](http://highscalability.com/) &
[https://stackshare.io/featured-posts](https://stackshare.io/featured-posts)
to be really useful. I constantly refer to these at work for design
inspiration.

------
sridca
Terraform manages infrastructure; if you also want to manage the installed
software in a declarative and reproducible way consider NixOps (based on Nix/
NixOS): [http://container-solutions.com/step-towards-future-
configura...](http://container-solutions.com/step-towards-future-
configuration-infrastructure-management-nix/)

------
paulddraper
> servers will be down for the next few hours. We are recreating our entire
> environment to replace AWS CloudFormation with @HashiCorp Terraform.

One of the greatest strengths of terraform (vs say, cloudformation) is that
you can adopt existing resources, and ZERO downtime is needed to migrate.

Why take downtime?

~~~
stimur
Because we needed to re-deploy our database, which unfortunately requires
downtime.

------
somtum
With infrastructure in multiple regions, do you have independent Aurora
databases running in each region? Do you have any data synchronising between
regions or are they running completely independently?

~~~
stimur
They are completely independent, that was the goal.

------
mcrmonkey
What are your thoughts on using modules inside terraform ?

~~~
_ix
Not the op, but I have some brief thoughts.

Given a bit of time with Terraform, the need for modules becomes obvious as
you identify common resources for the infrastructure you're modeling. I
thought writing modules first was the "right way," but starting with modules
ended up being a waste.

~~~
scaryclam
I'll second this. I started out not using modules, but figured I'd migrate
things over to using them as and when it would make things tidier. I never
migrated things.

Modules are really neat but I don't think they're a sensible starting point if
you're not doing a lot of repetition or don't exactly know what you need to
build yet.

------
gergnz
I find that both terraform and cloudformation have their pros and cons like
many things in life. It comes down to personal preference and how you think
(vim/Emacs?).

For me, I choose cfndsl to build my cloudformation templates. This gives me
the ability to actually write code. JSON, YAML, HCL aren't full languages. Yes
tf and cf try to provide language like concepts (iteration, etc) but really at
the end of the day they are just a definition of your environment and not
strictly code.

Note: I maintain cfndsl, so I'm totally bias.

------
somtum
One major drawback of Cloudformation is its inability to view changesets of
nested templates.

------
alexnewman
Two of my least favorite technology. I have been using pass for a while and
will never go back

------
merb
would've been easier to just migrate to k8s and used some kind of
ignition/managed k8s (and maybe ansible, if things still needs some manual
tweaking).

~~~
ben0x539
I'm really interested in migrating some of our own systems/services to k8s and
I'd love if you could elaborate a bit. How do IAM resouces or VPC subnets etc
map to k8s concepts?

~~~
kronin
You can use IAM to auth to a cluster using heptio authenticator
[https://github.com/heptio/authenticator/blob/master/README.m...](https://github.com/heptio/authenticator/blob/master/README.md)

You can grant IAM roles to individual pods running in k8s using kube2iam,
though there are new advancements seemingly coming, or already out, now that
amazon has announced eks.
[https://github.com/jtblin/kube2iam](https://github.com/jtblin/kube2iam)

Kops can provision and manage a cluster that incorporates multiple subnets
allowing you to have a multi-az buildout
[https://github.com/kubernetes/kops/blob/master/docs/high_ava...](https://github.com/kubernetes/kops/blob/master/docs/high_availability.md)

------
notyourday
Wait. People are discussing and giving accolades how marvelous it is that a
company is migrating infrastructure to a favor of the month because it is
using a flavor of some other month tools when the company has the audacity to
say they would be down for _hours_?

Are you kidding?

~~~
_ix
I don't think there is any shortage of literature on the need to avoid vendor
lock-in. Selecting a cloud agnostic tool like those developed by Hashicorp and
the opensource community offers the folks at 1Password additional flexibility
in their choice of cloud providers. AWS is great, but still...

I don't see any mention of Terraform Enterprise here, either. I imagine
they're perfectly capable of pursuing Terraform with the foss version,
although the enterprise complement has some pretty great additional features.

~~~
notyourday
It is possible I was not clear:

The blog post is patting itself on a back for migration that caused downtime.

It included this gem:

> Couldn’t you’ve imported all online resources? Just > wondering.

> That is certainly possible, and it would have allowed us

> to avoid downtime. Unfortunately, it also

> requires manual mapping of all existing resources.

> Because of that, it’s hard to test, and the chance of a

> human error is high – and we know humans are pretty bad

> at this. As a wise person on Twitter said: “If you can’t

> rebuild it, you can’t rebuild it“. _

