
Terraform Collaboration for Everyone - mwarkentin
https://www.hashicorp.com/blog/terraform-collaboration-for-everyone
======
ris
Looking at the two diagrams shown lower down in the article, I can't be the
only one that thinks the "Before" looks infinitely superior to the "After".

Yet another third party service to depend on which has unknown reliability or
future business model plans.

~~~
tjbiddle
While I understand where you're coming from, I can tell you're not a user of
Terraform. As someone who uses Terraform daily, and has brought it to nearly a
dozen other teams at my employer - these new announcements are pretty cool :-)
Including the announcement about a free tier for Terraform Enterprise (Which,
mind you - is optional, they are not forcing you to go through it)

~~~
ris
Thanks, man - but I am a terraform user. We use the S3 backend for state
management and it works really well for us.

The thing that I like _most_ about terraform is that ultimately, it's _just a
tool_. Just something independent and (mostly) pinnable & controllable that
runs on peoples' machines.

> Which, mind you - is optional

Everything starts out as optional before they start ramping up the
monetization model.

But of course, it's an open source project, so wherever Hashicorp want to take
this, the community _can_ just fork it. However, it's clear where the center
of gravity for future development is going to be - towards pushing Hashicorp
into your critical path.

------
mwarkentin
If you’re familiar with Atlantis and think that this sounds similar - you’re
right!

They’ve also hired the Atlantis developer:
[https://medium.com/runatlantis/joining-
hashicorp-200ee9572dc...](https://medium.com/runatlantis/joining-
hashicorp-200ee9572dc5)

~~~
lars_francke
I hadn't heard of Atlantis before, thank you. Very interesting.

Do you happen to know of a self-service tool (CLI and/or Web UI) that allows
us to do the following: Run these Ansible playbooks, then apply these
Terraform modules, then run these Ansible... etc.

We often have to build different environments for all kinds of use-cases
(usually for testing, debugging or demonstration purposes). Two examples: We
need a three node Kafka cluster on the Hetzner Cloud (Terraform + Ansible), we
need a five node Elasticsearch cluster on AWS etc. For that we'd like to
specify which Terraform modules to run with which variables, we then read the
Terraform state file to generate an Ansible inventory and run a set of Ansible
plays/roles.

~~~
fishnchips
As a hobby project, I'm building a tool somewhat similar to Atlantis/TFE. It's
free though not yet open source -
[https://docs.geopoiesis.io/manual](https://docs.geopoiesis.io/manual). Not
sure about your exact use case, but it's distributed as a Docker image and
apart from Terraform it allows you to run arbitrary scripts (for example,
`tflint`). We're using the project internally at Deliveroo, so it's production
ready, but I haven't had the time to do all the proper open-source work around
the landing page, and the documentation is still a bit meh. Still, give me a
shout (email in my profile) if it ticks your box, and I'm happy to help you
get started.

~~~
lars_francke
Thanks! I've put it on my list of things to look at. It doesn't exactly fit
our use-case but it still might be interesting for us.

Also thank you very much for the offer of help.

------
outworlder
It would be great to have an intermediate step between "you are on your own"
and "ENTERPRISE – contact us!". Some 'business' plan, perhaps? I know there's
'pro', but it still has the 'request info' button instead of pricing.

I am hoping this is what they want to do with the SaaS offering.

~~~
tylersmith
That's what it sounds like from the description:

> Going forward collaboration features will be available for free to
> practitioners and small teams, at an affordable price to businesses, and
> Terraform Enterprise will remain our world-class platform for organizations
> adopting Terraform at scale.

It's not released yet though, so it won't be reflected in the pricing page
yet. The beta starts "early next year".

------
marenkay
As a long-term user (2014/2015), I shall say this is nice but... it is kind of
late.

I always wondered why Terraform Enterprise / Atlas was in its simple
incarnation a tool to be used in CI/CD tooling in form of a containerized app
with a selection of storage drivers.

That would be a use case applying to hundreds - if not thousands - of
companies, and even big players on the market. Those might want the UI with
policies too.

Who wants to give a tool / backend full access to their infrastructure and not
be the one to control all aspects of that tool?

------
SteveNuts
This is great news, the state file was always the worst part about TF.

~~~
DerpyBaby123
In modest, small team use, a remote state file on s3 served me well. Did you
run into problems with it?

~~~
SteveNuts
As a general rule I try to avoid putting sensitive information onto services
that could accidentally be turned public facing.

~~~
mike-cardwell
You can (should) use client side encryption on the state terraform state
bucket.

~~~
kitotik
Exactly. I’m a little surprised to hear that this one of the biggest feature
requests from the community.

Once you are in hashicorp-land it’s not a big jump to spin up a consul+vault
cluster and store your encrypted state remotely.

~~~
jen20
There is still a chicken-and-egg problem assuming you want to use Terraform to
spin up the Consul+Vault clusters, though.

~~~
marenkay
Why worry about that? Has anyone ever solved the chicken-and-egg problem in
it? Can it even be solved?

Seriously, is there any other approach from declaring a new egg as new
baseline chicken?

------
carlsborg
What are the arguments for TF being the better option for a new infrastructure
build today?

Given that a) Cloudformation with Custom Resources and Macros is extensible,
AWS supported, and free and b) its hard to leverage many of the AWS services
while still maintaining provider-neutrality (e.g Serverless Framework 2.0 on
AWS runs on top of cloudformation templates now)

~~~
kokey
Terraform, certainly. It was a bit rough around the edges about a year ago but
it's been getting really good. I've only been using the free cli version.

Number one advantage is that it's much better at telling you what's changing.
For example, where CF will tell you it will change a security group, Terraform
will tell you what specific rule it's going to update with what content.
Terraform will tell you if it's going to have to wipe and replace an EC2
instance, or not, where CF will tell you it's 'conditional'.

Number two advantage is that you can go from any state to getting back to
managing things with Terraform again. You can start with a hand managed setup,
one created by CF, or one created with TF and then broken by manual
modifications, you name it, there is a way to get it back to being managed
with Terraform. Sometimes it's not easy, but it's always possible. With
Cloudformation there are many things that can be done where the only fix is to
delete the stack and create it again.

Then for the Serverless specific example, I recently replaced something that
was deployed with that with Terraform. The Serverless deployment had some big
problems because of Cloudformation, one was updating stacks with where Lambda
functions needed interfaces in many VPCs, the other was some Cloudformation
string limit issue where the only workaround was to delete the stack and
create it again, which is super slow because of those network interfaces and
there would be downtime while it's doing all this. With Terraform this went
from being a risky deployment with potential downtime of many minutes, to
Terraform only changing the resources that needed changing with each update,
taking only seconds.

~~~
ReidZB
I believe CloudFormation change-sets have more detailed information about
"conditional" replacement cases.

I also prefer Terraform. However, both Terraform and CloudFormation are warty
in their own ways—this is a space where there's a lot of room for improvement.
Fortunately, the as-yet-unreleased Terraform 0.12 [1] looks as though it'll
resolve most of my complaints...

[1]:
[https://www.hashicorp.com/blog/terraform-0-1-2-preview](https://www.hashicorp.com/blog/terraform-0-1-2-preview)

~~~
kokey
I've been generally impressed with how quickly niggly quirks with Terraform
seems to go away with new versions.

My main concern with Terraform is that there's nothing anywhere near a decent
competitor out there.

------
magd
Well, I'm now concerned that they'll kill of Atlantis.

~~~
marenkay
Why would they if they can just provide a Terraform for Github application for
a few bucks per month per team with it?

------
paulgrant999
Terraform is pretty good; but its strength is its weakness; the declarative
syntax (DAG planner) isn't perfect and once you start hinting it, it causes
all sort of chaos.

The solution to complex deploys that re-use portions of the deployments you
would want to put a context boundary on, is a series of modules that maintain
their own state. But this forces you to pass large amounts of the same data
(couples the modules strongly to the main file) over and over again.

As for collaborative, terragrunt works pretty good ;) provided you can give a
home to the statefile.

The real problem I have with terraform/aws (or insert provider) is that its
still time-intensive to actually do the deploys. The parallelization of the
terraform graph, is "simple" at best.

If anyone wants to collaborate on a fork of packer/terraform, let me know on
thread.

~~~
marenkay
Wouldn't the fact that passing resource to modules works in 0.12 help with the
mass exodus of output variables for modules?

What is time-intensive in terms of deploys? It has been my experience with
tens of providers that the time loss occurs when the requests have been handed
off to a providers API but not so much in Terraform.

As for Terraform itself, my biggest issue is that it has turned around the
view of resources and providers.

It should have been going in the direction of [https://github.com/google/go-
cloud](https://github.com/google/go-cloud) where you talk in terms of
resources, and then a provider loaded would just provide the implementation.

Now I have to build Terraform modules to export resource and hide away
provider specifics in the module.

~~~
paulgrant999
> Wouldn't the fact that passing resource to modules works in 0.12 help with
> the mass exodus of output variables for modules?

might. I skipped it, use soft-links, and hard outputs (variable files) to get
around it. cuts down on the graph, I can set explicit boundaries (on whats
getting eval'd) without having to go through the terraform DSL. equivalent to
running separate terraform scripts, on separate portions of the
infrastructure.

data source providers (understandably) are not static; there is no way to
cache the output of data providers between runs; which means any query
retriggers dependencies eval which rebuilds perfectly fine infrastructure.
hence why I just output static files. Now there is no chance for terraform to
think this variable might change, and ergo, no chance for rebuild of
infrastructure that doesn't need it.

> It has been my experience with tens of providers that the time loss occurs
> when the requests have been handed off to a providers API but not so much in
> Terraform.

Agree. So this is sort of the point. I don't need to actually bring up
machines in a typical DAG fashion - so some of the work is parallelizable. But
other parts aren't. If I need 20 machines, I know I need 20 machines. If the
AMI's are loaded on the cloud provider, there is absolutely no reason to have
them loaded sequentially (such as a "post-config" might require) i.e.
incurring large wait times. Once you start using "terraform" variables (off of
resources), that is what you get. So I prefer a different variable resolution
mechanism (i.e. lazy/evented).

Particularly now that you can ghetto-leverage the cloud providers tag systems,
in order to do "proto" service discovery (tag machines for eventual service
discovery/config i.e. similar to ansible roles).

I'm twisting the hell out of terraform (for things it is not intended to do).
but that is because I want to avoid adding yet another tool to the toolchain.
More declarative DSL's = more issues to diagnose.

> As for Terraform itself, my biggest issue is that it has turned around the
> view of resources and providers.

not so much a problem for me (not multi-cloud). I like knowing (at a glance)
what provider-specific features I can expect to be supported. The problem with
trying to abstract out cloud provider, is you go with the lowest common
denominator (in terms of declarative syntax).

> It should have been going in the direction of [https://github.com/google/go-
> cloud](https://github.com/google/go-cloud)

Never used it.

Also I've never written a terraform module. So its entirely plausible, this is
where my issues would be resolved (custom provider) and an "enhanced" dsl.

------
IloveHN84
Finally

