
Terraform 0.12 - burntcaramel
https://www.hashicorp.com/blog/announcing-terraform-0-12
======
alexbanks
I honestly love Terraform as a product. It was one of probably three tools
I've used in my entire career that made me feel immediately more productive.
After using it for a very short period of time I was shocked developers
continued to struggle through CF templates and the fragility the whole process
entailed.

~~~
mirceal
this is short-sighted. Terraform and CloudFormation are not even in the same
league. One of them works and actually can be used for Infrastructure as Code,
the other one does not roll back in the face of failure - it effectively craps
for reasons ranging from network failure, process crash, even normal
operation. One is heavy kool-aid with bugs that go unresolved for years, the
other behaves as advertised.

Sorry, but I have yet to meet a developer that used Terraform and willingly
wants to keep using it after seeing it fail. When your tool cannot keep track
of the resources it created or when it gets into situations where it doesn’t
allow you to do certain things (like deleting all resources created) and you
have to relearn how the underlying cloud works, it’s time to move on.

Do yourselves a favor and use Cloudformation (or your favorite cloud’s
equivalent and just move on with your life)

~~~
tptacek
I mean, this is harsh, but there's a running joke that the big feature
terraform is missing is a -twice flag so that it'll re-run itself on failure,
since that's what you end up having to do anyways.

Also, the terraform language, HCL? It's, I guess there's no better way to put
this: not good.

Am I misunderstanding the complexity of what Terraform is trying to do? To me,
it looks like a bunch of tiny API clients tied together with a topological
sort --- in other words, it's just another species of "make". It feels like it
carries a whole lot more complexity than that concept warrants, but _just
enough simplicity_ to make it not a serious programming language. It's, to me,
one of those frustrating uncanny valley systems.

I am prepared to be totally wrong about this.

~~~
ReidZB
Yeah, HCL's limitations have been an enormous thorn in my side for a long
time. HCL2 (Terraform 0.12) is a big step forward, but even still I pine for a
proper programming language, even if it has foot-guns.

I do think you're selling Terraform short, though. Sure, the core is the
toposort-create-things. But it also stores the state of its created things and
(crucially) has the ability to diff the actual state of resources against what
it thinks they ought to be.

Being able to inspect existing resources and diff state is also what lets it
import existing resources, so that they can be Terraform-managed going
forward.

Terraform is also capable of determining if its planned changes can be
performed in-place or if they require resources to be destroyed and re-
created. That boils down to a boolean flag on a field, ultimately, but it's
still something a dead-simple make clone probably wouldn't do well.

I've only really used Terraform seriously for AWS, so I'm not sure about the
other providers, but the Terraform AWS Provider has an enormous amount of work
behind it. Basically every resource API has schema validation written in the
AWS provider, and depending on the resource there are often eventual-
consistency issues handled by the provider. See for example [0].

In contrast, AWS CloudFormation: can't import existing resources; isn't always
sure whether an update will require replacement or not; and, as of ~6 months
ago, can detect configuration drift, _but not correct it_ (!). Of course,
CloudFormation wins in other areas...

[0]: [https://github.com/terraform-providers/terraform-provider-
aw...](https://github.com/terraform-providers/terraform-provider-
aws/blob/master/aws/resource_aws_cloudfront_distribution.go)

~~~
gnosek
Long-time user of Terraform here, never used CloudFormation. Where does
CloudFormation beat Terraform?

~~~
mirceal
Cloudformation will leave your infrastructure in a consistent state. Period.
It will consistently drive things from point A to point B or rollback to point
A if it cannot get to B. It will also correctly remember all the resources it
created and allow you to properly identify them and/or delete them. These
sound like tablestakes but Terraform cannot do this. Add the frustrations of
HCL on top and it’s a big no go for me.

~~~
ReidZB
I've had CloudFormation fail to roll back plenty of times, though never in a
situation where it actually mattered.

I've never had Terraform "forget" resources or disallow me from deleting them
(unless I requested that). Maybe those are bugs that've since been fixed?

It sounds like we've had very different experiences with these tools.

~~~
alexbanks
My experience with CloudFormation has been consistent with what you just said.
CF will fail to delete resources and then lock them in that state, requiring
AWS tickets and reps and all that. I have not had anything like that in TF.

------
bshacklett
My biggest hope is that this means that some effort can be directed back to
fixing bugs. I love what terraform can do, but I hate what I sometimes need to
write to make it work.

Also, I'll go out on a limb and say that I dislike the flexibility of
iteration allowed in HCL 2. I know that people overwhelmingly asked for it,
but my opinion is that it demonstrates a fundamental misunderstanding of how
the v. 11 and earlier system was designed, and just how powerful completely
declarative code can be.

~~~
kawsper
The biggest issue we have with Terraform (and other Hashicorp tools in
general, really) is their different configuration formats, but mostly it is
HCL limitations.

I haven't found a solution for it, but I have lots of resources that are
almost identical, except for a few arguments, but right now it seems like my
options are to either write my own Ruby script to output .hcl files, or live
with two resources that are almost identical, and live with the mistakes that
could entail by future modifications.

I haven't had the time to look into HCL2 yet, but maybe it is solved.

~~~
ivan_ah
+1 for the question about a "standard" or recommended way to parse and
generate .tf files from a script.

I have two use cases in mind: Use case 1: need to have a reproducible way to
generate terraform folders for multiple almost-identical deployments (this can
be solved via modules) Use case 2: need way to "promote" a deployment from
staging to production (load a .tf file, change a few basic params then save
again to a different folder --- doesn't really work with templates since I
want to load an existing .tf file)

One thing that might work is to use the combination of two tools: \-
[https://github.com/virtuald/pyhcl](https://github.com/virtuald/pyhcl) = reads
.tf, can export .tf.json \-
[https://github.com/kvz/json2hcl](https://github.com/kvz/json2hcl) = convert
.tf.json to .tf but feels hacky...

Any other recommendations for .tf parsing and generation? (preferably in
python or scriptable via python)

~~~
robfig
Your favorite tool that produces json should be able to work. I'd probably use
jsonnet, which has a multi-file mode so that one template file can render into
many json files.

------
avitzurel
We are very heavy users of terraform. We have thousands of lines of HCL and
our own providers etc...

Terraform 0.x before 0.12 was heavily limited due to the syntax, you just
worked around so many limitations. We are actively ignoring those limitations
and we started working without our own templates using jinja that just
generate vanilla terraform. No counts and no if/else needed.

Maybe with .12 we can move some of these back to plain terraform.

Upgrade so far is not smooth though, there are a lot of pains with the type
system vs the plain old “I’ll figure it out for you”.

ALL of this being said, I really appreciate Hashicorp’s work on this, we could
not imagine our life without terraform.

------
mellab2
Terraform is an amazing piece of technology but the biggest thing holding it
back is HCL. I guess the intention was for it to serve as a happy middle
ground between a full blown programming language and a configuration language
- enabling some abstractions, but yet rigid enough to stop developers getting
too carried away. I think unfortunately that trying to have it both ways
doesn't really work and ends up leaving those writing the code frustrated -
it's almost as if it's teasing you - you can feel there's all this power under
the hood and you are given things like modules which are nice, but you always
feel like you're on a of short leash.

I switched from Terraform to Pulumi for a personal project recently and
haven't looked back (no affiliation). Writing it with Typescript means you get
excellent IDE support (using VSCode here) and access to the enormous JS
ecosystem. I've also found myself creating a number of useful abstractions -
that I would never have bothered to with HCL - like IAM helper functions, eg:

    
    
        const limitedReadAccessPolicy = createPolicy(
          "product-table-read-access",
          allow(
            ["dynamodb:Get*", "dynamodb:Query"],
            [table.arn, interpolate`${table.arn}/index/*`]
          )
        );
    
    

vs

    
    
        resource "aws_iam_policy" "product_read_access" {
          name        = "madu_${var.env_name}_product_read"
    
          policy = <<EOF
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Action": [
                "dynamodb:Get*",
                "dynamodb:Query"
              ],
              "Effect": "Allow",
              "Resource": [
                "${aws_dynamodb_table.products.arn}",
                "${aws_dynamodb_table.products.arn}/index/*"
              ]
            }
          ]
        }
        EOF
        }

------
nodesocket
Terraform and Packer are the two most common tools I use with my DevOps
consulting company. Most all of my clients previously were creating cloud
resources manually within the U/I control panel. Moving to Terraform is a game
changer in terms of transparency, auditability, and automation. With the
release of Terraform Cloud[1] which provides centralized state storage,
locking, history, and integration with GitHub the HashiCorp stack is a no-
brainer.

[1]
[https://app.terraform.io/signup?utm_source=banner&utm_campai...](https://app.terraform.io/signup?utm_source=banner&utm_campaign=intro_tf_cloud_remote)

------
n42
Congratulations to the team for a monumental release! The significance of the
changes involved to ship 0.12 seems understated. This release paves the path
to implement many highly requested features and fixes. Most noteworthy of
which, in my opinion, is module counts.

------
luhn
I’ve been very excited for this release. I’m a huge fan of Terraform—When I
first started using it I quickly couldn’t imagine working without it, despite
its many rough edges. But it continues to improve (I can think of very few
projects that have had such large and consistent improvement) and this release
is another huge step forward. Props to Hashicorp for the great work.

------
kkapelon
If you love terraform, please also look at Pulumi (I have no affiliation with
them).

[https://pulumi.io/reference/vs/terraform.html](https://pulumi.io/reference/vs/terraform.html)

[https://github.com/pulumi/tf2pulumi](https://github.com/pulumi/tf2pulumi)

~~~
doliveira
Honestly I like the idea of using declarative code for infra stuff because
this way it becomes an inventory-like configuration language. There is a
reason why YAML dominates today.

~~~
kkapelon
Terraform does not use YAML. And in the last updates they try to make their
HCL more "programming language like". Pulumi is already there (they are using
an actual programming language)

~~~
doliveira
Of course Terraform does not use YAML. I mean that YAML or other approaches
like HCL allow for a declarative style while maintaining some constructions
like for-loops or conditionals.

Having your infrastructure in an inventory-like codebase makes it clearer to
reason about. It can't go too crazy like most programs end up like

------
vander_elst
I work heavily with the Deployment Manager from GCP, so I am basically used to
describe the infrastructure using python. What seriously bugged me about TF is
how difficult it was to do simple stuff like having a nested loop within a
resource. I know that 0.12 is supposed to help there and I am looking forward
to the moment where TF will evolve into a proper programming language,
supporting saner formats (already happening with json integration).

------
marenkay
As someone using Terraform from it's first release on I still think that
Terraform has one fundamental flaw:

It always looks at things using provider specific resource, while IMHO it
should just expose a bunch of predefined resource types (see rOCCI specs e.g.)
and then allow you to attach a specific provider to it.

IMHO the biggest win as a user would be having not to have an implementation
for every provider over and over. Do we really need to have a consul module
for Azure, AWS, Tudeluuu and god knows who? No.

That being said, Terraform in the long run still is the most reliable tool in
that space.

The whole situation about state management is... lacking. Experience says the
one thing no client ever wants in the cloud but always on prem is state.

~~~
ben0x539
> The whole situation about state management is... lacking. Experience says
> the one thing no client ever wants in the cloud but always on prem is state.

Hm, can you elaborate? S3 state seems perfectly serviceable, and I don't
immediately see why I would want to operate on-prem resources just to maintain
state.

~~~
marenkay
S3 is Amazon. In some cases you want to exert control over your state without
being dependent on external parties.

Currently Terraform Enterprise is the only approach to solving that and its a
good one but... only if you're one of the big guns and even those sometimes
think twice because the infrastructure state requires much more security than
that offers.

~~~
NikolaeVarius
There is nothing stopping you from backing up state in s3 and git. Just have
CICD to sync it up.

State really shouldn't have anything secret in it in the first place, so 3rd
party having access to it shouldn't matter.

------
robfig
Question that's been bugging me and I haven't quite wrapped my head around --

For light usage, I find managing Terraform's state to be a significant hurdle.
You basically have no choice but to set up secure remote storage unless you
want to check passwords into source control. In contrast `kubectl apply` is so
easy to use since it's stateless. It just creates or updates any resources
provided, and it even supports --prune if you want the set of configuration to
be treated as comprehensive.

It seems like the main things that Terraform adds:

1\. The ability to work with providers that require you to store their
generated IDs to reference later. With kubernetes, the kind and name of the
resource is enough to identify it; it does get assigned a UID, but you don't
have to include that in the configuration since keys that are excluded are
left as-is.

2\. The ability to work with multiple different providers. I'm not sure how
often you do have a single terraform project(is that the term?) with more than
one provider, but I guess using the same set of tools, even if the
configuration is provider-dependent, is nice.

Is that accurate? Does Terraform offer any other advantages?

If you were building a configuration mechanism for your system from scratch to
allow your users to configure it as code, would you make .a Terraform provider
over a command line tool that can apply [--prune] that same configuration?

~~~
Znafon
Terraform shine when you combine multiple providers, kubectl manages Kube but
Terraform helps you integrate K8s with the rest of your infrastructure
(monitoring, Sentry,GitHub and GitLab, etc.)

> I find managing Terraform's state to be a significant hurdle Have a look at
> [https://app.terraform.io/signup/account](https://app.terraform.io/signup/account),
> it's the free version of Terraform Enterprise and it makes managing your
> state very easy.

>If you were building a configuration mechanism for your system from scratch
to allow your users to configure it as code, would you make .a Terraform
provider over a command line tool that can apply [--prune] that same
configuration?

I would make a Terraform provider, you get plans, modules, the possibility to
integrate with other providers easily and creating a new Terraform provider is
actually very easy!

------
reilly3000
I've been chasing infrastructure-as-code for a while, but I keep getting
blocked by Terraform missing provider APIs (GCP Cloud Function Runtimes
outside Nodejs, most recently), Kubernetes complexity, and generally being too
lazy to sit down and bang out something in code that I can quickly iterate on
with a console or quick CLI commands. I've had some wins with Serverless
Framework, but outside that I've yet to see the payoff from the time I have
put towards it. Am I defective?

~~~
colechristensen
Just not working at a large enough scale.

Infrastructure as code becomes progressively more important as you add
resources until you're at the point where terraform does the job of a whole
team of administrators who would be spending all day clicking through UIs.

------
scarejunba
The upgrade path for this has been an absolute nightmare, made worse by the
fact that we sat on the `google` provider 1.16 (lots of GKE stuff like node
taints was moved out to `google-beta` in `google` 2.0 and TF 0.12 requires
`google` 2.5 at least). There isn't even a straightforward transformation for
many things. I get why it's this way, but brace yourself.

I also made the mistake of `terraform plan`ning and updating my code as I went
along. Just use `terraform validate`. Otherwise you're going to inadvertently
promote the statefile before you're done dealing with all the issues (and you
don't want that because it prevents you from aborting your upgrade and
switching back to 0.11 till you're all ready). Not a real problem because the
statefile is versioned but an annoyance nonetheless.

The type issues were mostly easy to deal with, except for where things that
were assignments now being blocks. For instance, look at this:

    
    
          master_authorized_networks_config {
            cidr_blocks {
              cidr_block = "10.0.0.0/8"
              display_name = "Example /8"
            }
            cidr_blocks {
              cidr_block = "10.1.2.3/32"
              display_name = "Example /32"
            }
          }
    
    

That looked like:

    
    
        master_authorized_networks_config = {
            cidr_blocks = [
              {
                cidr_block   = "10.0.0.0/8"
                display_name = "Example /8"
              },
              {
                cidr_block   = "10.1.2.3/32"
                display_name = "Example /32"
              },
            ]
          }
    

before and `terraform 0.12upgrade` isn't about to help you navigate this.
Especially if you previously assigned from a variable. In that case, it's
going to make this monstrosity of a `for_each` over that thing. Jesus Christ.

Still, I'm thrilled for the new stuff with the more type-safety. Not going to
complain. If this is the price, then I'll pay. I just wish they'd done more to
help the upgrade, but it's an 0.x release so fine.

------
Ycros
Good stuff. It looks like they've addressed a few of the ergonomic issues in
their language with this release. The old way of doing iteration and having to
use string interpolations just to reference variables were annoying.

~~~
mirceal
Hashicorp got it right with Vagrant and its Ruby-based DSL. At the end of the
day you could always drop down to a proper programming language and express
what you wanted if the DSL was lacking. Why they had to invent this HCL
monstrosity is beyond me.

------
SlowRobotAhead
I’m a little unclear on terraform in practice.

As you supposed to download this, use it’s language and syntax which is all
it’s own thing, to define you services and then export that to a YAML setup
that AWS CloudFormation (for example) is expecting?

I assume there are reasons I wouldn’t just define it myself in YAML directly?

~~~
pensatoio
As others have said, no exporting is involved. You write roughly-json-esque
code which you then apply to a cloud provider.

The state of the infrastructure is stored, ideally, in the cloud. You apply
your code during which terraform identifies which changes need to be made by
comparing the current state with your local changes and executes those changes
as you watch.

The end result is well-defined, testable, repeatable cloud infrastructure
provisioning.

Infrastructure as code. Store it in git. Profit.

~~~
nineteen999
Where do you store your state files? The only project I've worked on that used
Terraform stored state in a git repo, which seemed like a nightmare with
multiple people doing deploys to the same environment. Others have suggested
it would be better stored on a network accessible share.

~~~
haney
I use a versioned S3 bucket to store state in my project, terraform also
supports "locks" to avoid race conditions/parallel execution. I've set up
buildkite to run `terraform plan` and then if the output looks I can approve
the next step to run `terraform apply`.

Atlantis
([https://github.com/runatlantis/atlantis](https://github.com/runatlantis/atlantis))
is also a really cool project for managing teams working on terraform
projects.

------
ses1984
People often complain about HCL being very limiting, that they are sick of
templating HCL, etc.

Has anyone tried the approach of writing some go code that imports terraform,
rather than using the terraform CLI? This would give you the full power of
golang to set up your resources.

------
didip
First of all, I don't want to dismiss the fact that terraform is a great tool
and mostly on the right path.

But HCL is giving me Puppet DSL PTSD.

Folks at Hashicorp should just embed JS engine and let users write definitions
using a real language (JS).

It's totally doable using library such-as Otto.

------
anon1253
For a second I confused this with TerraGen
([https://planetside.co.uk/](https://planetside.co.uk/)) and got all excited
about reliving a nostalgic youth.

------
sriku
From a language and semantics perspective, any comparisons to Nix?

------
rossmohax
Language improvement is nowhere match to what Pululmi brings to the table
without sacrificing any features.

------
k__
Terraform still isn't stable?

On the other hand, I'm usong React-Native, lol

------
aynawn
Finally

------
zeeZ
> Terraform 0.12 is a major update

Looks like a minor update to me...

~~~
IloveHN84
Before stable version x.y.z, if x is 0, then y means major

~~~
rurounijones
I am astounded that after all this time it is still 0.y.z

I mean, it has been around for like 5+ years...

I am even more astounded that people are happy to use a product that by its
own definition is not stable.

Same happen in ruby a lot. You find a Gem that claims to follow semantic
versioning and it is still in 0.y.z after years of being used in production
which flies in the face of [https://semver.org/#how-do-i-know-when-to-
release-100](https://semver.org/#how-do-i-know-when-to-release-100)

~~~
hedwall
Where does Terraform claim to follow semver?

~~~
IloveHN84
Where doesn't it claim it? Using a versioning scheme like x.y.z _by default_
means semver

~~~
NikolaeVarius
No it doesn't. Why would you think it does?

------
techntoke
I moved away from Terraform a long time ago. Ansible was way more powerful,
handled errors and issues with state changes. Terraform was super picky about
how you had to operate, slow and HCL was a terrible markup language. I wasn't
really a huge fan of Ansible either though and more recently have been doing
things in regular shell/Bash scripts. Now with Kubernetes and service brokers,
there is no need for Terraform.

~~~
n42
Ansible can very easily, and often does, end up in situations where two runs
of the same playbook both have drastically different results. roles/playbooks
slowly become Bash scripts written in a YAML layer parsed as a Jinja2
template.. and the project turns into a mess of many layers of indirection.

it attempts to and encourages declarative configuration, but is very hard to
keep that way. it is difficult and requires determination to make Terraform do
something in a non-declarative fashion.

the end result is that when I look at a Terraform configuration I can very
easily tell what is going on, because the end result is exactly what I read.
where with Ansible, it very much depends on understanding the current state of
the server you are about to run this playbook on, and you just have to cross
your fingers and hope for the best.

~~~
aprdm
Maybe you never worked in a company with good ansible practices.

~~~
n42
following best practices with C will get you code without memory leaks or
overflow errors, yet here we are in 2019 still routinely dealing with those in
mission critical software written by people who understand what they are
doing.

Terraform is a tool designed to overcome the problems that "best practices"
will supposedly prevent you from introducing in your Ansible playbook.

don't get me wrong. Ansible is a wonderful tool for certain applications
(configuration management). I would just never use it to spin up
infrastructure again.

