
Kustomize – Templating in Kubernetes - davinkevin
https://blog.stack-labs.com/code/kustomize-101/
======
Benjamin_Dobell
There seems to be some confusion here, in no small part due to the title of
this article. But...

> _kustomize lets you customize raw, template-free YAML files for multiple
> purposes, leaving the original YAML untouched and usable as is._

That's the very first line of Kustomize's README[1].

Kustomize is _not_ a templating DSL. The conversations here mocking templating
DSLs are not relevant.

The next sentence[1]:

> _kustomize targets kubernetes; it understands and can patch kubernetes style
> API objects. It 's like make, in that what it does is declared in a file,
> and it's like sed, in that it emits editted text._

Kustomize is a _patching_ framework. It takes valid k8s resources, and allows
you patch them, with _partial_ k8s resources. The entire point of Kustomize to
_not_ invent something new.

[1] [https://github.com/kubernetes-
sigs/kustomize](https://github.com/kubernetes-sigs/kustomize)

~~~
haolez
> It's like make, in that what it does is declared in a file, and it's like
> sed, in that it emits editted text.

Is that different from a templating engine?

~~~
Benjamin_Dobell
Yes[1].

[1]
[https://en.wikipedia.org/wiki/Template_processor](https://en.wikipedia.org/wiki/Template_processor)

------
mkobit
`kustomize` is an interesting and useful tool, and it is cool to see it
available directly in `kubectl` in 1.14. A lower barrier to entry is nice to
have for tools like this. It is useful on its own, but I'm also looking
forward to additional features like being able to reuse a single patch on
multiple targets ([https://github.com/kubernetes-
sigs/kustomize/issues/720](https://github.com/kubernetes-
sigs/kustomize/issues/720)). I'd also like to see a clear schema for what the
YAML needs to look like and what the keys do. I wish it folowwed a similar
pattern as `kubernetes` resources where each resource had an `apiVersion`, and
the `kustomization` itself had an `apiVersion`.

Some newer documentation can be found at
[https://kubectl.docs.kubernetes.io/](https://kubectl.docs.kubernetes.io/)
which is still a little bit barren, but I expect it to improve. Some of the
older documentation can be found at [https://github.com/kubernetes-
sigs/kustomize/tree/a5bb5479fb...](https://github.com/kubernetes-
sigs/kustomize/tree/a5bb5479fbd50516a80fe1e39dec07612b90a0e4/docs) (before the
tool was integrated directly as a subcommand)

~~~
jcastro
Joe Beda (one of the k8s cofounders) just did a livestream on kustomize last
week and I learned a bunch of stuff:
[https://www.youtube.com/watch?v=NFnpUlt0IuM](https://www.youtube.com/watch?v=NFnpUlt0IuM)

Full Disclosure: I work with Joe @ VMware

------
GauntletWizard
I have to bring up Jsonnet [1] in every one of these discussions. Jsonnet
isn't the tool you want for managing a complex stack of configurable objects,
but it is the tool you need. Jsonnet gets big and ungly and unweildy at a
certain point, and that is also coincidentally the point where you need to
have rethought your model and refactored your config, just as much as your
code.

Simple yaml is just fine, even repetitive simple yaml, and you're often better
off building tooling to make repetitive updates rather than condense your
config. Jsonnet is for building when you've got dozens rather than handfuls of
objects.

Well defined Kubernetes objects don't actually take that much configuration,
don't generally need templating. Typically, my production and staging
environments are precisely the same deployment definition, except image and
replicas. Replicas isn't defined in the document - it's part of the working
state rather than something that gets "configured" (i.e. that field is owned
by an HPA, even if it's min and max values are the same)

All configuration is stored in Configmaps and secrets. Those are more likely
to be templated, but still probably not. If two values are the same in
production and testing (and they should be - service names are the same, with
namespace defining where they point), why configure it? Use a sane default.

[1][https://jsonnet.org/articles/kubernetes.html](https://jsonnet.org/articles/kubernetes.html)

------
markbnj
We built and deployed a new pipeline based on kustomize and have been
gradually moving some helm things over to it. The feedback from app developers
is quite good, and there's no question that it is much less obfuscating of the
underlying resources than helm is (if you follow all the chart conventions,
etc.). And then as noted in other replies it is in kubectl as of 1.14... which
I guess doesn't mean that much except it feels sort of like canonizing the
approach.

That said its design is quite opinionated and committed to a declarative
model, and there are some things you just can't do without falling back on
either generating a patch or performing unstructured edits. A good example is
something as simple as tagging an image with a string that isn't known until
build time (such as the commit sha). Another is shared configuration. You
can't glob object names and apply the same environment patch, for example, to
more than one resource. These constraints can be considered features, but
they're nonetheless constraints.

------
outside1234
Another interesting project in this space is Microsoft’s Fabrikate
([https://github.com/Microsoft/fabrikate](https://github.com/Microsoft/fabrikate)).
I like that approach better in that it separates config from resource
manifests and allows you to establish higher level components from lower level
ones.

------
sheeshkebab
Yaml engineering - now with inheritance, encapsulation, and polymorphism.
Soon, we’ll move on to functional and immutable configs augmented with
environment variables.

Good times.

~~~
gilbetron
I felt a part of my soul drift away when I saw an ex-coworker put a for loop
in a yaml file. Joe Beda (Heptio) talks about the problem here:
[https://www.youtube.com/watch?v=M_rxPPLG8pU&t=2960s](https://www.youtube.com/watch?v=M_rxPPLG8pU&t=2960s)

Just use Python (or Lua if you want something simpler/lighter). Or, like
Brigade, even JS. Hell, use BASIC. But these bizarre template-language-DSL
bastardizations are horrific when it comes to maintainability. Incremental
evolution from a config file to a full language is a path wrought with peril,
and only ends in damnation!

~~~
sciyoshi
Agreed on avoiding yaml templating at all costs, but I've also found that you
don't normally need a full programming language for these types of config
files - if you do find yourself reaching for those tools you might be better
served by something like Dhall [https://github.com/dhall-lang/dhall-
lang](https://github.com/dhall-lang/dhall-lang) (which has its own K8s
bindings as well)

~~~
mkobit
YAML templating is something I've come to truly hate. It seems that tools and
services want the configuration to be "simple" by using a format that is
fairly easy for humans to parse while not trying to force a language specific
tool onto the users. However, when they move into the inevitable phase of
wanting to be more configurable, they can move into confusing, unintuitive,
and surprising territory. It also doesn't help that everybody keeps inventing
their own way to do it. A few examples of YAML bastardization that give me
headaches trying to understand:

* saltstack Jinja templating [1]

* GitLab CI `include` directive for including and merging external CI files together [2]

A few tools that I have seen take a decently pragmatic approach:

* Kubernetes resources that use ConfigMap and `envFrom` that declaratively say where to resolve a value from [3]

* Circle CI commands which offer some reusability with its "commands" and "executors" type features [4]. To me, Circle CI has both good and bad aspects with some templating and some clever patterns

On the other side of things, there is essentially fully programmable type
configurations like Jenkins Pipeline Groovy `Jenkinsfile`, which can be a
nightmare, too.

I think it is tough to find a sweet spot between making it configurable and
expressive for users while retaining a low barrier of entry and not turning
the configuration into a complete program itself. Tools like Terraform are
trying to find that sweet spot as they slowly introduce more programmatic ways
of configuration while still being declarative, like the fairly recent
introduction of if statements and soon (I think) to be released for loops. As
soon as users of a tool and service have more complex use cases, there needs
to be some way to solve that. The most common way (it seems) is taking the
easy and familiar of introducing templating.

[1]:
[https://docs.saltstack.com/en/latest/topics/jinja/index.html](https://docs.saltstack.com/en/latest/topics/jinja/index.html)

[2]:
[https://docs.gitlab.com/ee/ci/yaml/#include](https://docs.gitlab.com/ee/ci/yaml/#include)

[3]: [https://kubernetes.io/docs/reference/generated/kubernetes-
ap...](https://kubernetes.io/docs/reference/generated/kubernetes-
api/v1.14/#envfromsource-v1-core)

[4]: [https://circleci.com/docs/2.0/configuration-
reference/#comma...](https://circleci.com/docs/2.0/configuration-
reference/#commands-requires-version-21)

~~~
vorg
> there is essentially fully programmable type configurations like Jenkins
> Pipeline Groovy `Jenkinsfile`, which can be a nightmare, too. I think it is
> tough to find a sweet spot between making it configurable and expressive for
> users while retaining a low barrier of entry and not turning the
> configuration into a complete program itself.

Nowadays, Jenkins pipelines can be configured in either the Jenkins-provided
"Declarative Syntax" [1] or the Apache Groovy-based "Scripted Syntax", with
the Declarative Syntax used as the default for examples on the Jenkins
website. I guess they've found the best way to not have users turn the
configuration into a complete program is to provide declarative syntax only in
the default option. It's good to see Kustomize is built with this in mind,
too.

[1]:
[https://jenkins.io/doc/book/pipeline/syntax/](https://jenkins.io/doc/book/pipeline/syntax/)

------
founderling
Often when I talk to founders who use Kubernetes, they could simply use docker
and a bit of shell scripting instead.

Those of you who use Kubernetes: What is a functionality it brings to the
table you would miss if you automated your docker handling without it?

~~~
hueving
Resume boosting. Using shell scripts is lame and straight forward. If you have
a declarative configuration though you can scale to thousands of nodes with
nary a thought.

~~~
marcinzm
If you have a single service that runs of thousands of nodes then okay. If you
have dozens of services that run across even dozens of nodes, that interact
with each other, that have resource constraints, that need rolling updates,
depend on external volumes, require security policies, etc. then you
eventually recreate enough k8s to make it simpler to just use k8s.

------
kjgkjhfkjf
Consider instead writing a small program in your favorite programming language
that generates the YAML, and then piping the output of that into `kubectl
apply`.

~~~
jazoom
Why would I want to write my own Kustomize when this one already exists?

~~~
Benjamin_Dobell
Yeah, I _replaced_ a custom resource patching solution with Kustomize. I was
very concerned my custom solution would inhibit on-boarding, and wasn't
thrilled with Helm. Back in November I was pretty happy when I stumbled across
Kustomize, it _claimed_ to do exactly what I wanted it to do. In reality there
were a few hiccups, but it's certainly moving in the right direction.

Granted, I still do have a bunch of "opinionated" scripts, but they delegate
the heavy lifting to Kustomize, and all the configuration itself is just
stock-standard k8s resources.

------
bobske4
A DevOps guy walks into a bar. He replaces the bartender with a docker-
container then replaces the whole bar with Kubernetes. He sits down and orders
one drink.

------
vbsteven
I was looking at Helm the other day for merging common base values with per-
cluster specific overrides but this looks like a simpler solution and
integrated into kubectl.

~~~
dilyevsky
Which part of helm (I’m assuming v2) is integrated into kubectl? It has its
own client and installs its own server component (tiller) which is difficult
to secure.

~~~
dlor
I think the op is saying that kustomize is integrated into kubectl, not helm.

------
dilyevsky
Apparently solution to heaps of unmanageable yaml is... templated unmanageable
yaml

------
dqpb
There really isn't any need to templatize kubernetes. They have auto-generated
client libraries for most popular languages which let you define
configurations as code, and serialize to json/yaml of that's what you want.

~~~
shaklee3
Can you give some examples? I don't understand what you mean by autogenerated
client libraries, unless you mean the python and go clients. Those aren't for
kubernetes yamls, though.

~~~
dilyevsky
Yeah client-go typed clientsets are codegened. It would be really labor
intensive (and error prone) to write go code for every deployment.

May I suggest a middle ground =) github.com/stripe/skycfg (checkout
//_examples/k8s that I added)

~~~
happythought
I’m interested in learning that model, but a readme in there that explains how
to go from nothing to a deployment might help.

Edit: also the fact that you have a 4 month old outstanding pull request isn’t
giving me any warm fuzzies about the maintenance and velocity of skycfg.

~~~
dilyevsky
nginx.cfg and app.cfg have an example deployment for nginx and the rest is
covered in main README.

Yeah not sure what is going on with that pr, tbh i forgot about it b/c we
implement it in our own runtime. I got a bunch of others merged quickly since
then.

~~~
happythought
Thanks, I saw those and they seem clear enough. So the go code is unrelated to
nginx.cfg and app.cfg?

~~~
dilyevsky
The go code under _examples/k8s is a demo client that will actually stuff
`main` output into Kubernetes api endpoint.

