
A Year of Helping Customers Build Production-Ready Kubernetes Infrastructure - migueloller
https://www.pulumi.com/blog/crosswalk-kubernetes/
======
antics
Oh wow! I co-lead these efforts at Pulumi. I don't think we expected this to
hit HN! :) Happy to answer questions about any of this.

Parenthetical, one thing that didn't make it into this post that I'm
particularly optimistic about is the `watch` command, which will continuously
watch your code and deploy it to (say) Kubernetes. It looks like this[1]. This
is very handy when you're trying to build a thing that has to integrate with a
bunch of other services, but there are too many to develop locally on
minikube. It basically lets you see instantly if something crashes, or
whatever. In general I'm very optimistic about that sort of experience—another
product in that space is Tilt[4] which I am personally a big fan of.

Another thing that gets kind of a passing mention that is cooler than it looks
here is our streaming `query` stuff. By exposing Kubernetes `watch`es as
streaming queries, it becomes very easy to write very cool apps like this
one[2] , which helps you debug services. You can actually see a bug in the
Deployment controller there—we create two `Deployment`s with the same labels,
and deleting one causes the controller to delete _all_ the pods. This would
generate a transient 500 for any request hitting that service. I think there
are probably lots of bugs in Kubernetes like this, but people don't notice
them because this level of visibility previously was quite hard. :) To get an
idea of how hard that sort of app is to write, you can see the code we wrote
for the prototype of this SDK here[3].

[1]:
[https://twitter.com/funcofjoe/status/1195411343966236677?s=2...](https://twitter.com/funcofjoe/status/1195411343966236677?s=21)

[2]:
[https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdM...](https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdMVIXzOD)

[3]: [https://github.com/carbonql/carbon-
ts/blob/master/examples/u...](https://github.com/carbonql/carbon-
ts/blob/master/examples/unix/serviceMonitor/serviceMonitor.ts)

[4]: [https://tilt.dev/](https://tilt.dev/)

~~~
migueloller
I’ve been very impressed with what you guys are doing. We’re extremely excited
about moving from our collection of YAML, bash, and Helm templates to Pulumi.
The big question mark we had was what local development would look like. Tilt
and Skaffold were the two big candidates we looked at for sane local
development with Kubernetes. Looks like Pulumi’s new `watch` could give us
what we need?

Are there any plans to have something like rsync support to enable things like
hot-reloading apps? Both Tilt and Skaffold have examples showing this
capability.

~~~
antics
We have a Docker[1] library which allows you to build and push Docker a
container—you can plug that directly into the `image` field and it should
work.

I'm _guessing_ that eventually `watch` will eventually make this more
automated, but we don't have any specific plans for that, yet.

[1]: [https://github.com/pulumi/pulumi-
docker](https://github.com/pulumi/pulumi-docker)

~~~
migueloller
I’ll look into this. What I think Tilt and Skaffold are doing is just setting
up a persistent volume to a local directory and letting all the reloading
stuff be handled by the end-user. As long as the persistent volume links the
local files to the cluster it should be good to do.

~~~
antics
Yeah, Tilt has a couple of different ways of providing similar functionality.
In one mode, they simply rebuild the container and ship update Kubernetes on
each file modification. In another mode, they boot a dev container in the pod
and then transparently sync the local FS content to the cluster, which means
that "time to live" is the amount of time it takes to compile your app in the
cluster. `watch` with the Docker library is a direct analog for the first, but
we have no analog for the second.

~~~
migueloller
Gotcha, so things like hot-reloading wouldn’t work right now. Still, looks
like y’all are going in the right direction. Thanks for all the details! One
more question, is there a way to interop Pulumi with Skaffold or Tilt?

------
rawoke083600
How many of your customers actually NEEDED kubernetes ?

~~~
MadWombat
_shrug_ If you have a bunch of services that you need to reliably run in
containers you are going to need an orchestration of some sort. You can
attempt to reinvent the wheel by doing your own or you can use an existing
solution.

If you are not in the mood for wheel reinventing, your choice becomes somewhat
limited. You have Kubernetes, Docker Swarm, Mesos, Nomad, a few platform
specific solutions like Elastic Beanstalk and maybe some more obscure
candidates.

If you don't like vendor lock-in, don't like living on the cutting edge and
don't want to write too much code, your choices are down to Kubernetes, Swarm
and Mesos.

~~~
antics
[Pulumi kube co-lead here]

I agree with most of this, but would go further. My experience (here and at
Heptio) is that no one _chooses_ Kubernetes. They choose Docker, and then they
(i.e., the infra team) effectively have to choose ECS or Kubernetes. In
contrast, what most app developers want is a better `docker run`.

I don't know how controversial that is, so I should say this is my conclusion
from ~50 intensive customer qualification conversations in my 1.5 years at
Pulumi and 1 year at Heptio. In each of these conversations I ask why the
customer is moving to Kubernetes. In mid- or large-sized companies (i.e., 500+
employees) the answer is completely unwavering: Docker makes it easy to ship
software without worrying too much about the host, and once you have that,
having a way to schedule containers to run _somewhere_ is very useful. And
that's effectively how the decision is made IME.

~~~
7532yahoogmail
I am very much hunting for a cli, automatable method to do:

\- app installs

\- orchestration of same across multiple boxes in a cluster

\- cluster monitoring: detect when a task or box goes down running the
service(s) elsewhere in the cluster

It seems like kubernetes is incomplete with respect to these needs. Kubernetes
depends on docker but neither does orchestration. In the context of this HN
post what do I need. I am doing golang on 100pct Linux. Other than toolng I
need for app installs the boxes themselves are provisioned by others. Here I
am focused on app installs only.

Any help much, much appreciated.

~~~
dankohn1
Take a look at Kubernetes, Helm, and the Cluster API.

~~~
Terretta
Can’t tell if grandparent post wants to manage apps on a cluster (see below)
or apps that _are_ a cluster (see [https://cluster-
api.sigs.k8s.io/](https://cluster-api.sigs.k8s.io/))

If it’s letting a dev manage apps on a cluster someone else let them put apps
on, while staying in dev mode, maybe devs could appreciate the cluster
provisioning SREs enabling app devs with so-called GitOps or AutoDevOps
approaches, prescriptivist ways of leveraging Docker, K8s, and Helm, such as:

[https://www.weave.works/blog/what-is-gitops-
really](https://www.weave.works/blog/what-is-gitops-really)

[https://docs.gitlab.com/ee/topics/autodevops/index.html](https://docs.gitlab.com/ee/topics/autodevops/index.html)

------
cbushko
Any tool that helps build infrastructure as code is great news. I hear good
things about pulumi.

In my case I have been moving our current infrastructure from AWS to GCP.
After much evaluation and a small stint with Helm, we decided to go 100%
terraform.

There is more to running a product than just deploying to kubernetes and
terraform providers have given us the flexibility to chain all these resources
together. For example, I can create a GCP service account (google cloud
provider), save its credentials as a kubernetes secret (kubernetes provider),
register a name in route 53 (aws provider) and deploy our product in
kubernetes (kubernetes provider).

Having it all in one tool that has state is really powerful and more
importantly follow the same patterns.

Obviously, I am a big fan of terraform

~~~
jen20
Everything you describe is also straightforward with Pulumi using `pulumi-gcp`
[1], `pulumi-aws` [2] and friends - Kubernetes is one of many libraries.

On top of this, you can build actual abstractions thanks to the configuration
being written using a general purpose programming language - I don't think
there is a first-party set of abstractions for GKE yet but there is for EKS
[3].

Many of the most compelling examples deploy entire infrastructure stacks from
"empty account" to multi-cloud Kubernetes.

Disclaimer: I have worked extensively on Terraform (both core and providers)
at and after HC, and have also contributed to Pulumi since it was open
sourced. Edited for formatting.

[1]: [https://github.com/pulumi/pulumi-gcp](https://github.com/pulumi/pulumi-
gcp)

[2]: [https://github.com/pulumi/pulumi-aws](https://github.com/pulumi/pulumi-
aws)

[3]: [https://github.com/pulumi/pulumi-eks](https://github.com/pulumi/pulumi-
eks)

------
houqp
Great write up, the experience of managing k8s end to end with pulumi is so
much better compared to terrafom.

------
reilly3000
One thing I'm murky on with Pulumi is around how to execute proper blue/green
or canary deployments with a GitOps approach. Any insights?

------
bsingh4
For AWS users AWS CDK is a far better take on infrastructure as code -
strongly typed TypeScript resources and higher level constructs that hide the
underlying details of load balancers, IAM, etc. Pulumi was not as useful the
last time I tried it - most resources were string key-value pairs with no
typing or code completion. You're still referencing documentation to do
anything and defining tons of low level resources to accomplish basic tasks.
Might as well be writing raw Terraform or CloudFormation configs at that
point, as they have more documentation available.

~~~
reilly3000
Something was wrong w/ your setup I think. All of their code has Typescript
annotations, and their autocomplete w/ intellisense is remarkably useful.

~~~
bsingh4
Here are two simple database definitions:

Database in Pulumi:

    
    
      const db = new gcp.sql.DatabaseInstance('app-db', {
        databaseVersion: 'POSTGRES_11',
        region: 'us-central1',
        settings: {
          tier: 'db-f1-micro',
          diskType: 'PD_HDD',
        },
      });
    

Database in AWS CDK:

    
    
      const db = new rds.DatabaseInstance(this, 'AppDb', {
        engine: rds.DatabaseInstanceEngine.POSTGRES,
        engineVersion: '11.5',
        instanceClass: ec2.InstanceType.of(
          ec2.InstanceClass.STANDARD5,
          ec2.InstanceSize.XLARGE
        ),
        storageType: rds.StorageType.GP2
      })
    

Not to mention they have higher level constructs like DatabaseCluster that
will handle the underlying details of replicas and create the lower level
resources for you.

~~~
antics
`DatabaseInstance` not only has TypeScript type definitions, it's actually
written in pure TypeScript: [https://github.com/pulumi/pulumi-
gcp/blob/master/sdk/nodejs/...](https://github.com/pulumi/pulumi-
gcp/blob/master/sdk/nodejs/sql/databaseInstance.ts)

Pulumi also does have a number of higher-level constructs, which you can learn
about here:
[https://www.pulumi.com/docs/guides/crosswalk/aws/](https://www.pulumi.com/docs/guides/crosswalk/aws/)

~~~
bsingh4
Why is every single property of type string - to see what to put there I have
to reference a comment in code? I shouldn't have to read comments to see what
the magic string is for the f1-micro tier, or the hard disk type or the
region. CDK has actual Enums for all of these things that work to make life
easier. You're not making it any easier to code up a DatabaseInstance - its
simply a bucket of keys and values where I'm looking up what those values
should be.

~~~
antics
Ah—unions of strings (e.g., `"foo" | "bar"`) effectively _are_ enums in
TypeScript and JavaScript. They are autocompleted and type-checked at compile
time, so you should get exactly the same behavior. Even exhaustiveness
checking. See docs[1] for details.

And, to the specific point: you absolutely should _not_ have to look up the
docs. These values definitely will autocomplete.

Anyway, aside from all that, unions of strings is "the idiomatic way" to do
this sort of thing in TypeScript and JavaScript.

[1]: [https://www.typescriptlang.org/docs/handbook/advanced-
types....](https://www.typescriptlang.org/docs/handbook/advanced-types.html)

~~~
bsingh4
None of the properties in the DatabaseInstance construct example posted above
are defined as unions of strings - the TypeScript documentation link you just
posted is irrelevant.

This is what one of the property types actually look like:

    
    
        /**
         * The MySQL or PostgreSQL version to
         * use. Can be `MYSQL_5_6`, `MYSQL_5_7`, `POSTGRES_9_6` or `POSTGRES_11` (beta) for second-generation
         * instances, or `MYSQL_5_5` or `MYSQL_5_6` for first-generation instances.
         * See [Second Generation Capabilities](https://cloud.google.com/sql/docs/1st-2nd-gen-differences)
         * for more information.
         */
        readonly databaseVersion?: pulumi.Input<string>;
    

In AWS CDK it was an easy task to select Postgres - in Pulumi case I had to
read the comment.

~~~
antics
Oh, I see. Some properties that are not as well-typed as they could be, that's
true. We are working on that. But there are lots of properties that do have
good typing. Instance types, for example.

