
GKE On-Prem Alpha - charlieegan3
https://cloud.google.com/gke-on-prem/
======
al_james
So the big question is... Can we point this at aws and get Google to manage
our aws kubernetes Installs...

Given the massive shambles eks turned out to be, that would be great.

~~~
chumboslice
As of right now, the alpha release is supported on vSphere 6.5... more to come
later :)

------
deforciant
So the war for enterprise on-prem Kubernetes just intensified :) RedHat vs
Google? :] I wonder how is Rancher doing? I guess life will be harder too?

Timing seems to be important, just before OpenShift finishes absorbing
Tectonic (or maybe Tectonic is absorbing OpenShift) with their installer, UI
and billing services. I guess there's still time for Google to land some big
deals :)

~~~
bgracely
The roadmap for Tectonic and CoreOS integration into OpenShift can be found in
these videos:

\-
[https://www.youtube.com/watch?v=1AelNjx6BB4](https://www.youtube.com/watch?v=1AelNjx6BB4)
(OpenShift) \-
[https://www.youtube.com/watch?v=LJOm4JbF4eQ](https://www.youtube.com/watch?v=LJOm4JbF4eQ)
(Red Hat CoreOS)

OpenShift is the most mature Enterprise Kubernetes that can be deployed and
managed in any cloud environment (public or private). It's great to see
further validation that customers want to run applications in both existing
data center environments and public cloud environments.

------
westonh
Hey Weston here from the GKE On-Prem team. Wifi is spotty at the conference
but I'll try to answer questions as they come in.

~~~
joshwget
Will Container-Optimized OS be used as the operating system for on-prem? If
so, any plans to spin this off as a more general purpose OS now that it needs
to support on-prem use cases?

~~~
westonh
Right now we're working with a Google-hardened Ubuntu image; same one we use
for GKE. COS is still TBD.

------
nine_k
> With GKE On-Prem, you get the Google Kubernetes Engine (GKE) experience
> directly in your data center. A quick and simple install and upgrade
> experience that’s validated and tested by Google. _GKE On-Prem also
> registers your cluster with Google Cloud Console in order to have a single-
> pane-of-glass view for managing all your clusters._

That latter is an interesting way to mentally merge your local DC and Google
Cloud.

~~~
eicnix
It will get even more interesting if you get decent federation support between
your local GKE clusters and GCP GKE cluster allowing you to scale out to the
cloud if you run out of capacity in your local cluster.

~~~
nine_k
_That_ would be great!

Depends on the latency between your DC and Google's, and the cost of traffic
(which could be significant).

~~~
chumboslice
Which is where Google's network comes into play.

------
patrickg_zill
An admission that cloud is not necessarily the best answer for all use cases.

It is smart of Google to recognize that and yet still have a product for it.
It will tend to commoditize data centers as well.

------
wmf
Hey GKE team, can you give any details about how this works? What does it
assume/require from the underlying infrastructure?

~~~
westonh
We're going to give a breakout session that goes into more depth on Wednesday
@ 4:35pm. IO244.

Some quick details: It's a bit of a split between what GKE runs and what the
customer runs. Alpha runs on vSphere 6.5 and we're packing up a Google-
hardened OS in much the same way we package GKE for GCP. A lot of the
integrations for things like networking and storage will be coming from
partners. We'll also have remote mgmt capabilities so we can manage the
cluster's control plane in much the same way our SREs do for GKE.

~~~
ryanSrich
Will this be something like COS or even CoreOS? Also, I'm curious to hear more
about this part:

> GKE On-Prem has a fully integrated stack of hardened components, including
> OS, container runtime, Kubernetes, and the cloud to which it connects.

Which runtime are you shipping? CRI-O? What type of outgoing cloud connection
is that? I have so many questions. I'm actually at the conference this week if
you're willing to grab coffee.

~~~
westonh
Happy to chat more. DM @westonhutchins and we can setup a time.

------
rkeene2
I previously worked on a similar (non-Google) product that was similar -- it
was an "On-Premise Cloud" [0], where the cloud provider managed and owned all
the hardware and software, and the customer created workloads on it, and the
physical hardware was scaled up/down based on demand.

The product worked well, but I think there was an uphill battle in explaining
the mechanics of the arrangement to customers.

[0] [https://knightpoint.com/what-we-do/offerings/on-
premises/inf...](https://knightpoint.com/what-we-do/offerings/on-
premises/infrastructure-platform/)

~~~
chumboslice
I'm not sure this is akin to that. This is, for lack of a better term at the
moment, more of an enterprise Kubernetes distribution.

------
alxbog
Does it need some special hardware to run on?

~~~
karangoeluw
Not sure what you mean by "special hardware" but if your hardware is capable
of running vSphere, GKE On-Prem should work for you.

Quick note though - We are exploring additional options, such as bare metal
support, based on customer demand.

~~~
gigi930
So this is like Azure Stack with no limitation on hardware configurations?

------
karangoeluw
Hi all. Karan from the GKE On-Prem eng team here. Happy to answer any
questions you might have.

~~~
abrodersen
Which IaaS vendors are supported?

~~~
karangoeluw
We will support vSphere 6.5 in Alpha.

------
sofaofthedamned
This is awesome! One of my problems with showing clients the ability of GKE is
having to pay a lot of money to demonstrate.

How complete is this? Can I do the usual ingress/LB annotations for GKE and
apply them to an on-prem instance?

~~~
wmf
Since this runs on vSphere presumably it will cost far far more than public
GKE.

~~~
karangoeluw
We are supporting vSphere in Alpha. However, we are exploring additional
options, such as bare metal support, based on customer demand.

------
swozey
Will masters be on-prem or will the on-prem services communicate to masters at
GKE over the master-authorized network?

Excited to check this out, completely came out of left field.

~~~
westonh
Masters will run on-prem. We have connection agent that let's us securely talk
to the Kube API Server from GCP. We wanted to ensure that the cluster is fully
functional even if the connection goes down.

------
mdelder
Great to see GKE coming to enterprise datacenters! IBM has been very
successful with IBM Cloud Private ([https://github.com/IBM/deploy-ibm-cloud-
private](https://github.com/IBM/deploy-ibm-cloud-private)) bringing an
enterprise Kube distribution for VMWare/OpenStack/Bare Metal in enterprise
datacenters since last year. I love to see the momentum of another Kubernetes
distribution helping create the de-facto next generation of apps for all kinds
of use cases.

------
bhouston
What is the pricing?

~~~
westonh
Pricing will be announced at a later date.

~~~
bhouston
We have an in-house cloud running Kubernetes and we also use Google Cloud with
Kubernetes. It is scary to move to GKE in-prem if we do not know pricing.

------
praseodym
Is this an externally managed service that runs on your own hardware, or is it
packaged software that could run in a fully offline environment?

~~~
westonh
While the cluster can operate in a disconnected state, much of the
functionality is provided by the connection to GCP. Things like UI
integration, policy syncing, Stackdriver, etc. Our early focus is on
datacenters that have a connection to the internet. However, we're starting to
look a lot more are airgapped environments.

------
cdnsteve
Very excited to test this out, we have lots of use cases that will benefit
from it.

~~~
karangoeluw
Let me know if you need a contact at Google to get started.

------
polskibus
What's the difference in running GKE on-prem vs Kubernetes?

~~~
karangoeluw
GKE On-Prem eng here.

GKE On-Prem is packaged with upstream K8s. So for your team that currently
uses `kubectl` to deploy or manage workloads, there won't be any differences.

~~~
polskibus
What is the value prop then, for someone who already runs open source
Kubernetes on-prem? Why should he pay extra for GKE on-prem?

~~~
karangoeluw
Excellent question.

GKE On-Prem is a Google provided, validated and supported distribution of
Kubernetes and extensions that offer a GKE-like experience in your on-premise
datacenter. It makes it easy to install and upgrade Kubernetes and provides
access to GCP services such as monitoring, logging, metrics, security and
auditing for your on-premise installation. It is the foundational component of
the Cloud Services Platform, and is how Google "brings the cloud to you".

CSP combines Kubernetes both in your on-premise datacenter (GKE On-Prem) and
Google-managed Kubernetes in GCP (GKE) with Istio and other CI/CD (Cloud
Build) and serverless (Knative) products. You can leverage this suite of
products to both modernize your existing on-premise applications and build new
applications in the cloud.

Additionally, Google will be offering phone and email support similar to the
existing GCP support packages.

~~~
ironjunkie
What is the benefit, if you get the same output from "kubectl" as regular GKE,
or from any other distro.

Basically, this is yet another paid packaged Kubernetes distribution, that has
the explicit goal to do "Hybrid clustering" so that it is easier to lure the
customer back to GKE. Do I get that right ?

~~~
karangoeluw
What we have found out is that most on-prem customers are eager to move to the
cloud. Practically it's not easy to just lift-and-shift. So think of this is a
ramp to the cloud.

Now, the benefit of upstream K8s is that your dev team can build apps and
containers without proprietary APIs; so when you are ready to move to the
cloud you are not locked-in.

~~~
ironjunkie
Thanks. I agree that lift and shift never happens easily in real life.

That being said, why would I not use the actual free upstream Kubernetes for
my on-prem distribution ? (with the help of one of the thousands installer out
there like kube-adm, kubespray, etc).

What I have seen working with Kubernetes for quite a while, is that the lowest
common demominator is the YAML definitions for your workloads (what you want
to run on your Kubernetes cluster). Those should be portable accross any
Kubernetes distribution, on-prem or on the cloud. As far as I can tell, today
this is already the case.

Is the benefit in this case that you can use the Google ecosystem for logs etc
?

~~~
karangoeluw
> That being said, why would I not use the actual free upstream Kubernetes for
> my on-prem distribution ? (with the help of one of the thousands installer
> out there like kube-adm, kubespray, etc).

None of them actually provision your infra for you (VMs, LB rules etc). GKE
On-Prem will.

~~~
ironjunkie
Ok, thanks I got it, you are bundling everything into a VMWare image, that
boots ready to use.

Is it fair to say that this is similar to Canonical Ubuntu MAAS + Juju
Kubernetes? I'm sure that Red Hat Openshift must have something similar also
to install directly on a pool of managed bare metals.

~~~
karangoeluw
I don't know enough about Juju K8s to really say similar it is. GKE On-Prem
will be an OS image, set of containers, config, tooling and support.

------
mt42or
Does it provide a way to install k8s nodes on a VMware cluster ?

~~~
karangoeluw
GKE On-Prem is a full K8s installation - so nodes and masters. We will support
vSphere 6.5 in Alpha.

~~~
stre
Is vSphere Essentials Plus supported as we have a good use case. Are there any
specific additional requirements eg. vsan or does it all run on the base Esxi
product?

------
ferest
would there still be chance for startups doing similar thing ?

~~~
ironjunkie
there are literally 60+ startup distributions that try to sell a "distribution
as a service" for the upstream code that is free.

I think most if not all of them will fail, and as usual, the big 3 or 4 will
win the market (if I had to bet: Google, Red hat, Canonical and maybe the guys
at Heptio that are really cool and got the right attitude)

