
The full-time job of keeping up with Kubernetes - twakefield
https://gravitational.com/blog/kubernetes-release-cycle/#
======
tw1010
There aught to be a name to the tendency that as tools get better and better,
the more your time goes from having your mind in technical-space to social and
news-space. It's like the authority to create goes from the individual first-
principles (by necessity) maker, to the control over development being in the
hands of an external group, and then all your time is spent keeping up with
what they're doing. A similar thing happened with a lot of javascript
frameworks. It also happened with the transition from building servers from
the ground up, to it all being managed by AWS.

~~~
lkrubner
I wish I could give you more upvotes. You are describing a somewhat hidden
psychology, which I think provides a rational basis for much of "Not Invented
Here" psychology. We tend to think that "Not Invented Here" psychology is
irrational, but in fact, the loss of control over possibly crucial technology
is an important cost, which makes all of us stop and re-consider whether we
really want to use some software developed by an external team.

~~~
jstimpfle
And it's not only that - Time spent not doing our own designs (and instead
spent memorizing how to use magical frameworks) is time not spent advancing
our technical understanding.

It's a sad state when otherwise very intelligent people think it's bad
practice to use plain C and a clean OS API that has been stable for decades
(because that was somehow "magic" und impossible to understand and error-
prone), and advise you to use the Boost filesystem module just to concatenate
two paths.

~~~
robocat
Concatenating two paths is difficult if you care about any of the following:
security, multiple OS's, Unicode (multiple code units, validity, combining
characters), file system restrictions, etc.

Your "two decades" maybe holds for Linux, but what about Windows or MacOS???!

I have seen too many people use string concatenation.

I think an intelligent person would recommend to use the normal library
(appropriate for your language, assuming it is well written) since usually
your program will be doing many other filename/path manipulations too.

~~~
jstimpfle
[https://msdn.microsoft.com/de-
de/library/windows/desktop/bb7...](https://msdn.microsoft.com/de-
de/library/windows/desktop/bb773571\(v=vs.85\).aspx)

~~~
robocat
Exactly my point - that function doesn't deal with modern windows. MAX_PATH is
260 characters. Yet Windows now supports longer paths (\\\?\ prefix). So I
presume there is another Windows function to combine long path names (and
maybe canonicalise Unicode better).

~~~
jstimpfle
> Yet Windows now supports longer paths (\\\?\ prefix).

I know. And this is the attitude that leads to complex software. It's a self-
fulfilling prophecy: "I can't do it on my own since the problem is so
complex". In which case the problem _does_ get complex.

I recommend not wasting time supporting this crazy feature (unless you are
writing infrastructure code for tools and you're required to - in which case
I'm sorry). 260 character paths are more than enough for any project. And I
certainly recommend against using the crazy abstractions from
Boost::filesystem.

(Personally I think it's an unfortunate example. I'd prefer to avoid paths
from the start, since nobody understands the semantics of hierarchical
filesystems. Alas, typically you need to deal with them to _some_ degree).

~~~
robocat
It is not a crazy feature.

It is a necessary feature for many development environments originally written
for Unix (e.g. nodejs), because Unix file systems usually don't have such a
low character limit.

Or maybe Windows has mounted or network sharing, a Unix filesystem... In which
case your program better deal with long paths (or just fail?)

Really... You are showing exactly why one uses a library so one doesn't need
to care about the "complex" details because one hopes the library does a good
job of managing it as well as possible for you (although you still need to
know the details to use a library function safely).

~~~
jstimpfle
> Or maybe Windows has mounted or network sharing, a Unix filesystem... In
> which case your program better deal with long paths (or just fail?)

260 bytes _is_ long. I've never seen a \\\?\ path in the wild, and I don't
want to. It's a misdesign (well, I'm sure the designer didn't _want_ to design
it...). But I'm repeating myself... I can't practically deal with paths longer
than, say, 64 characters (can't read them, let alone type them). Fix your
paths.

And again, this is off-topic. This thread was about not using a crazy third
party library, when the authoritative semantics (however insane) are contained
in the OS interface.

~~~
Klathmon
That is such an awful toxic viewpoint to take.

"Fix my paths"!? Fix your software! I work from a network share that is about
70 chars long to get to my one project's folder. The beauty of hierarchical
file systems is that I can cognitively ignore all the previous paths and just
work from there. However when your software doesn't work because "the solution
is a misdesign", it's not my path that is wrong, it's your software that is
broken and you that are stubbornly refusing to use solutions to well known
problems.

------
hueving
This reads like a giant ad for GKE. It emphasizes several times to just use
GKE for pretty lame reasons (Google has good SREs and Google started the
project).

The people that work on upstream k8s in Google (Tim et al) have a pretty
limited overlap with the Google Cloud people that run GKE. Upstream k8s is a
full time job so they are most certainly not spending their time also writing
internal GKE code.

I don't have an issue with GKE, but this article uses little evidence to
recommend it when it seems the conclusion should have been "maintaining a k8s
cluster requires a full time sysadmin. If your company has a culture of
pretending sysadmins are pointless, then you should pay another company
offering k8s sysadmin-as-a-service hosted on their hardware."

~~~
aberoham
Author here. I do have a bias toward GKE but that is after trying to use
everything else -- DIY, kubespray, OpenShift, Juju, Tectonic, Azure, Fargate,
kops, Rancher v2 preview, kubeadm, etc. In my experience nothing (yet) has
been as nice and clean as Google Cloud.

~~~
iooi
Have you tried EKS?

~~~
emmelaich
EKS = Amazon Kubernetes.

(I had to look it up)

------
mbrumlow
Every time a new framework or tool comes out and everybody jumps on it. I
always wonder if somebody will realize that you are trading one set of
problems and work for another.

As engineers we really need to stop supporting these sort of effort and take
the time to help each other become better engineers that write and maintain
our own code. We need to promote learning and mastering the underlying
concepts that things like kubernetes tries to hide and shield engineers from.

In most cases tools like kubernetes are so vast and huge so they can be the
solution looking for many problems.

It is also curious how once kubernetes became big how many small shops needed
"Google" level orchestration to manage a hand full of systems. And how hard
people ripped their software stack apart into many many micro services just to
increase the container count.

I think if most engineers took a step back and said "I don't know" and took
some time to truly understand the requirements of the project they are working
on they would find a truly elegant and maintainable solution that did not
require tweaking your problem to fit a given solution.

Every tool and library / dependency you add to your solution is only adding
more code and complications that you will not be a expert in and one day will
find your self at the whim of the provider.

Far to often do we include tens of thousands of lines of code of somebody
else's work all for a handful of lines of code that if somebody would have had
the confidence and support from other engineers around to try and truly
understand the problem domain could have implemented and owned the solution.

The general trend I see as I get older is that we are valuing the first to a
solution rather than a more correct solution. Only to be stuck with a solution
that requires constant care and work around.

So I plead to all engineers, devlopers, programmers or whatever you call your
self. Please stop and take a moment and think hard about how you would solve
any given problem without the use of external code first. Then compare your
solution to the off the shelf "solution looking for a problem". You might
surprise your self.

I will also like to point out that if when solving a problem your solution
looks like a shopping list of third party tools libraries and services; you
might not fully understand the problem domain.

\-- sorry for the rant --

~~~
gouggoug
I don't think your comment applies at all to Kubernetes.

K8s truly simplifies dev-ops and even the smallest team and website can
greatly benefit from it.

I speak from 7 years of experience managing my company's infrastructure's
website:

Before kubernetes, I ran my company's stack on very cheap bare metal from OVH.
It was great while it lasted, but as the company grew and my team grew, it's
become harder and harder to maintain this infrastructure. And I'm not only
talking about our production servers. In reality you have to maintain your
prod, your staging, and, worst of all, your local development environment.

Over time, your production staging and dev all become entirely heterogeneous.
Each environment ends up running totally different/incompatible versions of
all your stack's softwares, and no amount of Ansible/SaltStack/Puppet script
will save you from this. All those scripts become a nightmare to maintain and
your infrastructure as a whole becomes brittle with, for example, bugs
happening only in production, but never on your local dev environment.

K8s came as a savior to all my issues: I burned all my old ansible scripts and
rewrote all my infrastructure in k8s. Now my prod, staging and dev env are 99%
the same. It saves me a tremendous amount of time and headaches. I taught my
team how to use minikube and create a replica of our production with one
command line on their local computer.

K8s is far from being just a trendy buzzwordy shiny new cool toy to play with.
It solves real world problems that dev ops have. I am so glad this technology
exists and I hope to never have to go back to writing ansible/puppet/whatever
scripts.

~~~
botskonet
I hadn't considered the benefits for dev/local instances...

I've known about Kubernetes for some time, but my current job never deploys
anything that Kubernetes could improve, so I put it aside and hoped to someday
get a chance to toy with it.

I've setup vagrant images pre-loaded with our app for several non-developers
to use locally but it sounds like Kubernetes would be a far better way to
manage those as well as staging servers.

Unfortunately my current company is 100% against third-party
hosting/involvement so I couldn't even use it for staging - our stagings
servers are Windows-based, ancient, and internal...

~~~
theptip
For me (small startup) this is the killer feature of k8s. I'm not operating at
a scale where "cluster scheduling" is a thing I need to care about, though
self-healing and load-balanced services are nice.

To be able to stand up an exact copy of my application on a dev machine, or
even better in a review app per-branch (complete with DNS entry and TLS cert)
is incredibly valuable. You can run through essentially all of the deploy
pipeline before even merging, including SSL config tests etc.

~~~
bonesss
In addition to the dev benefits, there is also a built in cloud scaling story
and strategy.

It's not like every app needs that kind of robustness, but there is a certain
calming security in knowing that if any part of your Kubernetes deployed app
actually needs to go "web scale", or someone asks about 100 times the users
you have ever considered, that the answer is straightforward and reasonably
pre-configured.

------
macNchz
> The absolute safest place to run Kubernetes application is still Google’s
> GKE.

Interesting to read given I recently had a GKE cluster auto-upgrade its master
version from 1.6.x to 1.7.x at 8pm one night (however foolish it was to not be
subscribed to the release notes RSS Feed[1]), which somehow caused a cascading
nightmare of things breaking.

Logs stopped appearing in the GCP logging interface and in our own log parsing
pipeline, a bug related to a change in the format of the yaml specifications
meant that all the containers got stuck in a broken limbo state as we tried to
upgrade the nodes (yay for googling the error and finding open github
issues!), and then once we'd manually fixed all of our deployments and were
finally able to get our nodes rolled over to the new k8s version, all of our
load balancers started intermittently timing out until they were deleted and
recreated.

Surely we need to keep a closer eye on the release cycle, and we're guilty of
bandwagonning onto this cool new tech, but boy does it suck to get auto-
upgraded at night only to discover several breaking changes while in emergency
mode trying to fix things.

(1) [https://cloud.google.com/kubernetes-engine/release-
notes](https://cloud.google.com/kubernetes-engine/release-notes)

------
shruubi
This article concerns me, especially considering the first thing you see is
"There is no such thing as Kubernetes LTS (and that’s fantastic)".

What is so great about running your infrastructure on a platform that has no
intention of ensuring long-term stability? Irregardless of how well backward-
compatibility is maintained, the idea that we should all move our
infrastructure to something that lacks the fundamental promise of "updating
won't break everything" seems downright irresponsible.

~~~
aberoham
Author here. The point is that the stable APIs are rock solid, the design of
its API versioning and the community's pace of delivery is a huge part of why
it's all maturing so quickly.

~~~
shruubi
That's all well and good until it isn't anymore. And to top it off, your
recommendation is to either hire someone to manage k8s full-time or be at the
mercy of your cloud vendor and hope that one day you don't turn up to work to
find that the new update you weren't aware was being rolled out has just
broken everything.

You can't ask a large-scale company to take a dice-roll on k8s for their
infrastructure when the only promise of long-term reliability is "trust us not
to make any breaking changes".

------
halayli
This is a problem I faced using ansible, webpack + js modules, and more. it's
a moving ground and you always need to keep up to date with the latest changes
often which are breaking.

I tend to design my systems so that they work even if I haven't touched a line
in a year but when using such tools it's always a pain.

I wish things were as stable as a bourne shell and unix environment in
general. Not that they achieve the same but I just miss the stability I get
from raw unix tools.

~~~
deckard1
Github is already littered with the dead carcasses of software using shiny new
obsolete tools like gulp, grunt, and browserify to name just a tiny few.

You have to have a strong nose for bitrot and ruthlessly filter garbage out of
the firehose, to work in web development today. If that package or tool hasn't
been updated in 8 months, do you really want to learn it (and force your team
to learn it), since it's practically abandonware already?

It's a self-reinforcing cycle. You don't want to put your organization at
existential risk by betting on the wrong horse. But there is no choice. Unless
you want to get stuck having to hire Perl guys in an ocean of node ninja
rockstars. Your company will be the ugly girl that didn't get asked to prom
night. The pariah of Silicon Valley.

~~~
halayli
true but you don’t have to go that far. React, webpack, and sometimes well
maintained packages that depend on unmaintained packages suffer from this. I
am not picking on web dev, it’s a general problem.

~~~
purerandomness
It's a relatively new problem caused by an influx of inexperienced programmers
who have to experience these problems first-hand.

They do learn, but they can't teach what they learned because of the ever
increasing influx of new, unexperienced programmers.

I highly recommend this talk by Uncle Bob (have a tea and a foot bath, it's
long):
[https://www.youtube.com/watch?v=ecIWPzGEbFc](https://www.youtube.com/watch?v=ecIWPzGEbFc)

~~~
flukus
I don't think it's just the influx of new developers, it's too many rockstars
that only work on green fields applications, they can't and don't learn the
long term consequences of their decisions. Not all of them are new
programmers, they're just the epitome of 5 times 2 years of experience.

------
peterwwillis
trigger warning: bitter jaded ops person working in a real company

 _" [...] users are expected to stay “reasonably up-to-date with versions of
Kubernetes they use in production.” [...] the upstream Kubernetes community
only aims to support up to three “minor” Kubernetes version at a time. [...]
if you deployed a Kubernetes 1.6 soon after it came out last March, you were
expected to upgrade to 1.7 within roughly nine months or by the time 1.9
shipped in mid-December."_

Jesus christ this is so annoying.

Businesses don't have a couple hundred billion dollars sitting around to spend
on engineers to look at release notes, compare changes, write new features,
write new test cases, fix bugs, and push to prod, _every 3 months_ , just to
_keep existing functionality_ for orchestrating their containers.

We have LTS because businesses (and individuals) don't want to have to do the
above. They just want a reliable tool. They want the ability to say that if a
bug is found in 3 years, it will be fixed, and they can just keep using the
tool.

We don't give a crap about "Kubernetes’ domination of the distributed
infrastructure world". We don't want to use Kubernetes. We just want an
orchestration tool - commodified tooling. We want to stop caring about what
we're running. We just want the fucking thing to work, and to not have to jump
through hoops for it to work.

 _" Moving Kubernetes Workloads to New Clusters instead of Upgrading"_

UGH. We only do this for bastardized unholy stupid shit like OpenStack. Not
only is this not fun, it takes forever (you try moving 50 different clients
off the service they've been using for three years), and you have to have
duplicate resources. What the fuck is the point of cloud computing and
containers and all this bullshit if I have to have double the infrastructure
and juggle how it's all used just to upgrade some fucking software?!??!?!

 _" The Kubernetes-as-a-Service offerings, particularly Google Cloud’s
Kubernetes Engine (GKE), are the well-polished bellwethers of what is
currently the most stable and production-worthy version of Kubernetes."_

Oh. We're supposed to pay Google to run it for us.

....I'm just going to use AWS.

~~~
kev009
Sounds like full employment theorem at work. I haven't kicked the tires on
kubernetes yet but I don't really see what all the fuss is about. I liked
AMPLab and Mesos but I guess that doesn't have the branding power of big G.

~~~
cookiecaper
100% the case. k8s is not a terrible thing and there are good uses for it, but
the _vast_ majority of people who are using it don't understand what it's
doing, and don't understand that their application is not equipped to handle
that type of execution model.

Just yesterday I had someone tell me "Kubernetes was like magic and it made
everything easy". This is absolutely _not_ the thing a serious/honest
Kubernetes user would say. Kubernetes is a big heavy thing, and it's a lot of
trouble to maintain it; generally, far more trouble than using sane
configuration management on VMs would be (you need sane configuration
management in k8s too, so you aren't swapping one complexity for another).
We're reaching "MongoDB is web scale" levels of hysteria around Kubernetes.

Example of the insanity: my account on HN has been censured for the "tedious"
nature in which I would assert that Kubernetes is not a good platform for
databases.

There is a serious, concerted effort by Google to put k8s at the forefront,
and they are not playing games with it.

~~~
jacques_chester
> _There is a serious, concerted effort by Google to put k8s at the forefront,
> and they are not playing games with it._

Well no, and why would they? Containerisation is one way to crowbar workloads
out of VMs and AWS. Since they're coming from behind, their best strategy is
to deny everyone else any oxygen by creating an opensource winner. And it
worked: Amazon have added EKS to ECS, Azure added AKS to ACI.

Google does a lot of good, but they don't sink millions upon millions of
dollars into things just for the hell of it.

~~~
kev009
In general they have been pretty aloof at product marketing and development.
Microsoft came in with a weaker product early on and has really been cleaning
up with Azure going head to head with Amazon and GCE.

G as a whole has always been pretty much the opposite of Apple in terms of fit
and finish, and in the past it was no mongo in terms of dev swoon. But I agree
with cookiecaper, they have somehow cracked the code for mongo level of
installing meme based software architecture on the masses. I would _love_ to
understand how that works, for selfish entrepreneurial reasons :)

~~~
jacques_chester
I agree that Google is coming from behind in enterprise sales. AWS have the
long headstart and Microsoft have an existing, massive sales org.

Cloud computing fits almost none of their DNA. It requires intensive sales
rather than automation. It's a volume business with thin margins (rather than
a network-effect business with fat margins).

But it's also a business in which they have the best technology out of the
three and the first plausible alternative revenue stream they've ever found. I
don't think they are going to stop elbowing their way into this, and neither
are Microsoft.

~~~
peterwwillis
The difference is, Google is the only company that is both trying to sell
cloud services, and giving their product away to their competitors.

The company I work for uses multiple cloud providers, but does not use GCE.
They don't even think about it. I think the reason is a lack of confidence,
along with other business and technical reasons. Google just isn't a serious
company when it comes to supporting large businesses. Like it or not, that old
adage about never getting fired for buying IBM is still true.

~~~
jacques_chester
Kubernetes isn't the product, though. GCP is. Google doesn't care what
software you're running, as long as you run it with them.

------
maxxxxx
I think this is a symptom of the "release often" philosophy. With yearly or
longer releases you could actually keep up and read the release notes. With
stuff being released several times a year it's too much work to keep up unless
you are deeply into it at the moment.

I notice the same with my Android apps. I used to read release notes of new
versions but now I have it on auto update and am sometimes surprised that an
app I have been using all the time has completely changed and I don't know how
to use it anymore.

~~~
praseodym
No, it is the symptom of a high velocity project. If Kubernetes were to have
yearly releases, the list of changes would be four times as long and upgrade
path would be a major leap instead of four minor steps.

~~~
maxxxxx
I think a major leap is much easier to handle and to plan for.

~~~
shaftoe
I completely disagree. The complexity and risk of a change goes up with the
square of the size of the change. Having moved from upgrading when forced to
(almost) continuous upgrading, the number of moving parts in any given change
is small and the frequency means we become skilled at rolling out changes
safely.

~~~
gatmne
That's fine and dandy if you can afford to put up the resources to keep your
deployment up to date. For more constrained others, an LTS release can be the
deference between using the project vs not.

------
yeukhon
Kubernetes’ governance is becoming like Openstack and (I know this is
controversial), I hate Openstack, especially because it tried so hard to be
“AWS” compatible, and APIs are so awkward to use.

Cloudfoundry is better in terms of governance and project’s direction. Many of
the main developers work full time at Pivtoal. But it is hard to run your own
CF without significant investment like access management and “painless”
upgrade (etcd is a pain in the entire CF stack in my experience). Though I
have to admit the project is moving in the right direction in the past year or
so.

~~~
caniszczyk
Kuberentes/CNCF governance is completely different than Openstack. There's a
reason every major cloud provider is involved in CNCF versus Openstack. You
can see all stats for CNCF projects here, i.e.,
[https://k8s.devstats.cncf.io/dashboard/db/contributing-
compa...](https://k8s.devstats.cncf.io/dashboard/db/contributing-
companies?orgId=1)

CFF was setup in a completely different way, giving Pivotal a lot of control
in the beginning by allowing related entities to have votes and than
relinquishing that over time. It leads to a more single vendor controlled
ecosystem IMHO.

There's pros and cons to both approaches.

Disclosure: I help run CNCF.

~~~
jacques_chester
Pivotal's strong influence over CFF decisions comes from the fact that votes
are assigned according to how many fulltime engineers you devote to Foundation
work. Pivotal has more fulltime engineers on CFF projects than anyone else.

There are, as you note, pros and cons. It made sense at the time, as I think
there were concerns about vendor politics getting in the way of developing the
thing.

I guess one of these days we'll smash the CFF and CNCF together and stagger
out with some kind of bicameral system, just for laughs.

Disclosure: I work for Pivotal. Trolling is just my hobby.

------
scarface74
Combining my brief time trying to put together a proof of concept with
Kubernetes and reading articles like this, I'm so glad I chose Hashicorp's
Nomad. It's simpler to configure, more versatile (shell scripts, executables,
and Docker containers) and a decent third party UI - HashiUI. With Consul,
configuration is dead simple.

------
atulatul
Dr Dobb's articles below reflect a somewhat similar feeling

Just Let Me Code [http://www.drdobbs.com/tools/just-let-me-
code/240168735](http://www.drdobbs.com/tools/just-let-me-code/240168735)

Getting Back To Coding [http://www.drdobbs.com/architecture-and-
design/getting-back-...](http://www.drdobbs.com/architecture-and-
design/getting-back-to-coding/240168771)

