
Horrors of Using Azure Kubernetes Service in Production - pdeva1
https://movingfulcrum.com/horrors-of-using-azure-kubernetes-service-in-production/
======
minxomat
My $DAYJOB is leading a team which develops applications and gateways (for the
1k+ employee B2B market) that integrate deeply with Azure, Azure AD and
anything that comes with it. We do have Microsoft employees (who work on
Azure) on our payroll, too.

I can tell you, as I'm sure anyone in my team can, that Azure is one big
alpha-stage amalgation of half-baked services. I would never ever recommend
Azure to literally any organization no matter the size. Seeing our customers
struggle with it, us struggle with it, and even MS folks struggle with even
the most basic tasks gets tiring really fast. We have so many workarounds in
our software for inconsistency, unavailability, questionable security and
general quirks in Azure that it's not even funny anymore.

There are some days where random parts of Azure completely fail, like
customers not being able to view resources, role assignments or even their
directory config.

An automatic integration test of one of our apps, which makes heavy use of
Azure Resource Management APIs, just fails dozens of times a week not because
we have a bug, but because state within Azure didn't propagate (RBAC changes,
resource properties) within a timeout of more than 15 minutes!

Two weeks back, the same test managed to reproducibly produce a state within
Azure that completely disabled the Azure Portal resource view. All "blades" in
Azure just displayed "unable to access data". Only an ultra-specific sequence
of UI interactions and API calls could restore Azure (while uncovering a lot
of other issues).

That is the _norm_ , not the exception. In 1.5 years worth of development,
there has never been a single week without an Azure issue robbing us of hours
of work just debugging their systems and writing workarounds.

/rant

On topic though, we've had good experiences with these k8s runtimes:

\- GKE

\- Rancher + DO

\- IBM Cloud k8s (yeah, I know!)

~~~
gamblor956
My company uses Azure quite heavily, but not Kubernetes.

No crashes, ever. Way more reliable than AWS ever was. (GCP is our failover.)

So it seems that your experience is, from my POV, the exception. Maybe there's
something wrong with the way you guys have Azure set up?

~~~
politician
Which provisioning model are you using -- ASM or ARM? The last time I used
Azure we used the (deprecated) ASM model which was pretty stable instead of
the newer and often broken ARM model. We ended up staying on the deprecated
model until we moved to AWS (for unrelated reasons).

~~~
electricEmu
Wow, our experiences were different. I found ARM to be a great tool compared
to ASM.

The moment I removed the last ASM bits, my entire infrastructure became
reliably versioned and deployable.

~~~
politician
That's really great to hear!

------
QiKe
(Eng lead for AKS here) While lots of people have had great success with AKS,
we're always concerned when someone has a bad time. In this particular case
the AKS engineering team spent over a day helping identify that the user had
over scheduled their nodes, by running applications without memory limit,
resulting in the kernel oom (out of memory) killer terminating the Docker
daemon and kubelet. As part of this investigation we increased the system
reservation for both Docker and kubelet to ensure that in the future if a user
over schedules their nodes the kernel will only terminate their applications
and not the critical system daemons.

~~~
wpietri
Does it seem weird to anybody else that a vendor would semi-blame the customer
in public like this? I can't imagine seeing a statement like this from a
Google or Amazon engineer.

It also doesn't seems to ignore a number of the points, especially how support
was handled. I think it's bad form to only respond to the one thing that can
be rebutted, ignoring the rest. And personally, I would have apologized for
the bad experience here.

~~~
ummonk
While it might be phrased in a way that implies the customer is partly to
blame, the actual details would indicate the main problem was with Azure
Kubernetes Service. Critical system daemons going down because the application
uses too much memory is not a reasonable failure mode (and the AKS team
rightfully fixed it).

~~~
Drdrdrq
...and apparently forgot to notify customer of it, and in general communicate
with customer better.

I think this is the main reason for AWS lead. They simply treat customers
right (well, better than G and MS anyway).

~~~
rblatz
Yet Azure is the top cloud provider, and AWS is #2.

~~~
nemothekid
They are number one because Microsoft doesn’t break out Azure and Office 365
revenue.

~~~
cthalupa
My understanding, which may be incorrect, is also that they consider all SPLA
revenue as cloud revenue as well.

(SPLA is the licensing paid by service providers to lease their customers
infrastructure running Microsoft products. So if you pay some VPS or server
provider $30/mo or whatever they charge for Server 2012, and they turn around
and send $28 of it to MS or whatever, MS reports that $28 as cloud revenue)

------
paxys
Worth nothing that both Microsoft and Amazon's Kubernetes offerings are very
new (literally _weeks_ since GA). While "officially" ready, it is pretty naive
to rely on them for production-critical workloads just yet, at least compared
to Google Kubernetes Engine which has been running for years.

If you absolutely need managed Kubernetes, stick to GCP for now.

------
ageitgey
Here's a fun fact about Azure Kubernetes:

1\. Deploy your Linux service on k8s with redundant nodes

2\. Create a k8s VolumeClaim and mount it on your nodes to give your
application some long-lived or shared disk storage (i.e. for processing user-
uploaded files).

3\. Wait until the subtle bugs start to appear in your app.

Because persistent k8s volumes on Azure are provided by Azure disk storage
service behind the scenes, lots of weird Windows-isms apply. And this goes
beyond stuff like case insensitivity for file names.

For example, if a user tries to upload a file called "COM1" or "PRN1", it will
blow up with a disk write error.

Yes, that's right, Azure is the only cloud vendor that is 100% compatible with
enforcing DOS 1.0 reserved filenames - on your Linux server in 2018!

~~~
colemickens
The only way this would make any sense is if you were using Azure Files,
rather than Azure Disks. There's virtually never a time when it makes sense to
use Azure Files over Azure Disks, and even when it does, a change in the
application would be better advised than using Azure Files.

>Yes, that's right, Azure is the only cloud vendor that is 100% compatible
with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018!

This is hyperbolic bordering on flatly false. This is more reasonable and
accurate:

"Azure is the only cloud vendor that serves their Samba product from Windows
boxes, and thus leak Win/NTFS-isms into their Samba shares [that shouldn't be
used anyway]."

How would an ext4 filesystem, mounted under Linux, attached as a block device
to a VM, be subjected to Windows-isms? What you're implying _doesn 't even
make sense_.

~~~
parasubvert
I really would have to disagree with the statement one should never use Azure
Files over Azure disks.

1\. Most Azure VM types have very stringent limits on attached disks; a K8s
worker can easily blow past this limit.

2\. You have tremendous complexity to deal with: pick Azure managed disks vs
unmanaged disks on storage accounts (you can’t mix them on the same cluster).
You have to understand the trade of of standard vs premium storage and how
they bill (premium rounds up and charges by capacity, not consumption). And
you need the right VM types for premium.

3\. Managed disks each create a resource object in your resource group. A
resource group last I checked had hard limits on the number of resources (like
4000?). Each VM is minimum 3 to 4 resources (with a NIC, image, and disk)...
at scale this gets difficult.

4\. Azure disks require significant time to create. , mount and remount. A
StatefulSet pod failure will sometimes take 3-5 minutes for it’s PV to move to
a different worker. And worse when your Azure region has allocation problems.
Azure files are near instantaneous unmoubt/remount.

5\. Azure disks are block storage and thus only ReadWriteOnce. Azure files are
RWM.

So, sure, if you’re running a cluster database with dedicated per node PVs and
limited expected redeployments... use Azure disks. If you need a PV for any
other reason... especially for application tiers that churn frequently.. use
Azure files.

~~~
colemickens
This is all true, I sort of forget that I still have a sort of Azure Stockholm
syndrome. I'd say it's good feedback but it's nothing Azure doesn't know
about.

Maybe Azure Files performance has improved to the point where it's more usable
for storage scenarios. I suppose it probably comes down to the use case and
application behavior.

It would be good if Azure had someone testing out these scenarios and
interfacing with the larger k8s community, maybe through the SIG, for these
sorts of musings and questions.

------
nojvek
I believe this is a cultural problem with Microsoft. Probably similar to other
companies but it was very evident at Microsoft. People responsible to allocate
resources (The management chain) rarely dogfood the product.

While the Engineers and PM would complain a lot about quality issues,
management wants to prioritize more features. It was a running joke at
Microsoft: No one gets promoted for improving existing things, if you want a
quick promo, build a new thing.

So when you see a bazillion half baked things in Azure. That’s because someone
got promoted for building each of those half baked things and moving on to the
next big thing.

Going from 0-90% is the same amount of work as 90-99% and the same amount of
work as 99.0% - 99.99%. Making things insanely great is hard and requires a
lot of dedicated focus and a commitment to set a higher bar for yourself.

------
hb3b
I joined a healthcare startup in 2014 that had a small infrastructure on
Azure. Back then AWS weren't signing BAAs and Azure was the only player in
town. Being an early startup, the company didn't purchase a support plan from
Azure. One day Azure suffered a major outage (may have been storage related)
and over an hour later, I reached out to Microsoft for written confirmation
that we could forward to customers. Since we didn't have a support plan they
flat-out refused to provide any documentation whatsoever about the issue. They
wanted $10,000.

Azure - never again. Company moved to AWS within a quarter.

------
mgalgs
FWIW, Amazon's hosted Kubernetes offering (EKS) isn't stable either (DNS
failures, HPA is known to be broken, etc.).

~~~
shamsalmon
I know HPA is a legit issue but DNS failures seem to be fairly normal in
kubernetes. Scaling up kube-dns has helped us resolve that particular issue as
well as moving away from Alpine and into minimal Debian images. Alpine has its
own DNS issues that caused us much pain.

~~~
praseodym
Kube-dns (or CoreDNS in newer clusters) is pretty stable in my experience.
It's still a very good idea to run more than one replica so that you can
tolerate a single node failure, but if DNS failures are "fairly normal" that
definitely warrants some additional investigation.

~~~
sleepybrett
Most dns problems in kubernetes, in my experience, can be traced to udp
failures due to the iptables kubeproxy backend.

------
curiousDog
This is sad. From what I hear, one of the founders of k8s works on AKS.

Only a matter of time before GCP becomes the #1/2 cloud provider.

~~~
h4b4n3r0
I’ve seen at least 4 “founders of Kubernetes” by now. How many are there in
total?

~~~
rhencke
At least three to five.

[https://en.wikipedia.org/wiki/Kubernetes#History](https://en.wikipedia.org/wiki/Kubernetes#History)

------
taherchhabra
Had a similar experience with azure cosmos graph API. The API is half baked.
Doesn't support all gremlin operations. Even supported operations give non
standard output. Switched to aws Neptune immediately when they launched

~~~
wnsire
> The API is half baked

Doesn't surprise me. Cosmos was too good too be true :

\- Serverless \- Infinite Scalability \- Mongo API or Gremlin API or SQL API

It's obvious that it can't hold up to all it's promises.

~~~
the_new_guy_29
"Noone can give you what i can promise you"

------
AaronFriel
There are definitely growing pains with using Kubernetes on Azure. I've
wondered a few times if other platforms have similar issues and have seen more
than a few complaints about EKS.

Microsoft has some great people working on Azure, but I do feel like AKS was
released to GA too soon. Without a published roadmap and scant acknowledgment
of issues, I'm not sure I could recommend it to my clients or employer. It's
disappointing, because I've had few issues with other Azure services.

Full disclosure: I receive a monthly credit through a Microsoft program for
Azure.

~~~
markbnj
> I've wondered a few times if other platforms have similar issues and have
> seen more than a few complaints about EKS.

I can't speak to EKS but we've been running production workloads on GKE for
over a year with very good results. There have been a very few really
troublesome "growing pains" type issues (an early example: loadbalancer
services getting stuck with a pending external IP assignment for days) but
Google has been awesome about support, even to the extent of getting Tim
Hockin and Brendan Burns on the phone with us at various times to gather
information about stuff like the example I gave above. I give them high marks
and would recommend the service without hesitation.

~~~
curiousDog
Brendan Burns is the lead engineer on AKS AFAIK

~~~
mohamedmansour
Brendan Burns was the Lead Engineer on Kubernetes at Google around 3 years
ago, then went to Microsoft to lead this entire AKS/ACS/Container effort.

[https://www.linkedin.com/in/brendan-
burns-487aa590](https://www.linkedin.com/in/brendan-burns-487aa590)

------
rcconf
Hmm, this isn't great. Currently using Azure Kubernetes Service and we haven't
had many issues so far, but we just made the shift.

Hope I don't have to move over to Google cloud.

~~~
outworlder
Why? GKE works perfectly over there.

~~~
specialp
Many people (me included) don't trust Google to stay with any business besides
advertising. There's been too many times that they have ended services with
not much time to get off.

~~~
brazzledazzle
The Cloud products follow a Deprecation Policy documented here:
[https://cloud.google.com/terms/](https://cloud.google.com/terms/)

I can understand being upset about their consumer products but it doesn’t
really apply here.

~~~
dvasdekis
This is what always puzzled me about the Google-Alphabet strategy,
specifically the idea of having all the assets under a single share ticker
(GOOGL).

The more services you put under one banner, the more the stink of one disaster
is going to linger, and hinder adoption of the successes.

To me, a far simpler proposition would be a new brand & share issuance for
each new sub-company (eg. Waymo), with existing Alphabet shareholders getting
pro-rata shares in the new company.

~~~
brazzledazzle
I’d bet on it being about the recognition that Google has been a pioneer and
thought leader in the scalable systems/hosting space and they didn’t want to
throw out the baby with the bathwater.

------
parasubvert
DNS failures were almost certainly related to all the k8s system services on
the cluster not having CPU or memory reservations, and KubeDNS was flaking.

In general AKS is a vanilla k8s cluster and expects you know what you’re
doing. MS arguably should enforce some opinions about how things like system
services have reservations, etc, but none of this is vanilla. The trouble is
that K8s defaults are pretty poor from a security (no seccomp profiles or
apparmor/se profiles) and performance perspective (no reservations on key
system DaemonSets).

We’ve had this interesting industry pendulum swing between extreme poles of
“we hate opinionated platforms! Give me all the knobs!” And “this is too hard,
we need opinions and guard rails!”. I think the success of K8s is exposing
people to the complexity of supplying all of the config details yourself and
we will see a new breed of opinionated platforms on top of it very shortly. It
reminds me of the early Linux Slackware and SLS and Debian days where people
traded X11 configs and window manager configs like they were treasured
artifacts before Red Hat, Gnome and KDE, SuSE, and eventually Ubuntu, started
to force opinions.

------
spicyusername
This is probably why they're releasing OpenShift on Azure. To let Red Hat
engineers manage the kubernetes part.

[https://azure.microsoft.com/en-us/blog/openshift-on-azure-
th...](https://azure.microsoft.com/en-us/blog/openshift-on-azure-the-easiest-
fully-managed-openshift-in-the-cloud/)

~~~
netdur
I like openshift, I am not sure why developers are not hot about it.

~~~
wnsire
> I am not sure why developers are not hot about it.

Because it's designed entirely for enterprise customers. If you have a startup
you have very little reason to choose OpenShift compared to Heroku or AWS
honnestly.

I still love Redhat tho.

------
FlorianRappl
We have a larger migration project going on for months. So far not a single
failure occurred and our TEST environment is already fully migrated (quite
responsive and rock solid) since 2 weeks.

However, I do share that Azure indeed has released a lot of half-baked
features and services lately (last 1.5 to 2 years). I hope this trend does not
continue.

------
stefanatfrg
Couple of questions to the OP:

1\. What version of docker / container runtime is being used?

2\. What base image for your containers is being used? eg. alpine has known
DNS issues [1]

[1]
[https://www.youtube.com/watch?v=ZnW3k6m5AY8](https://www.youtube.com/watch?v=ZnW3k6m5AY8)

------
bsaul
Side question : what are the best practices for development ? Are you suppose
to run a local kubernetes deployment ( it looks like it's pretty hard to set
up) , or do you run everything outside of containers when developping and then
deal with k8 packaging and deployment as a completely separated issue ( which
looks like it could lead to discovering a lot of issues on the preproduction
environment) ?

------
gercheq
Azure is not bad but there are definitely some rough edges. We're having
trouble with their Bizspark Sponsorship biling
[https://news.ycombinator.com/item?id=17698948](https://news.ycombinator.com/item?id=17698948)

------
rdl
Key Vault (their HSM product) is even worse.

------
ubuntunero
interesting. thanks

------
partiallypro
It's a very new offering, the Linux App Services are still in beta, I have no
idea why you would roll it into production expecting no hiccups. AWS is also
new on this. Give it 6 months and let the kinks work out before migrating
workloads. Seems like common sense.

~~~
verst
App Service on Linux is an unrelated service that actually runs on top of
Service Fabric (a stateful microservice and container orchestration platform).

~~~
apurvajo
App Service on Linux is not in Beta, it is GA product for over year now with
SLA of 99.95%. It does not use Service Fabric in backend, it uses it's own
custom Orchestrator (which essentially removes the quirks of learning about
orchestration away from the user)

