
Operators and Sidecars Are the New Model for Software Delivery - kiyanwang
http://www.ofbizian.com/2020/07/operators-and-sidecars-are-new-model.html
======
Animats
This is really a book ad.

So, what's going on here, I think, is that you have some microservices which
call each other. Each microservice is apparently put in a container to avoid
dependency issues. Now you have a network overhead problem if the containers
are on different machines, so there's something called a "pod" to make them
run on the same machine. This is what the author calls a "sidecar". The author
may be tangled in his own jargon. Cloudflare says a "sidecar" is another
container inside a "pod", but one that is for some auxiliary function rather
than being one of the main microservices doing the work. Hence the name
"sidecar". The link in the article for "sidecar" is to the author's book, so
you have to pay to find out for sure.

Now you have to connect up the messaging, start, stop, and update things,
scale to multiple instances and machines, log, and monitor. So you need an
orchestration system of some kind. The "operator" decides what to do, and the
applications and the Kubernetes system, via the Kubernetes operator API, do
it. The "operator" is shoved into the "pod" as a "sidecar".

Since making an operator that does the right thing is a hard problem, there's
quite a collection of buzzword-intensive products to do some part of that job.

A better explanation is in the Kubernetes documentation.[1]

Did I get this right?

(About 15 years ago, I had to write something which did this sort of thing for
QNX real time programs for a robot vehicle. Had to deal with things like
inserting monitoring into a message passing system. The Robot Operating System
(which is not an OS, it's a message management layer on top of Linux) does
something similar. Now it's gone mainstream. Does it make things easier or
harder?)

[1] [https://kubernetes.io/docs/concepts/extend-
kubernetes/operat...](https://kubernetes.io/docs/concepts/extend-
kubernetes/operator/)

~~~
barrkel
Sidecars are extra containers that run alongside the main container in a pod.
Both sidecars and pods are k8s jargon terms. Pod is what k8s calls a running
image, a container in Docker lingo. Pods have metadata like resource claims
that k8s uses for scheduling.

The point of the article is a bit like the distributed system analog of aspect
oriented programming.

A typical problem might be that you want all your network communications to be
encrypted, but you don't want to configure SSL for every service, because the
libraries are all different, configuration files to point at rotating
certificates are all different, etc. So instead you run an SSL proxy in a
sidecar, and make the traffic go through that instead. You sort of bolt on
extra horizontal functionality onto your existing set of services.

Operators are slightly orthogonal IMO. I think the core pattern of k8s, the
control loop, is great for operational resilience, but it's a bit alien to
many existing bits of software, so operators are the glue which converts an
otherwise high-maintenance service into the control loop idiom.

I think the article does have a point, it points to a unifying theme for
writing apps for k8s, and it's not necessarily just a book ad.

~~~
kasey_junk
Correction: a pod in Kubernetes is a collection of running containers deployed
on the same host at the same time.

~~~
barrkel
Sure. And sidecars share a bunch of things with the main container. I am
simplifying in an inexact handwavey way. You should still think of a pod as
primarily a single container, and design deployment around that.

------
mfer
> Today’s developers are expected to develop resilient and scalable
> distributed systems.

The author start from within a silo. Many of today's developers don't have
these expectations.

Operators are non-trivial software. Some of them are over 50k lines of code
and still not near complete. It takes a fair amount of work to develop and
test them.

It is sometimes implicitly shared that you MUST use an operator. There are
times they are useful. There are many other times you don't need them. I'm
still amazed at how often I can get away with a Kubernetes Deployment or
StatefulSet and be just fine.

------
jayd16
I'm a bit skeptical of sidecars. It seems like a pattern born of organization
separation between ops and dev instead of an optimal solution.

I mean, I think I get the benefits. You have your magical smart pipes that
just work deployed along with but separate from the grubby app code.
Supposedly the app code is simplified because a lot of the complexity is
handled in the side car. They're separate containers so you can "compose" and
deploy them separately.

But in practice, why is this not strictly worse than a set of libraries?
You're spending overhead and do you gain, really?

The side car is language agnostic...but you need a language specific library
_and_ you need to interface with the side car through some indirection
(network, file, IPC) instead of in process execution. You DO consider the side
car or you're double implementing the complexity in both the app and the side
car anyway. Reaping the benefit is the same as depending on it, no? So now you
have a dynamically linked dependency, so to speak.

Why not just save a lot of overhead and use a library when its viable and
relegate the sidecar to a stopgap instead of promoting it as an optimal
pattern?

~~~
marmaduke
It's not an either this or that, but a matter of convenience. If it's easy to
build that library into your app in terms of testing, go ahead. But it might
be easier to validate the library and app separately, and then run them
together.

For instance, I have an app that needs to securely access sensitive data. I
can build that access into my main app, as a wild violation of "separation of
concerns" or build a sidecar which handles the access and exposes the data as
a read-only file system to the app container. I test both locally and
separately then rely on the container platform to ensure that the app only
accesses the data through a read-only volume mount.

Which is easier? If you are green field, monolithic or meet all devs each day,
maybe the library is easier. But what if the two parts are built by different
teams in different countries? and Ops is a third team in a third country?

~~~
_-___________-_
If the library has a well-defined API and is built and maintained by a team
following good engineering practices, and your app is built using a
language/platform that makes accidentally working around their API impossible,
the difference between that and a sidecar container being communicated with
via a readonly FS or local network seems largely academic.

~~~
icebraining
I don't think there are many languages where you can give certain permissions
(e.g. r/w access to a directory, or TCP connection to a certain host) just to
a single library in the application, in a way that can be enforced at runtime.

~~~
jayd16
What does this secure? The same party controls both the main app and the
sidecar so the assumption is an attacker can break into the main app only?

~~~
icebraining
> the assumption is an attacker can break into the main app only?

Yeah, the idea is that the main app is more vulnerable (both due to being more
exposed, and also because it's often more complex), hence it should have very
limited permissions into the internal infra.

This approach is also used in other situations, like browsers, where the
process that processes website content is very locked down, and can only
perform actions in the system (network, filesystem, etc) through a limited
channel to an outside process.

------
rswail
Allowing independent teams to deploy their service into production is a hard
problem to manage. One way is to provide them with self-service to
abstractions that are managed with guard rails.

Persistent storage, key/value stores, caches, naming, service discovery,
networking, access control, authentication, authorization, versioning etc etc.

Having that as an API means you can deploy safely. Having it as a library
means that you're missing the indication that these are networked services
that you are relying on and must accomodate.

Having it as a networked "sidecar" means that it can be upgraded
independently, so is an extension of the control plane beyond the lower level
system services of k8s and equivalent. It can express business logic in terms
of deployment policies, sign offs, audits, accounting etc.

------
efitz
Lacking Kubernetes specific experience, I have no idea what the article is
talking about. Is it just “you can write micro services in different languages
and deploy them in separate containers, and they can still talk to each
other?” Why is that observation insightful?

Is there some special inter- on intra- container communication mechanism that
the author is trying to highlight?

~~~
closeparen
A sidecar is an alternative to an application-level library for something that
needs to be present in every instance, e.g. knowledge about how to contact
other services, or authorization checks on inbound calls.

It's similar to another microservice in the sense that we're putting
functionality across a network API instead of a function call. But unlike
another microservice, a sidecar is deployed alongside each instance and
usually inside the same network namespace, so there is no discovery/routing
problem and latency is minimal.

~~~
est
I am wondering if those XML-SOAP people ever invented something like sidecar
before?

~~~
p_l
Advanced, proper SOAP usage allowed for various patterns that acted similarly,
but that generally falls under anything with middleware that jacks into
messaging system

------
fxtentacle
Lots of fancy headlines, but they don't even define what they mean by sidecar.
Plus this appears to be mainly about Kubernetes microservices, not about
monoliths or regular end user software.

~~~
Animats
Click on "sidecar" for the ad for the book.

~~~
fxtentacle
That explains a lot. I was reading it with the Firefox reader mode, so I
didn't see that this was pr text.

------
bibryam
hi all, the author of the article here :)

The book that is linked has been translated into 4 languages and does a good
job explaining what is a sidecar. No need to buy, you can download it for free
from here [https://developers.redhat.com/books/kubernetes-
patterns](https://developers.redhat.com/books/kubernetes-patterns)

The article is for Kubernetes savvy readers, and the main takeaways (which I
agree I failed to convey clearly) are these:

\- You can consume generic capabilities from a sidecar through HTTP/gRPC,
rather than include these capabilities into a microservice's runtime as a
library.

\- There are many sidecar examples today (some in a Pod, some shared) Envor,
Cloudstate, Skupper, etc. Sidecar is an implementation detail. It can be with
containers in a Pod, or it can be a shared sidecar per namespace or host
machine - next to a monolith. The best example today, that provides generic
developer primitives as a sidecar is Dapr.

\- Kubernetes Operators encapsulate operational knowledge into executable form
and shipped to software consumers as an "Ops-included" mentality.

I explained in more details the concept of consuming distributed system
capabilities from sidecars in this article
[http://www.ofbizian.com/2020/05/multi-runtime-
microservices-...](http://www.ofbizian.com/2020/05/multi-runtime-
microservices-architecture.html) and webinar
[https://www.youtube.com/watch?v=CZPEIJFJV9k](https://www.youtube.com/watch?v=CZPEIJFJV9k)

------
aliswe
Lots of negativity here, and perhaps a little bit of cynicism.

Kubernetes, and in a broader sense Docker, has a number of paradigms which in
my opinion won't make it to super-broad adoption.

Docker has multistage builds, but are very impractical to integrate with
todays mainstream CI pipelines. Most just suppose you want to clone the repo
in their pipeline container and build the target images there. Lots of those
power tools (eg. Feed integration...) do not cater for multistage builds.

Kubernetes:

Sidecars do seem useful, but then again not everything that can be separated
should be (or is easy to do). My client really went overboard with React HOCs
and basically some apps are just 1 long vertical LoC full of "with..." beyond
count.

Pod security RBAC Cron jobs Prolly a few more that I forgot about.

Not to say that they are bad. I just don't see them being used that much...

------
t-writescode
This website throws up an error with BitDefender and stopped me from going to
the link. Be warned.

------
atonse
How have people's experiences been with Consul in general? Any pitfalls? We're
rolling it out shortly for service discovery and some limited key/value
storage (and a separate cluster for Vault storage)

~~~
t-writescode
I have very limited experience with Consul and Vault. I do know that the older
versions of these services, when they were CLI, were very unpleasant,
especially the Vault.

Consul didn't immediately meet our needs since it has no form of an audit
trail, so the purposes I'd hoped to use it for weren't really fulfilled. I did
like what the service seemed to provide and how it provided it.

Just be warned that the lack of audit trail can be a problem if you want to
use it for feature flags, etc. It will still need to be backed by some sort of
store, like git.

~~~
eikenberry
The enterprise (paid) version of Consul 1.8 has audit logging.

------
zubairq
Really excellent article. I agree that alot of delivery of software will be
via sidecars and operators in the k8s world

