
Istio Observability with Go, gRPC, and Protocol Buffers-Based Microservices - Terretta
https://itnext.io/istio-observability-with-go-grpc-and-protocol-buffers-based-microservices-d09e34c1255a
======
gstafford
Also published (free access) here:
[https://wp.me/p1RD28-6k1](https://wp.me/p1RD28-6k1)

~~~
williamallthing
Have you considered doing a similar writeup for Linkerd?
([https://linkerd.io](https://linkerd.io))

------
sklivvz1971
I found this interesting and an excellent general overview of today's
monitoring practices while using microservices.

I have to say, though, that I am always perplexed by these toy microservice
architectures because they are not solving a problem but creating it. For
example, we would not need sophisticated distributed tracing if we used less
moving parts; we would not need to use highly optimized (but hard to debug)
protocols like grpc and protobuf if we did not have to rely on a massive graph
of calls for each service, and so on.

Of course, there are a bunch of lovely use cases for microservices, mostly
around "how do we make 1k developers collaborate over our codebase?" (answer:
enlarge the codebase!), which always sounded like a self-fulfilling prophecy
to me, but then again, I loathe working for huge companies.

If that is your case, then these tools will make your developer life a little
more bearable. If you are playing with microservices because they are fresh
and new, please consider a more conservative architecture first.

~~~
orf
These solutions didn’t appear out of thin air because people get bored one
day. Micro services solve tangible, real world problems.

gRPC isn’t necessary hard to debug, and it brings a whole host of improvements
over whatever ad-hoc mess you used before.

~~~
ioquatix
Improvements like...?

~~~
orf
Speed, schemas, code generation, built in backwards compatibility, extension
points, public service definitions, introspection to name a few.

~~~
Thaxll
You forgot real bi-directional streaming but otherwise I agree :)

------
chessturk
As a developer who uses go microservice architecture and is in the process of
containerizing our services at work, this is very valuable. Thank you!

As an elixir/erlang enthusiast, haven't we just reinvented BEAM and observer?

~~~
pdimitar
Yep. No small amount of Erlang/Elixir programmers are kind of amused how K8s
is trying to invent OTP for nodes.

~~~
mercer
I wish we could somehow convince the undoubtedly very smart people working on
K8s to just get on the BEAM and go from there? You know, work on Dialyzer, fix
my code highlighting issues in VSCode, make Elixir faster, somehow make it
easy to create cross-platform binaries, and give me a platform-native Observer
;).

~~~
pdimitar
I suppose it's not easy to part with the illusion that containers should not
be ephemeral.

I mean, that's probably easy to do for microservices; you fire off a function,
it does something useful and returns back _quickly_. Having K8s do a home-
grown lambda hosting for you is useful there.

But the moment you introduce any state, the whole idea of ephemeral containers
is much more of a hindrance than something actually useful. A singular BEAM VM
might run (realistically) tens of thousands of processes that do a lot of
useful work. It's just not practical to kill off such a container and hand-
wave away the huge costs of spinning it back up.

IMO K8s and Erlang/Elixir are mutually exclusive at this point, sadly.

There are projects in progress that attempt to solve the distributed
supervisors ideas -- Swarm, Firenest, Horde -- but even without that, any
Erlang/Elixir app can go a _long_ way before needing distributed coordination.

------
eeZah7Ux
The complexity and amount of code involved is staggering. I'm not sure how we
can possibly justify the Kubernetes, Istio, Docker, Envoy, Prometheus,
Grafana, Jaeger, Kiali, Helm stack.

What problem are we solving that we were not solving 15 years ago, again? How
much time and effort are being saved (by organizations smaller than google)?

~~~
twblalock
It attempts to solve the problem of managing elastic container infrastructure,
in which everything is changing all the time.

The industry is in a big transition. First we ran VMs in datacenters, then we
ran them in cloud providers. Then we started running containers and realized
that their ephemeral nature made it possible to treat deployment like code,
and Kubernetes is now the standard for doing that.

So, Kubernetes gives us easy abstractions for deployment. But having lots of
little ephemeral containers that are constantly changing creates problems of
security and visibility and routing. Service mesh is an attempt to solve that
problem.

I would point out that Kubernetes has become an industry standard at this
point. A few years ago it was reasonable to think Mesos would become a
standard, or that multiple solutions would coexist. In fact, Kubernetes has
destroyed its competition and Mesos is basically dead. It would currently be
insane to adopt one of Kubernetes's competitors or try to roll your own.

If your organization standardized on Mesos a few years ago, you probably
regret that decision today, and you are probably forced to plan a transition
to Kubernetes.

The service mesh market is very immature at the moment, and the barrier to
entry is still low. Witness the recent arrival of AWS App Mesh, which is
probably going to do quite well. I would be very wary of committing to a
particular mesh until the dust settles a bit and we see a clear winner.
Otherwise you run the risk of choosing the Mesos of the mesh world.

~~~
gen220
As a small counterpoint of anecdata, I work at one of those companies that
standardized on Mesos(with Apache aurora for scheduling, Kerberos+LDAP for
access control). While our devs have small axes to grind with aurora, I have
never heard our SREs complain about Mesos. The only complaint one might
plausibly raise is that it’s not Kubernetes(!), in the sense that 5 years from
now it might be hard to find an SRE with Mesos experience vs K8s experience.

Mesos’s “batteries not included” approach means it does not now and will not
ever have feature parity with K8s, but we’ve managed to cobble enough
batteries together to make it operationally sufficient for our needs. We
certainly aren’t planning a transition away as far as I can tell. Anyways,
just my 2¢! :)

~~~
twblalock
I expect that you will eventually be forced to transition in the next few
years. Mesos will fall further and further behind Kubernetes due to lack of
development and lack of integrations with new service meshes, telemetry
systems, etc. Most new projects boast Kubernetes integration, and omit Mesos
integration. Although you may want to hold out with Mesos so you can wait and
see if Kubernetes gets supplanted by a newer model.

Even Docker, which competes directly with Kubernetes via Swarm and Compose,
has felt the need to include a Kubernetes cluster in Docker for Desktop.
That's how comprehensively Kubernetes has taken over the container
orchestration market.

In any case I was not specifically criticizing Mesos, but rather using the
Mesos vs Kubernetes example to point out that committing to a particular
service mesh this early in the evolution of the concept is probably unwise.

------
nevi-me
Is Istio/Envoy supported on a non-k8s deployment?

We're deploying an application that's in containers but not running on k8s,
and has:

* an Angular front-end, using grpc-web

* a gRPC-web proxy (improbable-eng; golang)

* an nginx proxy (for Angular, and routing grpc traffic)

* an authentication manager (grpc+rust)

* an audit logging service (grpc+rust)

* a database service (grpc+kotlin)

* a 'logic' engine thing (grpc+kotlin)

* jaeger for distributed tracing

It's been a joy to develop, but I wouldn't mind reducing the 2 proxies to 1
with maybe Envoy. I tried rewriting grpc-web proxy in Rust, but gave up after
struggling and not having enough time to complete it.

~~~
Thaxll
How is Rust support for gRPC last time I checked it was very slow / broken
compare to supported languages? Also I quiet don't understand using 3
different languages for a tech stack, why don't you stick to one language?

~~~
steveklabnik
My understating is that the Linkerd folks have a production grpc client and
server.

~~~
nevi-me
Yes, I'm using Tower-grpc from them. Stable and has the features that I need.

~~~
kod
Do you not need bidirectional streaming, or do they support it now? There were
no examples of how to do it last I checked.

------
xissy
Can’t read due to medium free access limit policy.

~~~
gstafford
Also published on my own blog:
[https://wp.me/p1RD28-6k1](https://wp.me/p1RD28-6k1)

------
SlowRobotAhead
Things I don’t get:

1\. why are they choosing gRPC over REST if the application is entirely web
based. With REST you have a standardized system of verbs and not specific
application functions (GET /user/id , 200 OK <data> vs.... MyGetUserFoo(ID) ,
some return data unique to application). With RPC both sides need very
specific knowledge of the functions each other have and their arguments all
the time, service discovery seems harder and an update to application seems to
nearly always imply and update to web.

2\. What is the model supposed to be for typical protobuf schema sharing? I
like PBs, but it seems a little harder than JSON or CBOR or MessagePak or
other schemaless serializers in that the proto file has the same sync issue
among all of your endpoints.

3\. Wouldn’t be really swell if serializers were supported by cloud providers
and Microservices a little more? I have a project right now that every time a
message goes from our services to an endpoint it needs encoding and decoding.
This is kind of a header now for every Microservices we have, it would be nice
to not have the option of making so many mistakes.

~~~
ec109685
gRPC’s responses are backwards compatible, so you aren’t mandated to update
all clients at the same time.

~~~
lugg
You're still blocked from using later features potentially even reaching a
dependency chain deadlock until you update.

Systems like this work far better when you just work to not break backwards
compatibility full stop.

Similarly, they work better when you don't tangle up dependencies in a bunch
of shared code, generated or otherwise.

~~~
joshuamorton
I'm unclear what problem your describing.

Your always blocked from using later features until an update because we'll,
you won't have application code that can take advantage of those features
until you update.

