
Cloud Traffic - MindGods
https://www.tbray.org/ongoing/When/202x/2020/08/09/Service-Fabric-News
======
deanCommie
I do wonder what the equivalent review of AWS Lambda would look like from an
equivalent writer but one versed in GCP and K8S, encountering Lambda and IAM
for the first time.

I think one can easily make concerning arguments about all the things Lambda
abstracts away and how concerning that is.

HAVING SAID THAT, the substantial fundamental difference between something
like Lambda and something like K8S is the value proposition. Lambda works
backwards from your business logic. How to go from 0 to "working production
code" with the minimal amount of steps, yet will still scale essentially
infinitely (so long as you are willing to tolerate the latency costs)

K8S seems to me like it instead works backwards from your infrastructure. How
to go from one set of working infrastructure to another set of working
infrastructure, but one optimized for solving problems that your organization
doesn't have yet until you're at the point where you're looking to migrate
your whole cloud strategy to another provider because Azure gave you a 2%
higher discount than AWS did.

Meanwhile, by the time you've built your first working K8S-based architecture,
your competitor with Lambda is already in production serving customers.

I know, I know, it's not Apples and oranges. But K8S and Lambda are of the
same "generation", both launching in 2014. Lambda was AWS's big bet of "Here
is the amazing value you can get if you go all-in on AWS-native development",
K8S was GCP's equivalent "Here's how you can design your cloud infra to be
portable if you explicitly DON'T want to go all-in one cloud provider."

So while they offer almost diametrically opposite solutions (philosophically),
they are pursuing the same developers to convince them of the mind share.

Me, I'll take the one that actually cares about getting me to production with
working business logic code.

~~~
skybrian
It's odd to see this because Google App Engine is (was) not that hard and
launched in 2008. I'm no longer doing that kind of work, but it seemed pretty
good for people who want to keep it simple?

I wonder Tim Bray would be better off kicking the tires on Firebase.

~~~
justicezyx
> Google App Engine is (was) not that hard

It's incredible hard to use GAE in the same fashion as the LAMP or whatever
web framework stack at its time.

GAE is a primary example of overly confident technologists anticipating the
next trend:

They anticipate things happening too far in the future.

Of course, GAE represent the eventual next step for application development.
At the time, for anyone inside Google, that's obvious.

And what happened was that neither Google itself, nor the industry embraced
that model in the time frame that allows GAE to be relevant.

------
mrkurt
I love this quote:

> I worry a little bit about hiding the places where the networking happens,
> just like I worry about ORM hiding the SQL. Because you can’t ignore either
> networking or SQL.

The irony (or not irony, who even knows?), though, is that I always worried a
little bit about EBS for similar reasons. It's gotten really, really good but
it was a terrifying abstraction back in the day.

~~~
viraptor
> it was a terrifying abstraction back in the day.

How come? NAS, SAN, iSCSI, etc. existed and were commonly used way before AWS
or EBS. And EBS really is a dynamically routed network share in the end.

~~~
donavanm
You get a _lot_ more visibility and control with iSCSI, ATA over eth,
infiniband, etc. The downside is you need to use and understand that
visibility and control. EBS is surfaced to you as a simple locally attached
block dev. The network performance and challenges are not just managed for
you, they’re largely opaque. As the user you can’t observe the (equivalent to)
fiber channel topology, SCSI commands, discarded frames, etc. and that’s the
power of EBS as well, users don’t have to deal with anything besides a locally
attached virtual disk.

~~~
p_l
So much this.

There's also the part where EBS SLA and durability might surprise you, like
the great "internet disappeared" around 10 years ago.

~~~
donavanm
The last point is a good one around absolute vs subjective experience.

If you’re looking at 1M disks 99.999% availability is great. But if your
entire company runs on 10 instances your day will be utterly ruined when any
one of those root volumes goes away.

To that sad admin it feels like they’re sold 1/100,000 and got 1/10\. And
their impact is closer to “business critically failing” than “active automated
remediation workflow incremented.”

------
lstamour
> for that, synchronous APIs quite likely aren’t what you want, event-driven
> and message-based asynchronous infrastructure would come into play. Which of
> course what I spent a lot of time working on recently. I wonder how that
> fits into the K8s/service-fabric landscape?

Unfortunately, unlike Envoy and networking, the story for Kubernetes event-
driven architecture is not yet as stable. Generally it’s roll-your-own for
reliability. Eventually Knative Eventing will serve that purpose, but it
hasn’t hit 1.0[1] or anything more stable than that... the CRDs recently made
it to beta, though and an initial go at error handling was added relatively
recently (mid-to-late last year)[2]

1\.
[https://github.com/knative/eventing/releases](https://github.com/knative/eventing/releases)

2\.
[https://github.com/knative/eventing/tree/master/docs/deliver...](https://github.com/knative/eventing/tree/master/docs/delivery)

~~~
thrixton
There is Keda [1] which seems to fly under the radar but works well and seems
to be a great addition to the Kubernetes landscape

1\. [https://github.com/kedacore/keda](https://github.com/kedacore/keda)

------
jhgg
We "use" traffic director at work. We thought we'd be able to use it for
service discovery across our mesh, but it does way more than that in
problematic ways that leak internal google abstractions :<

I asked one thing of the product team (roughly), "we like the bits where it
tells envoy the nodes that are running the service, and what zones they are
in. we don't like all the other weighting things you try and do to act like
the google load balancer."

More in-depth we we've had traffic director direct traffic to instances within
a specific zone by weighing that zone higher before instances in that zone
would even start up, causing service degradation.

We considered writing our own XDS proxy which would run as a sidecar to envoy,
connecting to traffic director and stripping all the weights.

After some back and forth with the TD team, we came up with a solution to
instead fool it into not weighing things by setting the target CPU utilization
of our backends to 1%...

------
tptacek
_One thing made me gasp then laugh. Kelsey said “for the next step, you just
have to put this in your Go imports, you don’t have to use it or anything:_

    
    
        _ "google.golang.org/xds"
    

_I was all “WTF how can that do anything?” but then a few minutes later he
started wiring endpoint URIs into config files that began with xdi: and oh, of
course. Still, is there, a bit of a code smell happening or is that just me?_

It's not a code smell; it's idiomatic Go; the search he wants is [import for
side effects].

~~~
gen220
To be fair, you can write a lot of idiomatic code without ever using packages
that use `init()` [0].

(The most common usage is importing database drivers).

I've written Go for a while, and it still feels a bit exotic, compared to its
other control flow patterns.

Maybe "code smell" was the wrong word. But writing "surprising" logic in an
`init()` function would definitely not be considered idiomatic.

[0]:
[https://golang.org/doc/effective_go.html#init](https://golang.org/doc/effective_go.html#init)

~~~
tptacek
I guess I'd just say, when a Go programmer sees an "import _" in the README.md
for a library in Github, they generally know exactly what's happening; "oh,
you import this for side effects".

~~~
gen220
Yea that's fair.

It's admittedly a more exotic than the traditional alternative: importing a
module, initializing some struct, passing it to some other module's struct-
initialization.

Of course, now that I write it all out, and I imagine that I'd need to write
all that _just_ to give my gRPC service a load balancer... ok, I can accept
the fancy hand-waves, give me the underscore import! :)

------
harpratap
> There seemed to be an implicit claim that client-side load balancing is a
> win, but I couldn’t quite parse the argument.

Having 1 hop for load balancing (usually on a server that most likely isn't on
the same rack as your application) is worse than 0 hop for load balancing in
your localhost. No Single point of failure and much lower chances of domino
effect if a configuration goes awry.

> When would you choose this approach to wiring services together, as opposed
> to consciously building more or less everything as a service with an
> endpoint, in the AWS style?

I do not understand this. What's "wiring services together" vs "service with
an endpoint"? They both are one and the same thing in context of gateway vs
service mesh. Maybe you should read this -
[https://blog.christianposta.com/microservices/api-
gateways-a...](https://blog.christianposta.com/microservices/api-gateways-are-
going-through-an-identity-crisis/)

> but as with most K8s demos, assumes that you’ve everything up and running
> and configured

Because that is what the whole "Cloud" thing is about. You don't pull out an
RJ45 everytime you need to connect your server to your router, you just assume
your cloud provider did it for you. You don't compile your own Kernel & OS to
run your services, you just use a Linux Distro and get over it. Kubernetes is
supposed to be a cloud offering, you are not supposed to configure and run it.

> One thing made me gasp then laugh. Kelsey said “for the next step, you just
> have to put this in your Go imports, you don’t have to use it or anything...
> I was all “WTF how can that do anything?” but then a few minutes later he
> started wiring endpoint URIs into config files that began with xdi: and oh,
> of course. Still, is there, a bit of a code smell happening or is that just
> me?

Because you weren't paying attention to the Speaker. He clearly mentions the
drawbacks to sidecar as proxy model - additional latency (although the latency
is much lower than that of a single Gateway architecture, but even 1ms could
be disastrous for some applications). To cater to this high performance crowd
they have envoy as an application library model, which is of course more
difficult to adapt, but worth it when the added latency drops to nano seconds.

------
idlewords
The galaxy levels of architecture bloat implied here for a simple shopping
cart app make me deeply uneasy. How much of this complexity is necessary?

~~~
praptak
It buys you stuff. Whether it is necessary depends on whether you need that
stuff and how much you are willing to pay for it.

By stuff I mean things like support for migrating a multi-thousand VM
component between multiple cloud providers (and possibly your own
infrastructure) while maintaining service levels across geographical regions
and not having to make your other services aware of the migration.

Or maybe routing that takes into account the current state of the rollout of a
possibly-breaking upgrade of the backend VMs.

Want to go from "treat your VMs like cattle, not pets" into "treat your
(micro-)services like cattle, not pets"? Then _maybe_ this stuff is necessary.

------
buro9
> It’s impressive, but as with most K8s demos, assumes that you’ve everything
> up and running and configured because if you didn’t it’d take a galaxy-brain
> expert like Kelsey a couple of hours (probably?) to pull that together and
> someone like me who’s mostly a K8s noob, who knows, but days probably.

> I dunno, I’m in a minority here but damn, is that stuff ever complicated.
> The number of moving parts you have to have configured just right to get
> “Hello world” happening is really super intimidating.

> But bear in mind it’s perfectly possible that someone coming into AWS for
> the first time would find the configuration work there equally scary.

I feel like I missed AWS and K8s... I've been on-metal for the last 6 years,
and the 3 before that on virtual machines. My use limited to bucket storage
(S3, GCP).

So this sounds like something I could experiment with: Take a simple and
existing app and lift and shift a basic equivalent onto AWS, Azure and GCP in
whatever is the most idiomatic way. Compare the learning curves and publish
the code bases and config on github and see how much I missed (I just know IAM
will be an issue due to the number of times I hear people complain about it).

~~~
harpratap
It won't be so straightforward because each cloud vendor has multiple
offerings catering to how much control you want over your compute. The easiest
is Serverless (Cloud functions, lambda), then there is managed Kubernetes
offerings (which will standardize your experience over all vendors and on-prem
infra), third is opinionated PaaS (App Engine, Beanstalk, Heroku) and last one
is empty VMs (Compute Engine, EC2) which is very similar to your current
experience.

------
jrott
I loved this. Every time I try to work with Googles opensourced stuff it
always feels so complicated to me.

~~~
jldugger
>Every time I try to work with Googles opensourced stuff it always feels so
complicated to me.

It is, but so is the problem space, as evidenced by AWS's own clusterfuck of
services.

It kinda comes with the problem space they're trying to solve: how do you
efficiently run a bunch of services on a bunch of computers? Virtualization
solves one problem but you still have networking and service discovery to sort
out.

If you're just running a personal blog, and toy projects, then k8s is really
just more harm than help.

~~~
jrott
It is, but so is the problem space, as evidenced by AWS's own clusterfuck of
services.

It kinda comes with the problem space they're trying to solve: how do you
efficiently run a bunch of services on a bunch of computers? Virtualization
solves one problem but you still have networking and service discovery to sort
out.

Oh it absolutely is complicated. The way amazon does it seems to click better
with my mental model for how all of this is supposed to work though. I'm
really not sure if it's right or wrong or it just depends on how you model
these problems.

~~~
raincom
What's your mental model for AWS services?

~~~
jrott
LEGO bricks basically. They kinda click together in my mind with a sorta super
structure of IAM and VPCs

------
icco
Why does he think GCP regions aren't the same as AWS regions?

~~~
campers
It seems he means in terms of networking as in the prior comment: "I was
particularly impressed by how you could set up “global” routing, which means
load balancing against resources that run in multiple Google regions"

GCP has a global networking plane, so in a single GCP project you can have
resources and networking configured together from every region/AZ around the
world. In AWS (which I don't use so don't quote me, and please correct me)
regions are very separate from each other. You would need to create separate
accounts for resources in different regions and then configure the networking
between them.

~~~
jpgvm
The big difference is AWS doesn't provide a GSLB (global server load
balancing) service outside of Cloudfront. GCP on the other hand provides L7
GSLB.

You still need to create VPCs in each region and manage those but broadly
speaking building a multi-region architecture with GCP is a lot better. Hell
multi-zone is also easier in GCP because there are more regional primitives
available.

~~~
mwarkentin
Would [https://aws.amazon.com/global-
accelerator/](https://aws.amazon.com/global-accelerator/) be what you’re
talking about?

~~~
jpgvm
I haven't actually used this but this looks very similar to Google GSLB
actually, in which case that is great news.

------
x87678r
I love when cloud experts (Tim, Kelsey) look at other clouds and go "damn, is
that stuff ever complicated". Makes regular devs like myself feel a little
better.

------
apta
> Kelsey said “for the next step, you just have to put this in your Go
> imports, you don’t have to use it or anything:
    
    
        _ "google.golang.org/xds"
    

> I was all “WTF how can that do anything?

very suspect language design from golang

