go-micro seems like it does a bit too much, like service discovery and balancing within the framework when that's likely better handled by an Envoy/Istio.
I think not having to run a sidecar is pretty valuable. The entire "service mesh" industry is predicated on not modifying existing code, which I totally get (ops teams can rarely get the dev team to understand their constraints), but if you are actually "devops" and the developers can handle mTLS, tracing, authn/authz, etc. inside their app in a standard way, the software stack is a lot easier to understand and maintain.
Sidecar proxies also have the fatal flaw that they can't propagate a distributed trace, so if incoming RPC call triggers outgoing calls (common in microservices, for better or for worse), then you lose the link between them. To do that, your application has to remember the trace ID and pass it along from the server to the client. For reasons like that, I'm kind of bearish on premade service meshes. But, they are OK for mTLS.
gRPC has added a lot of code in recent years to support xDS, which lets clients discover endpoints, load assignments, locality, etc. and lets your app do the right thing without an intermediate proxy. It can just connect directly to the control plane and make the same routing decisions a proxy would. That's the ideal situation to me.
My experience is that tools like this don't scale in most organizations of even moderate size. Soon folks will want to write services in deno or ruby and the go first approach won't work anymore. Now you have "misbehaving" services connected to the system that may not implement load balancing or back off properly.
I've also seen it cause a lot of pain during upgrades since upgrading the service framework is implicitly upgrading so many other components and the framework is probably not being diligent about only improving one component (ex. Auth) per release. Sometimes that leads to basically never upgrading to avoid the risk.
It's true you need to do propagation for tracing but open telemetry has pretty broad support these days. That cost seems worth the stability and control.
Netflix was using client-side discovery only. They did run into the issue of supporting multiple languages and invented their own sidecar, but it was a huge productivity boost for them as the decision and customization on service discovery and traffic routing was distributed to individual teams.
If you're talking about the sidecar process I think you're talking about, that thing sorta sucked. It was a great idea, but execution lacked and it caused all sorts of issues as a result. Also ended up becoming sort of abandoned, while people still relied on it for critical stuff.
Netflix eventually moved away from HTTP+JSON to gRPC for backend services, and also started to expose internal DNS-based service discovery. This removed the need for that sidecar from _most_ places, since you could just generate code from protobufs, but it created other problems. Specifically, it was hard to apply the same resiliency patterns across multiple languages when calling those APIs, and so that would then contribute to weird incidents.
I'm not there anymore, but last context I have is that things are moving to an Envoy-based model now. The idea is that everything will use the sidecar, and resiliency patterns will be baked in at that layer. I suspect that'll be a better architecture in the long run.
> also started to expose internal DNS-based service discovery.
Then this is similar to Consul-based service discovery. In that case, do you know how Netflix continued to support their client-defined routing rules, like selecting a specific set of nodes of a given service to route traffic to? I'd imagine that the number of IP addresses in an ARecord is limited too, so a service won't retrieve the entire list of IPs from a DNS record?
First thing I do on seeing a new go lib is check the go.mod for dependencies.
~25 direct dependencies is too many
Firstly it's a security problem: not only have to vet this code, but also all the dependencies, and all their dependencies. It's going to be be quicker to just write the bits of this lib that I need (especially if I can copy/paste the tricky bits from here).
Secondly, it shows that the maintainer wasn't being careful about their deps. That when they see a problem, they hunt for a solution that looks like it fits rather than really trying to dig in to the problem and grok it and work out the best solution. I'm all for not re-inventing the wheel, but you have to understand your load, your road surface, your motive force and your steering mechanism before deciding on a wheel design. Adding a cartwheel to a Formula 1 race car is not going to bring the happy faces. Too many dependencies is a sign of lazy thinking
I'd be interested to see whether people said the same about other languages with dominant frameworks now. I've personally learned that at very large people scale, frameworks end up emerging aka the common libraries grouped together into a package that everyone then adopts as the starter. Frameworks leads to standardisation which leads to reuse which leads to generally better software development as can be seen by things like Rails and Spring.
A resolution for your confusion is that this is a result of Go being younger than the web. Languages turn over very slowly and most languages used on the web for the server are older than the web, or at the very least, older than the web being a major concern for them. Many "web frameworks" are, at their core, a way of hooking many different things together. Go's net/http is already that framework, so it is practical in Go to not have "a framework" tying everything together. You can take a router from here and a variable mapper from there and a template library from yet another place and they all already have the foundation to work together. Some of those places may even be from full "frameworks" of their own, and things still tend to work together pretty well.
I emphasize the age of Go because I don't think this is a virtue of the language per se (considered as a bag of features), or because the developers were super geniuses who could see what nobody else could. It's just that net/http was able to be spec'd and put into the standard library based on the decades of experiences other languages had. I would expect any other upcoming languages to be able to pull off a similar trick, as long as they try.
I certainly don't sit down with every Go web app I write with literally nothing but the standard library. I do have some non-trivial code written that way, but it isn't something I seek out as a terminal goal. It just isn't necessary to go commit to "Spring" or "Ruby on Rails" or some other framework just to get access to anything at all. I'm already in the Go ecosystem just by using net/http, you're not obligated to commit to a "framework"'s ecosystem just to get access to basic functionality because you're all but already tied in to a framework's ecosystem just by using Go, and there are many options for that basic functionality.
And to be clear, I'm more agreeing with you than disagreeing with you; I'm trying to explain the divergence between your understanding and what the Go ecosystem has because I consider it a worthy question to ask, and there is legitimately an interesting question to consider here based on experience from many other languages.
I like the rationale here, had not really even considered the era in which Go was created and the implications for that in comparison to languages and frameworks of the past. You're right though, it changes things. My experiences from ruby and java were just that frameworks ended up creating standardisation and Go being the language of the Cloud was probably going to need something similar. Adoption curves take time but I think you're right, it's more likely that an extension to the standard library for distributed systems concerns works better than say a framework.
Between net/http, database/sql, and html/template, you can make an MVC app. I agree with you that doing so isn't necessarily a worthy goal, but it provides a standard for all web frameworks/libraries to start with, and makes them more likely to get along.
I could understand Rails as it is for an already dynamic language. But Spring existence which very heavily rely on reflection always perplex me even though I am forced to use at work. It throws runtime exceptions which would otherwise been compiler error in Plain Java.
Further it turns some straight forward business logic in to some cultish bulshittery of endless packages, classes, annotations and so on. But hey, there is no xml config now, so it all is super modern stuff.
Just yesterday, I had to look into perf issue of Spring based project. It has probably 500 line of core business logic. But it is littered in tens of packages and dozens of java source files.
So it is standardization for sure but it does not seem to be leading to better results. It is however giving a great cover for bad stuff because now code follows all best practices as per Spring framework.
Every time I have to look into an "Enterprise" Java program it feels like playing a game of hide and seek with the hundred lines of business logic that exist in the midst of reams and reams of accidental complexity. It drives me nuts.
Sure, but the ecosystem encourages it. If it were _just_ bad coding I’d expect to see it everywhere, and have yet to run into a Go or Rust project that suffered in Tia way.
There's a point of diminishing returns to all technology. Even the folks developing Rails or Spring know that. This is why new things emerge which "simplify" that experience once more, but inevitably we just end up back in that same place. If I'm to examine the experience of hundreds or thousands of developers forced to use a certain tool within an organisation, they quite actively contribute back to fixing a lot of those problems within the frameworks. It always seems simpler to opt-out of it but then you end up having to reimplement a lot of the same things and maintain your own bespoke thing. Inevitably it becomes a game of tradeoffs, yes frameworks are multi-layered and sometimes that can be a problem but the alternative also doesn't scale when you need 1000+ devs to do something the same way.
In addition to the points raised in the other replies, I would observe that Go lacks most of the meta-programming features that frameworks rely upon to deliver value. You can't monkey-patch, you can't annotate functions, you can't really introspect types or do typical dynamic programming stuff.
You can often use package reflect to hack up something approximating a solution to this class of problems, true. But, like package unsafe, package reflect sidesteps a bunch of language assumptions, conventions, idioms, etc. which means any code that uses reflection is going to be subtle, brittle. It's a tool of last resort, really, an escape hatch. Not something that you should consider part of your tool set.
This is all by design, I think. Go tries to ensure that a straight-line reading of the code accurately represents the execution flow. So you can't really implement the inversion of control patterns that frameworks usually rely on without fighting the language.
How does the official grpc framework integrate with micro? It looks if I use Go grpc, I’d have to codegen the server code that depends on the grpc framework.
Go Micro has gRPC plugins for its client/server packages. You basically import these and it's immediately using gRPC under the covers. Go Micro has it's own code generation that looks similar to gRPC and based on protobuf.
Can someone explain whats going on? Just rewriting the URL and hiding the comments doesn't really make this the same post anymore. Might as well have just deleted it.
Yea, I wrote both. Go Micro was the first thing I open sourced in 2015. Eventually I moved on to building Micro, for a number of years both projects were in the same github org and worked together. After about 6 years I could see a split in the user base but also my own usage. People adopting Go Micro were those who wanted to plugin their own infrastructure and tools, whereas I wanted a more complete platform experience with Micro.
go-micro seems like it does a bit too much, like service discovery and balancing within the framework when that's likely better handled by an Envoy/Istio.