Please, for the love of interop, do not use GRPC. Not every language supports it (JSON is way better anyway). There are a number of times I’ve wanted to build on something in k8s. But I’m not learning go just to write an api controller. I’m not going to write a GRPC implementation just to write an api controller in my language of choice either.
Full disclosure: I really hate GRPC as a wire protocol.
Worse than using a binary protocol and an IDL, I think, are the myths that gRPC proponents seem to believe: e.g. you can't delete fields and all fields are optional, ergo <voila> you now have guaranteed backwards compatibility.
Webassembly is also gaining steam for server workloads due to several advantages (less overhead, better capability based security, composability, ...).
See Spin [1], Wasmcloud [2], Lunatic [3], etc.
My system is based on a distributed Webassembly runtime.
The reason for taking inspiration from Kubernetes is making deployment of distributed workloads on that runtime easy.
A nice benefit for a Kubernetes alike system is that the equivalent to operators and controllers can be much more light-weight WASM actors that are easier to develop, deploy and scale.
It's like the JVM but more languages can run on it and it has buy in from more companies. Imagine all the promises of Java in the 90s/2000s but the open source community is actually interested in it.
What would you do differently if you could start from scratch?