
Gravitational Wormhole: WireGuard for Kubernetes - aberoham
https://gravitational.com/blog/announcing_wormhole/
======
kevin_nisbet
Author here. Feel free to reach out if you have questions or thoughts about
the project.

~~~
solatic
Hi Kevin, I'm wondering where you see Wormhole fit into the ecosystem
alongside heavier systems like Istio, which offer many more features than just
mTLS. Do you think Wormhole and Istio could be deployed together, using
Wormhole for mTLS and turning it off in Istio, and some advantage in doing so?
Do you see Wormhole's focus as being tuned to situations where the relative
heaviness of solutions like Istio are not necessary and only the lightest
solution possible is desired? Does that mean that Wormhole should be viewed
more as "include this with my application designed for small / lightweight
environments" or "include this with my Kubernetes distro, and config
applications running on that distro to best use Wormhole"?

~~~
kevin_nisbet
> I'm wondering where you see Wormhole fit into the ecosystem alongside
> heavier systems like Istio?

I see wormhole as a simpler / lighter weight solution. Like flannel, but
building an encrypted network instead of a vxlan (or xx plugin) network. The
trade offs are optimized towards simplicity and being a lighter weight option
with less capabilities.

>Do you think Wormhole and Istio could be deployed together, using Wormhole
for mTLS and turning it off in Istio, and some advantage in doing so?

I don't think I have a deep enough understanding of Istio to know for sure how
easy it is to use them together and have never tested this sort of
configuration. I wouldn't consider wormhole a replacement for mTLS, as
wormhole isn't capable of embedding identity of either end into a connection.
You can think of it like establishing ipsec tunnels between networks.

Where I think wormhole would come into play in this theoretical setup, is if
you want a subset of traffic to be encrypted, which either doesn't play well
with istio for various reasons or you benefit from additional layers to the
security model.

> Do you see Wormhole's focus as being tuned to situations where the relative
> heaviness of solutions like Istio are not necessary and only the lightest
> solution possible is desired?

Yes, exactly.

> Does that mean that Wormhole should be viewed more as "include this with my
> application designed for small / lightweight environments" or "include this
> with my Kubernetes distro, and config applications running on that distro to
> best use Wormhole"?

Yes. Although using wormhole doesn't require any application involvement.
Wormhole operates as the networking configuration for the cluster, and simply
uses an encrypted protocol for doing so. So any applications that run on say a
flannel cluster, should not realize wormhole is there, we just provide a
kubernetes network. Of course there is always an implementation detail we may
do slightly different, but that's likely a defect in the implementation we
would need to address.

~~~
sleepybrett
Whats the performance impact?

------
sloppycee
Big warning to readers:

If Gravitational asks you to complete an 'engineering challenge' they are
using you for free labour.

See:
[https://news.ycombinator.com/item?id=19784787](https://news.ycombinator.com/item?id=19784787)

~~~
alexk
James, we respectfully disagree.

I think it's helpful if you (or anyone reading this) compare two projects side
by side.

This is the project you have submitted:

[https://github.com/jameskeane/scrambler](https://github.com/jameskeane/scrambler)

It is a prototype of IPSec CNI plugin using Bash. We are happy you are
promoting your project online and as we've discussed during the interview, you
could use it as you wish as it's your work.

This is the project that we have created at Gravitational:

[https://github.com/gravitational/wormhole](https://github.com/gravitational/wormhole)

It is using Go, Wireguard and is designed for production use under Apache 2
license.

I understand that similarities in the problem statement may have led you to
believe that we could re-use the project idea, I'm assuring that you that we
have not.

There are many OSS CNI plugins in the market solving the similar problem in a
similar fashion (e.g. flannel with IPSec backend).

As a matter of fact that's the main reason we have picked this as a problem
statement - to make it easier for candidates to look around for prior work.

For example, this is a submission of the engineer we've hired based on their
similar execution of the assignment from April 2018:

[https://github.com/webvictim/k8s-ipsec-
cni](https://github.com/webvictim/k8s-ipsec-cni)

James, I wish you the best and hope that my response clarifies some confusion
around this.

Sasha, CTO and co-founder at Gravitational

~~~
sloppycee
So you're admitting that you asked candidates to work on a project you
intended to monetize?

Were you lying when you explicitly told me the opposite or did you decide
after my submission (going by commit dates) to make this a "real" project?

You didn't steal my code; you stole my _ideas_ and are now presenting them as
your own original work. Shame on you.

------
fulafel
> If you’re running Kubernetes in a network you don’t fully trust or need to
> encrypt all pod network traffic between hosts for legacy applications or
> compliance reasons, Wormhole might be for you

Is there a good analysis somewhere about how typical kubernetes setups trust
the network and what badness an advesary could do with kubernetes network
access? How sound is this default deployment setup from security POV?

For example I think DNS is used internally for service discovery, and incoming
TLS is often terminated and proxied onwards as HTTP - those could be both
MITMed, right?

------
OJFord
I thought I understood this, and that it replaced (and no doubt did a better
job of) what I'd already done - WG to get nodes on the same network, CNI on
top.

But requirement 2 confused me: > A Kubernetes cluster with IPAM enabled
(--pod-network-cidr= when using kubeadm based install)

So, do node machines need to already be on the same network or not?

~~~
kevin_nisbet
Sorry for the confusion, this requirement isn't about the node networking.
This is about assigning subnets to each node for the pod IP addresses.

Kubernetes includes an optionally enabled IPAM, that can allocate subnets from
a large subnet, into the /24 subnet assigned to each node. This becomes the
spec.podCIDR field in the node object when enabled.

Instead of implementing another IPAM for the node cidrs, wormhole uses the
built in IPAM within kubernetes. The only thing is, the ipam needs to be
turned on. On a kubeadm based install, this is done by setting the pod-
network-cidr flags during init. This sets --cluster-cidr / \--allocate-node-
cidrs on kube-controller-manager.

~~~
OJFord
But what's the value of that flag if I'm going to use Wormhole only
afterwards?

I set up a Wireguard /17 (iirc) mesh, and then kubeadm'd with --pod-network-
cidr as that node's wg endpoint /24.

I suppose what I'm not understanding is how I can set up the cluster
sufficiently to be able to kubectl apply wormhole if I haven't already setup a
wireguard (or...) network between the nodes?

~~~
kevin_nisbet
Ok, I think I might understand my miss-understanding. If I understand
correctly, you're running you're kubernetes API / connectivity over a
WireGuard network established between hosts?

Because the kubernetes APIs are protected by mTLS between nodes, the API
connectivity doesn't need to be over WireGuard, unless you don't have bi-
direction connectivity for other reasons (like NAT). Although, I don't
generally recommend splitting nodes between disparate networks, as it's not a
good failure model for kubernetes.

How this is modelled, is when setting up a kubernetes cluster, you establish
node trust using the installation method, and whatever network exists between
the hosts.

This creates a kubernetes installation, but doesn't setup network. When using
the flags to kubeadm, this creates a kubeadm config file, and turns on the
kubernetes subnetting. So say you were to configure 10.100.0.0/16 as a pod-
network-cidr. The would cause the kubernetes controller manager to assign a
/24 to each node for pods. So node 1 gets 10.100.1.0/24, node 2 gets
10.100.2.0/24, etc. The addresses from the node cidr are then assigned the
pods as they are scheduled by the cni plugin. The wormhole plugin, then takes
care of setting up all the networking that covers 10.100.0.0/24, because
kubernetes has effectively been granted management of this subnet.

So, yes, this assumes some sort of network exists between nodes, which is the
network that has been used to create mTLS trust between kubelet/api on all
members.

------
leetbulb
Very cool to see WireGuard being used in a mesh implementation. This reminds
me of Weave[0] which has worked well for me. I'll definitely be experimenting
with Wormhole.

[0] [https://www.weave.works/oss/net/](https://www.weave.works/oss/net/)

------
nhoughto
We were recently discussing creating something like for this, the current set
of CNI options is wide but shallow. As mentioned in the article CNI is
something you want to be as simple as possible, we’ve had trouble with weave
and all it’s complexity. Flannel plus encryption is perfect!

~~~
sloppycee
Check out Scrambler:
[https://github.com/jameskeane/scrambler](https://github.com/jameskeane/scrambler)

It's basically this, but with ~100 lines of bash. As simple as it gets.

------
laacz
Just afraid that time will come, when searching for black hole, neutrinos,
gravity will primarily yield these kinds of topics. Not actual ones.

------
badloginagain
Doesn't Istio provide inter-pod encryption, or am I totally off the
reservation here?

