
How Tailscale Works - panarky
https://tailscale.com/blog/how-tailscale-works/
======
tux21b
Thanks for the blog post. I am following the project closely since Brad
announced working for you.

My main concern currently is the coordination server which does not fit the
zero trust claim.

I know that the traffic between peers is end-to-end encrypted and you did a
good job designing your DERP protocol. However, the ability of the
coordination server (login.tailscale.com) to add arbitrary nodes to my private
network without my consent scares me.

Maybe you can use Wireguard's PSK to add an additional pre-shared-key to all
nodes that is not managed by the coordinator (and never transferred to it)?
This would make the setup slightly more difficult (you need to login to
tailscale.com AND you need to provide your PSK), but it would at least ensure
that the system can never talk to foreign nodes added by the coordinator
itself (because the foreign nodes do not know the PSK).

If you already use the PSK for something else, another passphrase/key-file
that is never transmitted and is XOR'ed over the PSK will do the trick. The
firewall configuration that is pushed to all clients should be probably signed
by a local key too, since it is rather critical.

Another possible attack of the coordinator would be if he pushes a
configuration with correct VPN IPs and correct public keys, but changes the
mapping between them. Since the IP within a wireguard network is usually used
as an identity, this might be a huge problem.

~~~
mrlambchop
Well, this just listed out my concerns pretty well and I spent the last hour
installing this on my devices. Specifically, how do I explain to a fortune 500
IT department how this is secure in a few simple sentences?

~~~
crawshaw
Large companies that want to remove us from the trusted group should run the
coordination server on-prem.

We have considered other certification options, but so far they boil down to
running a part of the coordination server on-prem. Still exploring the space
though.

(I work at Tailscale)

~~~
tux21b
Sure, large companies can easily run a on-premise coordination server. But
large companies usually also have a network department and several other SDN
and VPN solutions already. They are probably using highly tuned and
standardized ipsec tunnels with hw accelerated AES encryption and special
devices from different vendors.

I think tailscale could become the #1 solution for personal users and small /
mid-sized companies (some of them might not even have an IT dept). At least
the easy installation, the single process design (no separate IKE service
etc.) and the opinionated modern cryptography - that deliberately does not
allow any configuration at all - would make it a really good candidate for
those use cases.

This users can probably rent a cheap $3 VPS easily and use it as an on-prem
coordination server. However, this increases the installation effort
considerable and trusting the cheap $3 VPS to be an essential part of your
cooperate network might be a deal breaker. Many of those users might not even
have a dedicated team for server maintenance and might not apply security
updates regularly.

So, please add an (optional) additional PSK (or key pair / certificate) to
each node that is not shared with the coordination server and can be used to
sign / verify the configuration on each client. Users that do not care don't
have to specify anything (they just have to log-in and everything works).
Users that do care would have to login and provide the PSK on each device.

(And to be clear, the problem is not that I do not trust you. I am sure you
are doing a great job and I would really like to use your service. But you are
an US based company and with the new data protection laws within the EU it
might not be easy to convince all customers that potentially sending all their
data to a third-party company within the US is necessary.)

------
manigandham
Nice, UX looks good. Other alternatives are Zerotier [1], Pritunl [2], and
Slack's Nebula [3].

1\. [https://www.zerotier.com/](https://www.zerotier.com/)

2\. [https://pritunl.com/](https://pritunl.com/)

3\. [https://github.com/slackhq/nebula](https://github.com/slackhq/nebula)

~~~
zimbatm
I was coming here to say that it looks a lot like ZeroTier. The main
differentiator as far as I know is the support for SSO and MFA.

With ZeroTier, the client connects to the network using a private shared key
and then needs to be approved on the control plane independently.

~~~
api
ZeroTier original (and still main) author here. Objectively speaking another
difference is the ACL scheme which seems perhaps less powerful than our own
rules engine but easier to use. The ACLs are closer to intents while ZT rules
require some TCP/IP knowledge (like raw iptables or pf).

We have plans to address this ourselves more with a higher level UI to build
rules from intents and common patterns. Also have more auth integration
planned, but the list is long.

~~~
DenseComet
Do you have any news to ZeroTier 2.0? I'm really looking forward to it and
being able to deploy my own controllers easily!

~~~
api
Significant performance improvements in the core, reworked CLI, crypto
improvements, professional security audit of both design and code, easier root
server federation, easier DIY controllers, better multicast, and of course bug
fixes.

The firm doing our security audits is an extremely well known one. Don't want
to reveal the name quite yet.

It's possible that some of that won't land fully formed right away in 2.0 but
will follow shortly thereafter, but the performance, auditing, crypto, and
multicast will be there.

------
InvaderFizz
It's not clear from the docs. Can a two node Tailscale setup connect to each
other when both sides are behind a IPv4 only CGNAT(Verizon/AT&T LTE)?

There is mention of STUN, but CGNAT is a whole new level of ugly and there is
often NAT behind the CGNAT.

If not, do nodes that can get incoming connections become relay "super nodes?"

~~~
apenwarr
(Article author here.) Yes, this is what DERP is for. Nowadays automatically
designating nodes as supernodes doesn’t work as well as it used to, since many
devices are battery constrained and cloud devices tend to have metered
bandwidth. We might someday have a way to add your own DERP relays for your
domain (since the DERP code is open source anyway). The tricky part is
guaranteeing reliable delivery in that case.

------
benreesman
It sounds like these guys have done their homework (from a distributed systems
PoV, I don’t know much about the security side).

IIUC it’s an application of one of my favorite design patterns: “this part
needs CP, the rest is more flexible”.

It’s remarkable how far this gadget can take you.

~~~
api
Do you mean least privilege/ minimal centralization? I wrote this rant-ish
post on the concept a while back.

[http://adamierymenko.com/decentralization.html](http://adamierymenko.com/decentralization.html)

It describes a route whereby I got to the concept, but I doubt I'm the first
one there. Everything gets reinvented over and over again in this field.

------
caleblloyd
Tailscale looks really cool. BUT for a modern zero-trust solution I would like
to also see a way to connect Load Balanced Applications in addition to
devices.

Seems like I would need to setup a bastion device (virtual machine / container
/ other) with port forwarding for each Load Balanced Application since ACLs
are set on the device level. The bastion would then be a single point of
failure. Is there another way?

~~~
crawshaw
We have been thinking about load balancing as part of a general concept of
services we want to roll out. Totally agree we need something. If you want to
discuss it further, want to file an issue on github?

(Tailscale co-founder)

------
dochtman
I was wondering, does this work with any domains that advertise (for example)
OpenID Connect, or does it require one of the well-known providers?

~~~
crawshaw
It can, though right now we only expose a few well-known providers. For
business customers we can add providers, send us an email!

Hopefully we can make this self-serve at some point. It's on the TODO list.

(Tailscale co-founder)

------
guruz
Seems a bit like TINC-VPN

[https://www.tinc-vpn.org/](https://www.tinc-vpn.org/)

~~~
JeremyNT
Tinc is really quite excellent, but it lacks the central authority to
dynamically handle authentication and key management like tailscale.

I think the ideal case is Tinc continuing to manage the control plane for mesh
networking, integrated with Wireguard as the actual VPN under the hood (given
its rapid emergence as the de-facto "Linux VPN"), combined with a (currently
absent) central identity management system like what tailscale provides.

------
anon102010
I have what I think is a probably pretty common enterprise use case.

Windows Server with file share behind a sonicwall firwall. Large (now very
large) number of remote users need to access file share.

Currently users connect using a VPN that terminates at the firewall, with
access rules allowing access to file share.

Is this use case supported with tailscale? Ie, run something on server, allow
users to run something, they can then connect to file share?

Anything to do on firewall to make this all go faster / easier?

As others have mentioned - we cannot let third party add nodes to our network
without approval. Adding a PSK or something to let us lock network down a bit
more would be good I think.

Edit: Installing at home (google wifi for router) and work machine (behind
sonicwall) - can't get RDP/Remote Desktop to connect but client install /
login is great and would be much easier than current approach if it worked.

------
mindfulplay
Is there any page that describes how to actually get VPN work end to end with
tailscale?

I am a newbie and would like to understand.

~~~
crawshaw
Install the client on two devices and log in with a Gmail account. That's it.

They'll be assigned IPs and can reach each other.

(I'm a co-founder of Tailscale.)

~~~
skrebbel
what's local networking got to do with gmail?

~~~
TheDong
There needs to be a mechanism to distribute keys to nodes / authenticate nodes
/ associate nodes.

Tailscale offers a centralized server to allow nodes to provision themselves.
It bootstraps that trust/account system of of logging in with some account,
such as via a google oauth login.

~~~
skrebbel
Right! That makes sense, thanks.

------
ksmail99
I have need to connect all of my developers to multiple AWS accounts and have
them work from hotels, home and company sites. AWSVPN only works for single
account and needs certificates to be distributed to everyone. I like the way
your solution is described but am unclear whether it will work for us?

------
DictumMortuum
I tried to use it for creating a mesh network of my family's computers, but
switched to zerotier.

I use ansible to keep all our computers up to date and maintain an ldap server
for some basic information; my parents are old and not tech savvy, so I do my
best to make it easy for them.

Initially I set up tailscale, but the lack of an android client and the
difficulty of setting up the raspberry service made me to switch to zerotier.
Thing is, I really wanted that to work, as the ping with wireguard between the
mesh instances was so low that surprised me.

Oh well, zerotier achieves what I want for now and it has the added benefit of
letting you choose the private IP range that you like. I'll definitely keep an
eye out for tailscale in the future though.

------
zamubafoo
Oh man, I have a PoC of something like this written up in Python. I was more
targeting a OpenVPN replacement than a mesh network, but it'd literally just
be changing the central server's backend with something like etcd.

Good job on open sourcing your work and best of luck.

------
ngrilly
Great post! Can Tailscale be used to create a network between 10,000 IoT
devices and a few servers? Is it scalable enough to support 10,000 peers on
each node? Is it possible to script the registration of each IoT device to the
control plane?

------
AlphaSite
Is Tailscale SDWAN for edge VPNs?

------
spydum
Great write-up, when I checked out TS a month or more ago, documentation was
severely lacking, and the repo wasnt out there. Lot of docs confused things as
they talked about some logging services (pivoted product??).

In any case the one concern I had was the coordination server being able to
add new nodes as another commenter mentioned. Indeed when I first set it up,
they added some dev node for chat to my peers by default! That worries me a
bit. You basically have to trust their coordinator is super secure. Otherwise
if someone can abuse it, they could add nodes to your very private network.

------
eps
Add UDP relays and TCP hole punching and it'll finally be a better take on
Hamachi :)

[https://swapped.cc/hamachi](https://swapped.cc/hamachi)

~~~
tprynn
If you read the article, it seems like it has both of those features, though
I'm not sure what exactly you mean by "UDP relay".

~~~
eps
It has neither.

------
badrabbit
This is a good post,answered most of my questions. Is it free though? It seems
I can't find out the price without signing up.

My first concern was how it will be abused by malware and blocked by different
networks. For example ngrok.io is classified as "proxy avoidance" which means
if I use it and go to a client site that blocks it,it will create an awkward
situation. But I think requiring 3rd party idP might solve this issue with
tailscale.

------
ericcholis
How much of a drop-in replacement is this for a Remote Desktop Gateway
connecting through to users' workstations via RDP? Our users love RDP, but i'm
more in love with the security that something like this could provide.

------
aliswe
A more relevant question for me is What is Tailscale? Didn't get that from
either the article or really from the start page. Is it a VPN with "networking
features"?

~~~
tux21b
I guess you can call it that. You can start it on all your devices (e.g.
laptop, PC, phone, tablet, Docker containers, etc.) and it will assign a fixed
IP to each of them. Using this fixed IP all those devices are able to
communicate securely with each other, no matter where those devices are
located at the moment. Tailscale discovers the fastest, direct route between
each device and routes the packages accordingly.

Example: Start it on all your devices and you can always access the shared
files from your PC at home on your phone or tablet, no matter where you are.
If you happen to be at home, no packages will be sent over your internet
connection (so you can easily use it to stream 4k movies for example), but the
way to access those files is always the same (fixed IP) and always secure.

An additional advantage of Wireguard (which is used behind the scenes) is that
it maintains a strict mapping of IP addresses and identities. Therefore, you
can usually restrict access to services by IP (instead of using TLS
certificates or complicated authentication mechanisms).

------
forgot_old_user
Really fascinating work! I enjoyed reading this article. Do you actually have
tunnels between every device? Wondering, how does this scale?

~~~
crawshaw
Tunnels are created between any two devices that try to talk.

It scales great, not concentrating connections makes it easy. Tunnels are very
lightweight and the only machines with lots of tunnels are servers that are
already provisioned for talking to lots of machines.

(Tailscale co-founder)

~~~
forgot_old_user
Interesting :) Follow up, trying to cement my understanding here. If two
devices may talk to each other (as defined in the ACL), only then they get to
know each other's public keys, right? If so, how would this ACL be negated
later- do the devices forget eachother after this disallow?

~~~
crawshaw
Yes, you only get the public keys of machines you can talk to. Later the
server asks the client to forget the key if they can no longer talk.

Additionally, we rotate keys. For security sensitive users we can rotate these
keys daily.

~~~
forgot_old_user
Thank you! Fascinating work. Easily readable, yet highly informative article.
Hope you are the ones that kill off VPNs!

------
neophyt3
from where can I learn the fundamentals on this topic. I tried searching for
data plane but most of them are k8s specific and few of them were pointing to
layer3 of OSI model but I couldn't understand maybe my fundamentals are not
too strong.Any reference such as books or articles will be much appreciated

~~~
raoulj
This could be too high level, but I stumbled onto the founder's (crawshaw)
website and found one of his blog posts very helpful for understanding basic
fundamentals of the product (i.e. Zero Trust networks and microsegmentation)

[https://crawshaw.io/blog/zero-trust](https://crawshaw.io/blog/zero-trust)

------
RabbitmqGuy
Can i use tailscale to access the internet as if i was in another country?

------
nubela
Is there an open source version of this?

~~~
krasin
tailscale is open source:
[https://github.com/tailscale/tailscale](https://github.com/tailscale/tailscale)

update: apparently, only Linux client is open-source. My bad.

~~~
slenk
That is only the linux client?

~~~
crawshaw
Tailscale co-founder here: that's the guts of all the clients. (It almost
works on macOS too as is, just a little more work to do.) We are still open
sourcing our code, there's more to come but we haven't found the time to clean
it up and get it ready to release. Soon hopefully!

~~~
j88439h84
Is the client the important part of the software, or is there a server side?

~~~
crawshaw
The client contains our most sophisticated code. It's also the most important
from a trust perspective, it is responsible for securing your data, our
servers cannot see anything of yours because of the client. So it's vital the
important parts of it are open source.

That said, I'd like to open source a server. We need a reference
implementation of our control protocol so it can be properly analyzed by
security experts. Not sure what form that should take yet, but I'd like it to
be usable.

~~~
rhuber
It is, admittedly, impossible for me to be unbiased in this discussion
(coauthor of Nebula, hi), but I strongly disagree that your client code is
your most important component, from a trust perspective.

Your coordination server tells every node about every other node and
distributes the keys for the entire network. Everything on a tailscale network
implicitly trusts your coordination service.

If an individual client is compromised, code or otherwise, the effect is more
limited than your coordination service being compromised, in which case the
entire system's trust is broken.

------
thor24
Any else getting a 500 on the blog?

~~~
bradfitz
Our FD limit was misconfigured on our new blog. Whoops.

------
fulafel
Does this work well with IPv6?

~~~
ngrilly
WireGuard fully supports IPv6. So I guess Tailscale should be able to support
it too.

------
2easy
500 internal server error

------
rbruggem
Nice job

------
nif2ee
HN promoted Tailscale worth of million dollars of advertising so far for free.
This is really disgusting and the biggest proof that relations >>> product.

Rename this website to Tailscale News.

~~~
jrockway
I mean... the people that founded the company have been long time contributors
to the site. You've probably read a hundred of Avery's articles here.

What are you doing that's interesting? Post it here and you can get the free
advertising too.

~~~
nif2ee
When every Wireguard post gets immediately hijacked to become a promotion for
this company, when at least 2 threads get to the HN frontpage each week then
something is totally very fishy. I understand they are popular and have strong
connections with many powerful people here. But at least this systematic
promotion should be mentioned as promotion and not some innocent thread
because it's totally organized and intentional.

