Thanks for the blog post. I am following the project closely since Brad announced working for you.
My main concern currently is the coordination server which does not fit the zero trust claim.
I know that the traffic between peers is end-to-end encrypted and you did a good job designing your DERP protocol. However, the ability of the coordination server (login.tailscale.com) to add arbitrary nodes to my private network without my consent scares me.
Maybe you can use Wireguard's PSK to add an additional pre-shared-key to all nodes that is not managed by the coordinator (and never transferred to it)? This would make the setup slightly more difficult (you need to login to tailscale.com AND you need to provide your PSK), but it would at least ensure that the system can never talk to foreign nodes added by the coordinator itself (because the foreign nodes do not know the PSK).
If you already use the PSK for something else, another passphrase/key-file that is never transmitted and is XOR'ed over the PSK will do the trick. The firewall configuration that is pushed to all clients should be probably signed by a local key too, since it is rather critical.
Another possible attack of the coordinator would be if he pushes a configuration with correct VPN IPs and correct public keys, but changes the mapping between them. Since the IP within a wireguard network is usually used as an identity, this might be a huge problem.
Well, this just listed out my concerns pretty well and I spent the last hour installing this on my devices. Specifically, how do I explain to a fortune 500 IT department how this is secure in a few simple sentences?
Large companies that want to remove us from the trusted group should run the coordination server on-prem.
We have considered other certification options, but so far they boil down to running a part of the coordination server on-prem. Still exploring the space though.
Sure, large companies can easily run a on-premise coordination server. But large companies usually also have a network department and several other SDN and VPN solutions already. They are probably using highly tuned and standardized ipsec tunnels with hw accelerated AES encryption and special devices from different vendors.
I think tailscale could become the #1 solution for personal users and small / mid-sized companies (some of them might not even have an IT dept). At least the easy installation, the single process design (no separate IKE service etc.) and the opinionated modern cryptography - that deliberately does not allow any configuration at all - would make it a really good candidate for those use cases.
This users can probably rent a cheap $3 VPS easily and use it as an on-prem coordination server. However, this increases the installation effort considerable and trusting the cheap $3 VPS to be an essential part of your cooperate network might be a deal breaker. Many of those users might not even have a dedicated team for server maintenance and might not apply security updates regularly.
So, please add an (optional) additional PSK (or key pair / certificate) to each node that is not shared with the coordination server and can be used to sign / verify the configuration on each client. Users that do not care don't have to specify anything (they just have to log-in and everything works). Users that do care would have to login and provide the PSK on each device.
(And to be clear, the problem is not that I do not trust you. I am sure you are doing a great job and I would really like to use your service. But you are an US based company and with the new data protection laws within the EU it might not be easy to convince all customers that potentially sending all their data to a third-party company within the US is necessary.)
ZeroTier original (and still main) author here. Objectively speaking another difference is the ACL scheme which seems perhaps less powerful than our own rules engine but easier to use. The ACLs are closer to intents while ZT rules require some TCP/IP knowledge (like raw iptables or pf).
We have plans to address this ourselves more with a higher level UI to build rules from intents and common patterns. Also have more auth integration planned, but the list is long.
Significant performance improvements in the core, reworked CLI, crypto improvements, professional security audit of both design and code, easier root server federation, easier DIY controllers, better multicast, and of course bug fixes.
The firm doing our security audits is an extremely well known one. Don't want to reveal the name quite yet.
It's possible that some of that won't land fully formed right away in 2.0 but will follow shortly thereafter, but the performance, auditing, crypto, and multicast will be there.
(Article author here.) Yes, this is what DERP is for. Nowadays automatically designating nodes as supernodes doesn’t work as well as it used to, since many devices are battery constrained and cloud devices tend to have metered bandwidth. We might someday have a way to add your own DERP relays for your domain (since the DERP code is open source anyway). The tricky part is guaranteeing reliable delivery in that case.
Tailscale looks really cool. BUT for a modern zero-trust solution I would like to also see a way to connect Load Balanced Applications in addition to devices.
Seems like I would need to setup a bastion device (virtual machine / container / other) with port forwarding for each Load Balanced Application since ACLs are set on the device level. The bastion would then be a single point of failure. Is there another way?
We have been thinking about load balancing as part of a general concept of services we want to roll out. Totally agree we need something. If you want to discuss it further, want to file an issue on github?
Tinc is really quite excellent, but it lacks the central authority to dynamically handle authentication and key management like tailscale.
I think the ideal case is Tinc continuing to manage the control plane for mesh networking, integrated with Wireguard as the actual VPN under the hood (given its rapid emergence as the de-facto "Linux VPN"), combined with a (currently absent) central identity management system like what tailscale provides.
I have what I think is a probably pretty common enterprise use case.
Windows Server with file share behind a sonicwall firwall.
Large (now very large) number of remote users need to access file share.
Currently users connect using a VPN that terminates at the firewall, with access rules allowing access to file share.
Is this use case supported with tailscale? Ie, run something on server, allow users to run something, they can then connect to file share?
Anything to do on firewall to make this all go faster / easier?
As others have mentioned - we cannot let third party add nodes to our network without approval. Adding a PSK or something to let us lock network down a bit more would be good I think.
Edit: Installing at home (google wifi for router) and work machine (behind sonicwall) - can't get RDP/Remote Desktop to connect but client install / login is great and would be much easier than current approach if it worked.
There needs to be a mechanism to distribute keys to nodes / authenticate nodes / associate nodes.
Tailscale offers a centralized server to allow nodes to provision themselves. It bootstraps that trust/account system of of logging in with some account, such as via a google oauth login.
I have need to connect all of my developers to multiple AWS accounts and have them work from hotels, home and company sites. AWSVPN only works for single account and needs certificates to be distributed to everyone. I like the way your solution is described but am unclear whether it will work for us?
I tried to use it for creating a mesh network of my family's computers, but switched to zerotier.
I use ansible to keep all our computers up to date and maintain an ldap server for some basic information; my parents are old and not tech savvy, so I do my best to make it easy for them.
Initially I set up tailscale, but the lack of an android client and the difficulty of setting up the raspberry service made me to switch to zerotier. Thing is, I really wanted that to work, as the ping with wireguard between the mesh instances was so low that surprised me.
Oh well, zerotier achieves what I want for now and it has the added benefit of letting you choose the private IP range that you like. I'll definitely keep an eye out for tailscale in the future though.
Oh man, I have a PoC of something like this written up in Python. I was more targeting a OpenVPN replacement than a mesh network, but it'd literally just be changing the central server's backend with something like etcd.
Good job on open sourcing your work and best of luck.
Great post! Can Tailscale be used to create a network between 10,000 IoT devices and a few servers? Is it scalable enough to support 10,000 peers on each node? Is it possible to script the registration of each IoT device to the control plane?
Great write-up, when I checked out TS a month or more ago, documentation was severely lacking, and the repo wasnt out there. Lot of docs confused things as they talked about some logging services (pivoted product??).
In any case the one concern I had was the coordination server being able to add new nodes as another commenter mentioned. Indeed when I first set it up, they added some dev node for chat to my peers by default! That worries me a bit. You basically have to trust their coordinator is super secure. Otherwise if someone can abuse it, they could add nodes to your very private network.
(Tailscale employee here) UDP relays - that's roughly our DERP relay network. It currently operates over HTTPS because that's the last-resort "most likely to work in hard networks" transport, but adding a UDP option for efficiency is in the books.
TCP hole punching - I don't understand this, can you elaborate? By running Tailscale you can just talk to the machines on your Tailscale network, no hole punching required. At the transport layer we might do NAT traversal shenanigans to get the mesh network up, but that's invisible to the "user layer". I'm guessing you mean something slightly other and I'm just not parsing correctly?
Relays are helpful not only when UDP is blocked, but also when both nodes are behind NAT that does random port overloading and port prediction doesn't work.
TCP hole punching uses symmetrical TCP open trick to establish a proper TCP session between nodes each behind its own NAT. Roughly the same idea as with UDP h/p, but requires a coordinating server to make it work. Useful for cases when UDP punching doesn't work or when UDP is blocked completely.
Oh, that kind of TCP hole punching. No, we don't do that, because we run over UDP only. So far, I don't think we've seen network conditions where UDP is blocked but the NAT is friendly enough to permit TCP traversal (same problems as UDP traversal, only with extra state to manage). I suppose we might get to it eventually, but there's a lot of simpler cases we can handle better first :)
This is a good post,answered most of my questions. Is it free though? It seems I can't find out the price without signing up.
My first concern was how it will be abused by malware and blocked by different networks. For example ngrok.io is classified as "proxy avoidance" which means if I use it and go to a client site that blocks it,it will create an awkward situation. But I think requiring 3rd party idP might solve this issue with tailscale.
How much of a drop-in replacement is this for a Remote Desktop Gateway connecting through to users' workstations via RDP? Our users love RDP, but i'm more in love with the security that something like this could provide.
A more relevant question for me is What is Tailscale? Didn't get that from either the article or really from the start page. Is it a VPN with "networking features"?
I guess you can call it that. You can start it on all your devices (e.g. laptop, PC, phone, tablet, Docker containers, etc.) and it will assign a fixed IP to each of them. Using this fixed IP all those devices are able to communicate securely with each other, no matter where those devices are located at the moment. Tailscale discovers the fastest, direct route between each device and routes the packages accordingly.
Example: Start it on all your devices and you can always access the shared files from your PC at home on your phone or tablet, no matter where you are. If you happen to be at home, no packages will be sent over your internet connection (so you can easily use it to stream 4k movies for example), but the way to access those files is always the same (fixed IP) and always secure.
An additional advantage of Wireguard (which is used behind the scenes) is that it maintains a strict mapping of IP addresses and identities. Therefore, you can usually restrict access to services by IP (instead of using TLS certificates or complicated authentication mechanisms).
Tunnels are created between any two devices that try to talk.
It scales great, not concentrating connections makes it easy. Tunnels are very lightweight and the only machines with lots of tunnels are servers that are already provisioned for talking to lots of machines.
Interesting :) Follow up, trying to cement my understanding here.
If two devices may talk to each other (as defined in the ACL), only then they get to know each other's public keys, right?
If so, how would this ACL be negated later- do the devices forget eachother after this disallow?
from where can I learn the fundamentals on this topic. I tried searching for data plane but most of them are k8s specific and few of them were pointing to layer3 of OSI model but I couldn't understand maybe my fundamentals are not too strong.Any reference such as books or articles will be much appreciated
This could be too high level, but I stumbled onto the founder's (crawshaw) website and found one of his blog posts very helpful for understanding basic fundamentals of the product (i.e. Zero Trust networks and microsegmentation)
Tailscale co-founder here: that's the guts of all the clients. (It almost works on macOS too as is, just a little more work to do.) We are still open sourcing our code, there's more to come but we haven't found the time to clean it up and get it ready to release. Soon hopefully!
The client contains our most sophisticated code. It's also the most important from a trust perspective, it is responsible for securing your data, our servers cannot see anything of yours because of the client. So it's vital the important parts of it are open source.
That said, I'd like to open source a server. We need a reference implementation of our control protocol so it can be properly analyzed by security experts. Not sure what form that should take yet, but I'd like it to be usable.
It is, admittedly, impossible for me to be unbiased in this discussion (coauthor of Nebula, hi), but I strongly disagree that your client code is your most important component, from a trust perspective.
Your coordination server tells every node about every other node and distributes the keys for the entire network. Everything on a tailscale network implicitly trusts your coordination service.
If an individual client is compromised, code or otherwise, the effect is more limited than your coordination service being compromised, in which case the entire system's trust is broken.
HN promoted Tailscale worth of million dollars of advertising so far for free. This is really disgusting and the biggest proof that relations >>> product.
When every Wireguard post gets immediately hijacked to become a promotion for this company, when at least 2 threads get to the HN frontpage each week then something is totally very fishy. I understand they are popular and have strong connections with many powerful people here. But at least this systematic promotion should be mentioned as promotion and not some innocent thread because it's totally organized and intentional.
You said you would stop doing this, but instead you've done it even more. As I explained before, it's against the site guidelines. Therefore we've banned this account.
I haven't seen any evidence of manipulation to get Tailscale on HN, nor has it been on HN particularly much: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que.... As far as I can tell, HN readers are genuinely interested in it, partly because they know the founders, and partly because the people involved are experienced bloggers and good writers. This article, for example, seems unusually well put together. When the underlying topic is already of interest, that makes a big difference.
If I was your friend like them, I'd be millionaire but instead I am banned for exposing you. I always knew that life isn't fair but I am counting on coronavirus to make it a little fairer
My main concern currently is the coordination server which does not fit the zero trust claim.
I know that the traffic between peers is end-to-end encrypted and you did a good job designing your DERP protocol. However, the ability of the coordination server (login.tailscale.com) to add arbitrary nodes to my private network without my consent scares me.
Maybe you can use Wireguard's PSK to add an additional pre-shared-key to all nodes that is not managed by the coordinator (and never transferred to it)? This would make the setup slightly more difficult (you need to login to tailscale.com AND you need to provide your PSK), but it would at least ensure that the system can never talk to foreign nodes added by the coordinator itself (because the foreign nodes do not know the PSK).
If you already use the PSK for something else, another passphrase/key-file that is never transmitted and is XOR'ed over the PSK will do the trick. The firewall configuration that is pushed to all clients should be probably signed by a local key too, since it is rather critical.
Another possible attack of the coordinator would be if he pushes a configuration with correct VPN IPs and correct public keys, but changes the mapping between them. Since the IP within a wireguard network is usually used as an identity, this might be a huge problem.