Hacker News new | past | comments | ask | show | jobs | submit login
Add ability to choose a custom coordination server (github.com/tailscale)
530 points by hooooonk on Aug 15, 2022 | hide | past | favorite | 112 comments



This is a a beautiful thread. All the concerns that come into play with a commercial business, a very human representative of that business, and someone outside expressing a need. A need that some would see is contradictory to the pure business goals.

Doing business out in the open like Tailscale is doing is so refreshing. Having seen Brad Fitz communicate in other places, it is obvious he isn't doing the right thing for the user only because these conversations will be scrutinized in public. Tailscale has a lot of great technical people who actually are even better communicators.

This is the lesson for me in this thread: remember to optimize hiring good communicators and not just good technical people. This conversation would have quickly gone south if it was purely about the technical or business reasons for doing it. After all, the root word of communication is commune: to connect. We often forget that.


The people on that thread are very dope. I've filed tickets with Denton before and he's ridiculously thorough. My mesh is a little complicated on my side, and he was super patient as I walked through the relevant stages of determining where and why I was getting packet loss.


Denton was my manager at Google for a couple years. He won some award for being one of the best 10 (? some very low number) managers in the company. Thorough and patient are definitely two of his good qualities. :) Tailscale did well to hire him.


I remember when we used to say things like that about Google, circa 2000.


Google used to have that, probably lost it, and then they lost people like Brad Fitz.


Maybe Urs etc shouldn't have criticized Brad when he posted his well-intentioned "Do not buy Nest Protect smoke alarm".

https://twitter.com/bradfitz/status/566072337020112896?lang=...

The product was a terrible piece of crap that never should have been released.


I don't have an opinion on that product, but no smoke alarm is "hushable" due to state laws, as far as I know.


I'm not aware of any state law that prohibits a hushable smoke alarm.

Short-term (<15minute, repeatable) hush modes are considered a safety feature. They prevent angry homeowners from removing the smoke alarm entirely, a far more dangerous situation than a short term hush.

NFPA 72 recommends a desensitization method for ionization smoke alarms and requires one for smoke alarms installed near a cooking appliance, and several states have followed in codifying this into law. Hush is not required for photoelectric smoke alarms, under the theory that they are less prone to cooking and shower-related false-positives, but this feature is still allowed and they are a common user convenience.

UL 217 allows smoke alarms to be provided with a <15 minute desensitization feature which prevents an alarm from triggering unless obscuration/ft exceeds 4%.

Also... the Nest v2 got an in-app hush feature. My understanding is that V1 had some kind of false-positive issue that triggered a ">4% non-hushable" alarm, though, so this wouldn't have mattered.


So what happens when the alarm goes off? Hope it’s a fire and the device gets burned down?


And Avery (CEO), who was by far the best presenter (and thing-namer) I ever worked with at Google.


Earning large sums of money corrupts everything. Eventually people realise that being 'evil' is highly profitable.


I think it might be being a public company more than it is about earning large sums of money. The people who own public companies ultimately have control of the company, and typically care only about how much money is generated.


Yup, this what what I eventually grew to dislike the most about working for a public company. At the end of the day the stock price was the only thing that mattered. It would have been easier to stomach if the shareholders had a long-term vision of a healthy successful company whose value accumulates over time by making good long-term business decisions. But, quarterly returns were the name of the game. The executive compensation was tied to that performance and that cemented it into the basic strategy of all decisions.... (And this was for a large (20000+ employees), well established (40+ years old), corporation in a relatively stable sector (health care tech).


The problem with public companies is they effectively have no shareholders; none that can actually do anything to affect the company. Which leaves the company to be run by the managers for the managers.

Bogle (the father of the index fund) talks about how the index fund and friends has warped the benefits of ownership to leave companies effectively "unowned".


How is that any different from a typical private company?


A private company (and some public companies) have shareholders/owners that are large enough and active enough to control the company (Ford, famously).

Most just are kind of "managed", especially after a "star CEO" or similar moves on.


VCs and other private investors are usually more active.


Yeah, I guess I was thinking more bootstrapped/Koch-like/small business than private-with-investors


The alternatives to index funds are either that ordinary middle-class people can't participate in ownership of companies at all, or that they have to pay extortionate fees to fund managers. No thank you.


The choice isn't between index/active fund ownership - the question is how to make it so the company is not managed for the benefit of the managers, but actually for the benefit of the owners, however many or few there are.

Some companies in Europe have what might be part of the solution, with the union et al having board representation.


I've heard this before: "The product isn't the product. The stock is the product."

This can be true in VC-backed private companies too, but with public companies it seems to be even more amplified since the investors unlike angels/VCs often don't even know let alone care what the company really does.


Once the MBAs and accountants take over a company things like that can't happen anymore.


I gave Google the benefit of the doubt right up until they introduced their real name policy, haven’t had an active account since.


The developer of Headscale, Juan Font, mentioned that Tailscale also reached out to them when they tried to implement a new version of their control protocol. They also gave Juan some docs on the design to help with implementation on the Headscale side. It was posted to HN[1], but didn't seem to gain much traction.

[0] https://twitter.com/juanfont/status/1507760973037322240

[1] https://news.ycombinator.com/item?id=30829522


TIL about headscale: https://github.com/juanfont/headscale

Good on tailscale for fostering the community and not playing this like most startups would!


This sort of thing would make me buy-in to tailscale, honestly.

I have this pattern where i avoid software i can't self host, because while i love paying for my products - i don't want to be cloud trapped. So some software/hardware, say Ubiquiti's suite - i buy and run locally. But sometimes cloud features are nice, so i enable them.

I don't _want_ Cloud, but knowing i'm not trapped makes me far more likely to be a customer. Knowing i can opt-in to cloud, opt back out, etc is great.

In practice i often end up buying Cloud. But i only first bought and continue to pay for services when i know i'm not trapped. The moment they start to make self-hosting a pain, or inhibited - i'm out.


I feel the same way. Unfortunately the only ways to make money in software today seem to be (1) cloud SaaS, (2) cryptocurrency which is mostly scams, (3) surveillance, (4) addictive Skinner boxes that suck money from people ("loot box" games).

Of the four (1) is the least shady.


No doubt, Tailscale people are awesome.

From a security perspective, I would design this feature differently. If the capability to change the control server exists, then, rather than hiding it, I would want to see a prominent UI element displaying the control server name, url and "more details" to the user to see before they connect/login.

This is important to prevent any social engineering based security hacks.


One interesting thing about Tailscale Android App is that it uses Gio UI(Golang GUI Framework).

https://gioui.org/


I believe they also fund its development. A model for other tech shops to follow (fund the FOSS projects they make their millions with).


Big for tailscale IMO. Will just drastically increase the footprint of tailscale even when different control servers. Might lead to some form of federation protocol.


Exactly. Hobbyist users won't make huge amounts of money, but they could bring it to work and that'll probably be hosted.


Its really impressive how much Tailscale care about the UX of the whole product all the way down to the level of not over complicating their menus.

The whole Tailscale experience for an enduser (ie not Tailscale admin) is so much nicer than compared with something like OpenVPN in a place without MDM.


> down to the level of not over complicating their menus

to be fair, their mobile UX has plenty of warts and behaviors that don't match platform expectations and are confusing (like what happens when you tap on any of the listed machines that you have access to).

This one seems to at least have been partially motivated by making sure that accessing tailscale without paying is not too visible.

I'm saying this as a huge fan (and paying customer) of tailscale.


>> making sure that accessing tailscale without paying is not too visible.

Not sure what this means, Tailscale is free for the vast majority of non-corporate users, and I would imagine that anyone who's using it so intensely that they need the "personal pro" plan is probably someone techie enough to dig around and find out about headscale.

Also headscale isn't entirely free if you're paying for a VPS or other server to host it on.


You wouldn’t pay tailscale if you used headscale, you’d just be using the efforts they funded by using their app.


The solution of only showing menu opens if you repeatedly open and close the menu seems very bizarre to me though? Is there precedent for that?


Android's developer settings unlock if you tap the build number X amount of times (I forgot how many)


7 times I think


I think the point is the users are very unlikely to do it. Tailscale's UI is designed to be used by as many people as possible and even a single additional menu hurts that goal


Good UX is the whole point since they build on top of WireGuard which is already available, no?


This idea of "protecting the users from themselves" can be dangerous. Several very large corporations are currently telling their users to do very insecure things because telling them to use an extra option in some circumstances would be too confusing for their puny brains. Even though it's a security feature and doing things securely is kind of a big deal.

An extra field in an "Advanced Settings" menu should not need to be hidden behind some "Press About 5 times" secret gauntlet. Users are not so stupid that they will fill out an "Advanced" form field they don't understand, and even if they do, you can always make a connection attempt to see if the input was valid.


I love Tailscale but I am wary of allowing them access into my personal network. This way, I can use them for my stuff without my paranoia getting in the way, and I can recommend the hosted option for work, as it works perfectly.


"allowing them access into my personal network"

In what form? The private keys never leave the nodes, hence there shouldn't by any access per se (see https://tailscale.com/security/). Of course TS has insights into your networks, i.e. what servers it is installed on, what you connect to - so metadata.


They can add a node and connect to whatever other node they want.


Haven't thought about that and this should certainly be addressed - especially if I'd use Tailscale for business.


This seems solvable. They talk about that here: https://tailscale.com/kb/1099/device-authorization/

But it still seems like they could flip that feature off if they got compromised. To remedy that, feels like they could support a preshared secret that they don't control / see being shared as a first step: https://tailscale.com/kb/1099/device-authorization/#generate...


I think the Tailscale part that actually knows what auth keys look like is proprietary, but at least for Headscale they seem to be just unguessable database row identifiers: https://github.com/juanfont/headscale/blob/ade4e23e149e7846b...

Nothing in the Tailscale design has ever pointed to features that would guard your infrastructure from them.


Yeah, they control this setting so they can just disable it, or override it.

Since Headscale exists, though, this problem is solved quite neatly.


You can't generate a secret with a webapp they control without them controlling the secret too.


I assume the notion was that the nodes will all know the secret but the control server won't.


Exactly. Pre-shared secret out of band.


This is precisely my worry. I do use tailscale but I have this itch in the back of my mind... I have a lot of sensitive and personal stuff in my machines, and even though it would be hard for someone to get into my VPN, tailscale themselves could easily add a node and join my VPN without me even noticing.


For enterprises, they do offer self-hosted control servers, but for personal use, something like headscale will put you in more control: https://news.ycombinator.com/item?id=28574477


I don't really understand how self-hosting headscale is actually any more secure. The control server needs a stable IP address, so I'd need to run it on a VPS or something, which means I'm still trusting a third-party to not mess with my network.


My home machine has a static IP, so I run it there. Does it need a static IP, or just a hostname?


I guess it probably only needs a hostname. Although I'd still feel uneasy about running it at home, because I don't want any incoming connections to my home network unless it's over Tailscale, and headscale would need some kind of firewall exception.

Maybe headscale could run at home, served over a tunnel[1] to a VPS. But honestly, if I ever lost confidence in the trustworthiness of Tailscale the company, I would just connect my devices with some other overlay network like Yggdrasil[2] or Tor.

[1]: https://github.com/anderspitman/awesome-tunneling [2]: https://yggdrasil-network.github.io/


I think Nebula is the current favorite, but it's not as plug and play as Tailscale.


You can mitigate potential risks by configuring your OS firewall to only allow independently encrypted and authenticated traffic through Tailscale. SSH and TLS covers most needs anyways, so it doesn't require additional work.


Eh, sure, but if I'm going to reimplement a VPN, why use Tailscale?


Not sure what you mean by reimplementing a VPN. I was talking about restricting the type of traffic that flows through Tailscale. Tailscale is still the only software responsible for handling external traffic in this scenario.


I mean if I'm going to encrypt and authenticate my own traffic, it makes the VPN redundant. Is that what you meant? I might be confused.


I use Tailscale as a reasonably secure and hassle-free entrypoint into my own network. I could alternatively just expose my SSH / HTTPS servers to the internet, but that would require much more effort to maintain. Not to mention that it would expose my network to even more attackers, not just theoretical ones.

I also believe that traffic inside homes should be secured regardless since routers can be hacked. So in my case, I didn't consider it a duplicated effort. I had my traffic already encrypted and authenticated when I started using Tailscale.


Just bought the Personal Pro plan (even though I don't really need it), and will be recommending Tailscale to friends and employers whenever possible. This really addresses the potential lock-in / privacy concern, and more generally makes me believe they have their users' interests at heart, rather than just trying to extract whatever they can.


Believe it or not, this increases the likelihood of me recommending Tailscale for business use. Good for them!


If you're looking for a tailscale alternative (ish), I built a plugin for HashiCorp Vault to manage Wireguard keys: https://github.com/candiddev/vault-plugin-secrets-wireguard

It's a bit more involved than Tailscale with the benefit that it uses Vault's robust authentication options.


But completely missing the thing that makes Tailscale great, imho: ACLs.


This was focused more on secure key distribution. The plugin could be extended to include tags and firewall rules for the groups/peers similar to Tailscale's design and convert them to PostUps that modify nft or iptables.


Yup! And that's where it gets significantly more complicated quickly.


Does the vault read command produce a full mesh network? I’m assuming that’s what it does, but it isn’t spelled out in the readme.


Yea, reading the `/wg-quick` endpoint will produce a rendered config for a wg-quick interface that contains all of the peers in the group. Combined with the Vault agent example, it will update the node automatically as peers are added/deleted.


There is also Netbird.io A fully open source Tailscale alternative.

But honestly, headacale is better and more advanced than it.


It's a shame that this is getting downvotes. This looks fantastic. Thank you for posting it.


Also a good lesson about including screenshots in any PR with UI changes. Once the maintainer saw the changes visually they were much more willing to iterate and get the PR merged.


That’s pretty amazing. Thank you Tailscale!

I’ve been running my own Homebrewed WireGuard configuration mechanism for N=~10 machines using Ansible because I did not want to use a proprietary product. This gives me the confidence to eventually switch over to headscale.


How does a start up like Tailscale gets to advertise on front page of HN so frequently?

I mean every feature and blog.

As a start up person, I am really curious about their marketing.


People submit articles about them, and other people upvote them. It's really not that complicated.


There's a few hundred topnotch startups with high-caliber ex-FANG talent out there. Why is it that only one of them is getting to the top of HN for literally every single thing they put out, even when it is of relatively niche interest?


I know the team closely and while they deserve praise, the level of fawning over Tailscale at HN regularly, periodically is just unnatural and makes me suspicious. Even in this thread, a lot of the comments are generic fanboi-ism without much substance - - "The people on that thread are very dope" - "X was manager at Hooli and Tailscale did very well to hire him" - "CEO was the best presenter ever at Hooli" As you correctly noted, they get to the top of HN for every single feature and every single blog post. They are a good startup, but nowhere near being top of the pack to be getting such singled-out treatment.


I want to use tailscale for me and my wife. I’d like to give them my money to manage it for me because adding yet another self-managed service to my home network seems tedious.

It seems to me like I would want two “users”, but that pushes me into their pro or business billing. Which tier should I go for?


Sign up and just add another user to your account. Their limits are soft and they probably won’t bother you if all you have is an extra user. Tailscale is fair and flexible this way.


Interesting. I would never use a hosted VPN service because I want to be the only one who controls access. For this reason I don't use something like ZeroTier either (even though that can technically be self-hosted, it's not easy). But Mesh VPN is a great option.

I wonder if Headscale can also use internal credentials? As far as I remember with tailscale you had to log in with Google or Microsoft which is another total deal-breaker. But I haven't looked at it in ages as the hosted variety was a non starter anyway. Edit: Indeed they now have local logins, but still I would want to be the only one who controls access :)

I don't have a problem with paying for a good product, it's just the control that's an issue for me. For something as crucial to security as this, it needs to lie with me alone. Though I do prefer to just buy software outright instead of subscription models. Since I will do my own hosting, I don't think this is too much to ask. Perhaps they could offer a paid tier for people using headscale.


>I wonder if Headscale can also use internal credentials? As far as I remember with tailscale you had to log in with Google or Microsoft which is another total deal-breaker. But I haven't looked at it in ages as the hosted variety was a non starter anyway.

from the project page:

https://github.com/juanfont/headscale

>Node registration

> • Single-Sign-On (via Open ID Connect)

> • Pre authenticated key


Hmm, the SSO is too cumbersome with me (I don't want to set up my own OpenID service just for this). The preauth key might work though. I'll try it out!


But don't you want your own OpenID service? I highly recomment Authelia - easy to set up and works for so many services that allow a custom OIDC service.


Thanks for your tips. If it's that easy then yeah maybe. I would like to do more with fido2. And keycloak is indeed complex.

I would need more security keys but it sounds like there is a non-gui method? That'd be fine for me.


with the red flag that they don't support registering multiple security keys via gui

why im still using keycloak


I wouldn't consider this a red flag, but a missing feature.

But nothing against keycloak - keycloak is the gold standard. But compared to Authelia, Keycloak is really cumbersome to get up and running and also to maintain.


That's great!

What about iOS?


This has apparently been possible for a long time on iOS but you have to use an MDM profile to change the setting. https://news.ycombinator.com/item?id=28573354


They say this, but then looking at a recent issue they say it's not done in iOS yet, only macOS: https://github.com/tailscale/tailscale/issues/4299


Yep it doesn't work on iOS, hopefully the Tailscale team add a similar debug menu that they added on Android and make the login server URL configurable through there, or a working configuration through a .mobileconfig would also be fine!


I wonder why they don't open source the iOS client like they do Android. There is precedent for open source iOS apps still available on the App Store, so that's not a limitation. I would gladly dedicate some time to adding this as a PR (as I'm sure a million others would, too).


> I wonder why they don't open source the iOS client like they do Android.

Mostly because developing for iOS and macOS is terrible, especially when your app needs to have "entitlements". Tailscale uses a "Network Extension entitlement" which is linked to our corporate Apple account. Even onboarding new employees and getting them up to speed on xcode/macOS/iOS development is painful. It often requires a bunch of messing around with Keychain and random reboots (not just Xcode restarts!) because something in the macOS kernel gets confused. For some development we also need to disable System Integrity Protection. And make sure there aren't duplicate copies of certain files between /Applications and ~/Library/Developer/whatever.

And then once you get it all working, some cert or login or something in Xcode or Keychain expires in a few months and you have to re-learn the whole esoteric dance once again.

The whole process of developing Network Extensions is pretty terrible.

Even if we open sourced it, you couldn't just git clone it & hit play in Xcode. Even if you paid Apple $100/year, you still couldn't, because your Apple account isn't blessed enough with the right to use a Network Extension.

It's hard enough for us to support Apple platform development internally without helping the world learn Xcode/code signing/entitlements/Keychain.

I've been and remain a huge open source fanboy for about 25 years now. If I thought we or the community would benefit from it being open source, I'd argue for us open sourcing it. But it just doesn't seem worthwhile. Or maybe I'm just still angry at the platform.


Definitely appreciate the reply, and I don't mean to trivialize network extension development on iOS. You may still be angry at the platform, and on the other side of that coin, I'm a fan of yours who just wants to use this same feature on iOS that Android got. But again, I appreciate being a small team and wanting to focus on dev experience. If there's any hope to prioritize custom endpoint config on iOS, that would be great, and I'd be quick to use it!

For what it's worth, my want is to use Tailscale (or similar) for a family setup, which ends up feeling like enterprise to me as a tech guy, but without the pricing of it. Things like SSO are big to me, but I can't justify paying enterprise pricing for it. If there was such a plan, I'd be your first customer.


That is totally fair. I will say that I got quite a lot of value from being able to see how tailscale-android works when building my own gioui app[0]. I suspect that being able to see the same thing for a modern iOS app would be useful to some small set of developers, even if they couldn't produce a fully working tailscale binary on their own dev machines.

It really does feel like Apple just doesn't care that their app policies are hostile to developers because they have such a strong monopoly on mobile app distribution.

[0]: https://github.com/psanford/wormhole-william-mobile


Keep in mind that for security sensitive applications, being open source isn't necessarily solely about me wanting to build and run my own copy, it’s also about verifying the code yourself. I know that breaks down because without verified/reproducible binaries (impossible with app store distribution afaik) you could still publish malicious nonsense, but assuming that you publish what’s in the repo, being able to verify the code functions how you say it does is not nothing.

I have dealt with mac/iOS network extension BS before too so I feel you there. But, on that front, it also means I’d know a bit about what’s going on and find the code insightful.


Then, as a Tailscale customer for business, it'd still be great if you would prioritize the MDM stuff for my personal uses. Or just expose a field in the settings, even if it's behind some hidden menu.


Tailscale employee here:

In general the OSS operating systems get OSS clients, whereas the closed-source ones don’t necessarily get the full OSS treatment.


Appreciate the reply, Tailscaler.

That makes sense and seems like a legitimate line to draw. Being a little bit of a pain, I would point out that the Google Play store and Google's components for Android are not open source. If you're willing to make Tailscale available via Google Play, and open source the client, maybe iOS can as well? And while we're at it, macOS apps can be installed out of band of an app store, so maybe that can be open, too. It could even share a codebase these days.


I'm not a Tailscale employee, but Tailscale works on FireTV, so it is clearly not reliant on Google-provided services. They also provide a build for F-droid from the same code base that doesn't use any external services. https://github.com/tailscale/tailscale-android/blob/main/and...

As far as I can tell, the only Google Play Services API the app distributed on the Play Store uses is Google account authentication via the Play Services Google account picker.


(Personal opinion here) Keep in mind that we're still a relatively small (but growing) team, and I think that a big part of not open sourcing everything is a simple matter of having the capacity to properly engage with the community.

For now, we've optimized for doing that for operating systems where users most expect it.


I’m hoping for this too - for me I like self-hosting, and I also like throwing money at companies who produce good products. If there was like a $20-per-device tailscale-pro app which was identical except for allowing a custom server, I’d buy that without thinking twice...


That's good news, but there's now a plethora of WireGuard-based Tailscale clones.


Interested in these! Any chance you could drop a list of clones?


NetMaker is another: https://www.netmaker.org/

Compared to Tailscale the main weak point of these clones seems to be ACLs. Tailscale's system is very robust and surprisingly simple. The others I've seen are less well developed. I'm sure they will get there in time, but for now Tailscale is definitely the best if you want to control access.


The recent ones I've seen are these:

* NetBird (https://netbird.io/)

* Firezone (https://www.firezone.dev/)


But none of them as advanced as Head/Tailscale though.

No support for exit nodes for example, or DNS hostnames.


I agree, but they will get there. They just looked more polished, but that's all at the moment.



I am utterly disappointed by the lack of Scheme jokes in this thread.


Brilliant. Already a user of both offerings.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: