Hacker News new | past | comments | ask | show | jobs | submit login
Mullvad: Diskless infrastructure using stboot in beta (mullvad.net)
423 points by NabiDev on Jan 12, 2022 | hide | past | favorite | 129 comments



For people wondering how the hell a user can audit the server is diskless or whatever, the goal appears to be using TPM to provide remote attestation for all code in the boot path. See https://www.system-transparency.org/.


Correct! Thank you for highlighting that.

Here are some additional details for those interested. We intend to make use of TPM for remote attestation of the current boot chain, reproducible builds to provide a strong link from source code to build artifacts, and a transparency log for a historical record of previously used boot chains, artifacts, WireGuard server keys, and related signatures.

As dtx1 mentioned elsewhere in this thread, diskless VPN infrastructure is currently in use by many other VPN providers. That is not a novel feature of course. What is novel is user-auditability of running VPN infrastructure. We were the first VPN provider to state our intention to make our infrastructure user-auditable AND provide a realistic roadmap with the specific technologies needed to do so. See the link above.

I believe the technologies we use in System Transparency will ultimately reshape the VPN provider industry into a highly competitive space focused on maximizing the transparency of VPN infrastructure. Or not, but at least OUR users will be able to audit us. :)

Either way we’re looking forward to the future. The opportunity for improvement is immense.


Niiice. I really love the concept of reversing the usual DRM use of remote attestation--forcing customers to prove they're running only software allowed by the megacorps. Instead of DRM, it's proving the corporation/server is trustworthy to the customer.

I think I could get behind more of this use!


Check out tpm2-totp. I stumbled across it while looking for a way to store totp secrets in my tpm, and was really impressed with the clever use of totp to verify a boot chain.

https://github.com/tpm2-software/tpm2-totp


If you're from Mullvad, let me tell you that I love your service.

I had one technical support question, and I got an immediate response from an actual person who knew this problem, gave me a simple workaround (toggle between wireless connection and not), and told me that a permanent fix was in the works. Likely that fix rolled out because I haven't seen the issue in months.

The idea of an account that doesn't need a password because there's no critical information saved is such a nice one, too.

Keep up the good work - I recommend you to everyone.


Thank you!


This all sounds very exciting. Where will you draw the line between public and private – at the moment your consumer-facing app is on github, but less "server side" stuff (in common with many other VPN providers). I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently. I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled)


> Where will you draw the line between public and private – at the moment your consumer-facing app is on github, but less "server side" stuff (in common with many other VPN providers).

All source code for all software on our VPN servers must eventually be public, and all build artifacts must be reproducible by 3rd parties.

> I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently.

Exactly, but we will also have to measure each artifact in the boot chain into the platform TPM, and allow anyone to issue a challenge to the TPM to get a signed quote of the boot chain measurements.

> I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled)

We’ll have to constrain our own ability to access the VPN servers. We cannot be allowed arbitrary root access as that would make the TPM measurements meaningless from an audit perspective. Well, you’d be able to conclude we have root access, so not totally meaningless.


Isn't network monitoring and logging the bigger issue for a VPN service? How can you provide transparency of the network?


> Isn't network monitoring and logging the bigger issue for a VPN service?

Great point. I have a hard time comparing the importance of the integrity of our VPN servers with monitoring and logging of network traffic happening upstream of them without a long discussion of the many nuances. Let’s leave it at; they are both very important issues.

> How can you provide transparency of the network?

That’s very hard. Individual AS’s upstream of the VPN server you’re connected to might change their routing at any moment, and suddenly your traffic makes its way through an unexpected AS or jurisdiction, which may change the monitoring situation completely.

Enabling our users to use multiple hops is one mitigation, another is vetting our data center providers. In collaboration with some of our Internet providers in Europe we have deployed protections on layers below IP. Come to think of it, I don’t think we’ve blogged about that. Thanks!


I hate to be this skeptical, but let's say this is 100% possible (I have my doubts, see previous attacks on things like TPMs and SGX, but I digress). You probably could get 90% of the logging capability by putting monitoring in front of and behind the server, and associating connections by traffic/time.

It just seems like this goal of using technology to prove they're trustworthy is unlikely to actually work for a VPN company due to the threat models.


You are correct that System Transparency is not a universal remedy for all threat models. Indeed the word "secure" is undefined until you have a threat model. Most threat models are implied and undocumented assumptions.

At some stage in an R&D project one should shift from exploration to threat model-driven development. Most people, myself included, tend to focus on technical solutions, and argue back and forth how "oh, but it can be broken using X".

System Transparency aims to provide remote auditability assuming (1) the server hardware specification is correct, (2) a correct cryptographic hash of the contents of the SPI flash containing the platform firmware, and (3) a keypair generated on and only accessible to the platform. This is very simplified of course.

An attacker aiming to tap incoming and outgoing network traffic from our servers, who has physical access to the VPN server's Ethernet port, or an upstream router, isn't in the scope of System Transparency to protect against. We need to use other means for that.


Traffic analysis and correlation analysis is indeed a powerful tool, and in general only communicating at a constant bandwidth between all nodes at all times is the only way to completely defeat it (which is what, I understand, some military systems do). That's inherently highly wasteful, however.

To get around this, Mullvad offer very transparent comprehensive multi-hop routing systems [1]; you can bounce your wireguard tunnels around in layered wireguard tunnels (a bit á-la tor) by just choosing a series of ports to tunnel on and to. My understanding is that each one of these adds non-deterministic latency to your connection and probably would help to make such attacks harder at the very least, because from the point of view of an "all seeing" adversary the fact that all of these servers talk to each other all the time makes it very much harder to know where any packet could have gone. Yes, you can see each individual link but the metadata is lost.

I signed up for Mullvad when the UK's Snooper's Charter came into force and the local health inspectors suddenly had the rights to see my DNS record. Since then, I've had it installed on my router and just route everything through a custom wireguard (originally openvpn) tunnel. I've had some issues with my ISP randomly bandwidth limiting traffic on the odd port to 1 MByte/s, but frankly that makes me more inclined to put everything behind an encrypted tunnel. I don't want my ISP to do traffic shaping and I do want them to just leave me alone and let me communicate in peace. I have absolutely nothing to hide, but now have to accept that I partly live in a country where everything is surveilled all the time, and warantless, unaccountable investigation of my (highly personal!) online habits may be happening. I think Mullvad's excellent product, sensible architecture and reasonable price is worth paying. I'm an academic, unlikely to be of interest to three-letter acronyms, and therefore it matches my needs very well.

[1] https://mullvad.net/en/help/multihop-wireguard/


How do we audit that the TPM chip is secure? What happened when a bug in the silicone is found later?


We assume it is, just like we assume CPU works as advertised. In other words, TPM is part of TCB.


So what is the point? I already assume the code on their server is not malicious by using it. What extra trust does an untrusted TPM chip give me?


System Transparency reduces your trust assumptions on us. As a VPN provider we are in an immense position of power over you. We aim to reduce your trust assumptions on us to a few things that we would need to explicitly lie about in order to betray you.

As an example, let's say that we offered any of our users to at any time during the year show up at our office and inspect our VPN hardware, without warning us beforehand. In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense?


> In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense?

Have you ever thought about doing something like that with some big youtube personalities? Maybe have them hire some pen testers, randomly show up to one of your datacenters, and post recordings of what is done and attacks that could be possible. Since your software is open pen testers could prepare some things to try to attack days in advanced. I'd love to see something like this with Level1Techs or something.


> big youtube personalities

You mean the same people that are being "sponsored" by VPN providers for years? Why would we want to trust youtubers of all people?


It reduces TCB. TPM is smaller than entire server.


Damn it, this is a really cool use case for TPMs! So far, every use case I've heard has made me wish they were never invented, but this made me reconsider...


I've been following Mullvad for a long time and my impression (from countless reviews and comments here on HN) has been quite positive. But here's what I don't understand: Why are the servers located in Sweden, a country that's known for online surveillance[0] like no other country in the EU? From the Wikipedia article[1]:

> The law permits the signals intelligence agency, National Defense Radio Establishment, to monitor the content of all cross-border cable-based Internet traffic to combat "external threats" such as terrorism and organized crime.

[0]: https://www.opendemocracy.net/en/can-europe-make-it/didier-b...

[1]: https://en.wikipedia.org/wiki/Internet_in_Sweden#Internet_ce...


Thank you for asking.

> Why are the servers located in Sweden,

We have 762 servers spread across 38 countries. Less than 10% of our servers are located in Sweden [0].

> a country that's known for online surveillance

My cofounder and I started Mullvad as a protest against the growing mass surveillance of Sweden as well as other countries. Our intent was direct political action through entrepreneurship. Incorporating the company in Sweden was the obvious choice, since we are both Swedes. We felt that incorporating elsewhere would have been more risky, complicated, and costly.

Mullvad has excellent lawyers who continuously monitor the legal situation in Sweden. Right now there is no Swedish law that can compel Mullvad to start logging [1]. Should the situation change we have contingency plans.

[0]: https://mullvad.net/en/servers/ [1]: https://mullvad.net/en/help/swedish-legislation/


Thank you, this was the response I was looking for!

> Our intent was direct political action through entrepreneurship.

Again, I didn't mean to question your intent – as I said my impression of your company so far has been a very good one! :)

> We have 762 servers spread across 38 countries. Less than 10% of our servers are located in Sweden [0].

My apologies, it's been a few years since I last looked at your server locations (so I didn't remember) and I was probably getting the wrong impression from the fact that your post is only mentioning server locations in Sweden.

> Right now there is no Swedish law that can compel Mullvad to start logging [1].

But at the same time all cross-border traffic in and out of Sweden (so for anyone using Mullvad outside Sweden: virtually all traffic) is being monitored and (probably) logged, isn't it?


I highly doubt your VPN traffic will pass Sweden if you're outside Sweden and signing onto a Mullvad VPN server located outside of Sweden. The server list may be fetched from Sweden I guess? I haven't looked at the apps traffic to be honest, I have huge respect for the Mullvad team.


Sorry, I should have been more precise: For everyone outside Sweden using Mullvad's Swedish servers. My point was just that the fact that "[r]ight now there is no Swedish law that can compel Mullvad to start logging" does not mean that Swedish intelligence agencies cannot collect any data about you (= the Mullvad user using their Swedish servers).


Locating all their servers in IndependiPrivastan probably wouldn't stop intelligence agencies from collecting data. The NSA, for example, does not care even a tiny bit about the privacy of non-US citizens and sets up equipment around the world to gather and analyze internet traffic.


I agree. But the NSA's resources are also vastly bigger than those of the Swedish intelligence agencies.


I have so much respect for everyone at Mullvad. You are the only VPN provider I trust! I have been a user for years now, and it has always been 5 dollars a month, which is quite generous. It is so cheap that even the poor can afford privacy. You guys have put a ton of effort into making your service as privacy respecting as possible. Not only that, the tech is on the bleeding edge (WireGuard, socks5, etc.) built right in. As a cybersecurity researcher, I could not be happier with the product. I hope you stay true to your mission, thanks again!


Also, will Monero ever become a supported payment method?


How is this nonsense the top comment?

OP's been doing a poor job of "following Mullvad for a long time" and apparently has never used Mullvad or visited its website. Heck, it's on their Wikipedia page[1].

If they had, it would be immediately apparent that Mullvad has servers all over the world. They have for many years -- perhaps since it's inception. It takes a bare minimum of effort to learn that.

1. https://en.wikipedia.org/wiki/Mullvad#Service


Your annoyance is understandable, and yet this question and direct response from a founder is a very informative interaction to have recorded in this thread. Much more informative than if GP had simply stated the results of their Wikipedia research.

Sometimes a mediocre question is the landing pad for a great answer. No need to begrudge the question.


While I agree the founder's response was informative, it does not justify the completely erroneous statement/accusation that prompted it.

That most certainly should be called out -- especially when it was, at the time, the top voted comment.


People make mistakes. Maybe if you were less you combative your comments would get more upvotes.


I didn't feel like I was being combative, only pointing out OP was misinforming people. They didn't do even basic research, contrary to their statements. That negligence should be called out.

And I'm not here for upvotes. Who's being combative, again?


I have apologized for the factual incorrectness of my comment and explained how I had arrived at that conclusion / brain glitch: They are only mentioning Swedish servers in their blog post and for a second I had forgotten that, like any other big VPN provider, Mullvad of course has servers in many locations around the world.

What more do you want? Upvotes are not under my control and I also cannot edit my original comment anymore. No reason to get personal.

FWIW, I still think that it's important to raise awareness for the deficiencies of Swedish privacy law (irrespective of Mullvad and where their servers are located). I suspect that at least some of the upvotes were also given because people agreed with that.


Because it's a swedish company? The location of the servers is kinda irrelevant in that regard. They'd have to provide government access if there is a lawsuit that demands it. If there isn't one than your critique is entirely pointless


> Because it's a swedish company?

I'm not sure what point you're trying to make. If online privacy is as important to them as they say, they could have easily registered their company (or a subsidiary) in a different EU member state.

> They'd have to provide government access if there is a lawsuit that demands it.

IANAL but I am not entirely sure this is true in the EU. Either way, my question was "Why are the servers located in Sweden?" Whether or not this due to the company owning the servers being in Sweden is irrelevant.


I think you’re overthinking this. The people who run the company are based in Sweden. So they registered the company in Sweden, because that’s where they are. Then they hosted the servers in Sweden, because that’s where they are and where the company is registered.

Registering the company somewhere else wouldn’t do them any good when they’re living in Sweden, because the legal system isn’t fooled by sleight of hand like that. Likewise, hosting the servers elsewhere from where they’re based. Both would expand the number of entities with the ability to compel them to disclose data, because as long as the company owners live in Sweden, Sweden has that ability.

Unless you’re asking why they didn’t move to another country to start their company, which is surely a larger ask than the “easily” you suggest.


You are allowed to start companies in other countries, and thereby avoid local laws, without moving i.e. changing you country of residence.

If you believe any different, please say why so. Just "wouldn’t do them any good" is pretty meaningless.


I assumed it was pretty clear why it wouldn’t do them any good:

All of their executives and their staff are in Sweden. It doesn’t matter if the company is registered on Mars, the Swedish government can come knock on their doors, because Swedish laws apply to people in Sweden.

The most mundane way to demonstrate this is to imagine they don’t register a company at all. If a bunch of Swedish people get together and start doing business w/o registering a company, it’s clear that Swedish law applies to them. Why would filing some paperwork with a foreign entity grant them immunity from the laws in the country they live and work from?


A server in Sweden cannot easily be raided by the Swedish, is the first reason.

The second reason is "Swedish laws apply to people in Sweden" seem to make assumptions about what the government can force people to do, or specifically, punish people for not doing. In many cases, authorities just threaten/raid the data-centers so never have to bother take that route.

Lastly, I'm not sure this is true: "Swedish laws apply to people in Sweden" - I'm not sure this applies to Swedes working for foreign corps, there are a whole load of laws that apply to local corps only. In fact, that are laws that apply to Swedish corps even when their staff reside abroad - unless "government can come knock on their doors" is a reference to physical coercion.


This isn’t really responsive to what I’m saying or what you asked me.

I didn’t make any assumptions about what Swedish law can or cannot do. Swedish laws apply to people in Sweden. If Swedish law says that you can’t use Helvetica font on your website, and the punishment is 10 years of hand-tracing a better font on stone tablets, then they’re able to apply that law to a Sweden-based web developer, regardless of whether or not he works for a company that’s registered in Spain.

Likewise, yes, the Swedish government surely has many laws with carve outs for different use cases. Taxes are a great example here: there are laws that apply only to activities of foreign corporations, and laws that apply only to local corporations. But the Swedish government gets to make those laws and determine which apply to whom. Likewise, you are correct that Sweden can make laws that apply to Swedish corporations even when their staff reside abroad. This is because by registering in Sweden, the business has given the Swedish government a measure of control over their activities.


> I didn’t make any assumptions about what Swedish law can or cannot do

> If Swedish law says ... they’re able to apply that law ...

This is a big assumption, and depends if you mean literally that they can do this, or if they can do so sustainably. Any country can violate international practise, but are unlikely to do so (at least in Europe) because of the consequence on international relations.

A law on Helvetica font would require legal authority. Very often, companies themselves are help liable for the actions of a company - laws that allow the government to punish individuals would have to specifically criminalise the act even for locals acting on behalf of those corps. These kind of laws are much rarer, at least in US/Europe, and not the kind of law we are talking about here which appear to apply to corporations. There is a good reason for this; as soon as any nation officially declares it would punish individuals like this, corps will leave - or at least no longer employ natives into decent positions.

> But the Swedish government gets to make those laws and determine which apply to whom.

Technically, but not really, they have to remain compatible with their international agreements, and their economic ambitions.


Laws that apply to companies follow different rules than laws that apply to individuals, it feels like you’re conflating the two..

Just because employees reside in Sweden doesn’t mean the company resides in Sweden, legally.


Companies consist of their employees, who have to live somewhere.


A company is a legal entity that is, in many ways, separate from the employees it consists of. I'm surprised there's any disagreement on this point.


> It doesn’t matter if the company is registered on Mars, the Swedish government can come knock on their doors, because Swedish laws apply to people in Sweden.

But it does matter. In most EU countries limited-liability companies (like the Swedish Aktiebolag) are legal entities that are completely separate from their owners and employees. Your idea of Swedish authorities "knocking" on people's doors (who own a company registered abroad) and "convincing" them to hand over customer data appears to be more along to lines of https://xkcd.com/538/ but in this case (in the particular case of a country like Sweden that has a well-respected legal system) it doesn't seem to be grounded in reality.

For instance, Swedish law likely compells companies to hand over customer data under certain circumstances. But if you're the "just" the owner of that (limited-liability) company, the company's customers are not your customers, so authorities cannot compell you to give them access to those customers' data (because you are a separate legal entity).


Your theory is an American could start a company that violates US laws so long as they form the entity somewhere else?


Depends what you mean. US law takes into account the existence of foreign nations already; some explicitly end at the borders, others not so.

It also depends on which country, and to what extent agreements exist between those countries wrt policing their own territories. Those that don't have such agreements, are often also limited in what extent they can do business in the US.


GP didn't say anything like that. They were talking about

> avoid[ing] local laws

(i.e. legally) which is a whole different matter.


How is that different from saying Swedes can avoid Swedish law by incorporating somewhere else?


This is not what you said. You were talking about "a company that violates US law" (emphasis mine).


Completely false.


> Registering the company somewhere else wouldn’t do them any good when they’re living in Sweden, because the legal system isn’t fooled by sleight of hand like that

This isn’t true at all, at least as long as we assume that you’re dealing with the courts and not some secret police.


Really? How easy is it to create a company in another EU country.

In the US, it is mostly painless to create a company in another state. Fill out some forms online and pay a few hundred. If you don’t have a presence there, you have to pay for a registered agent in the state. That’s about $100/yr. You may also need a mailing address, but you can get a post office box that will accept and scan your mail for another hundred bucks a year. Star and local taxes.

What about the EU?


To my knowledge here in Germany registering a company is pretty much the same whether you're from Germany or from a different EU country. You don't need to live in Germany or anything.

Of course you still need to know your way around local taxes, legal obligations (of the shareholder, of the company) etc. etc. but that's the case anywhere in the world.


In the EU you will find very different laws and tax systems depending on the country. Quite large differences in regards to culture as well.

One example: If you're a cross-border worker you'll likely have to file multiple tax decelerations (one for each country), which can be quite complicated if you don't speak the local language (and even if you do).


> If you're a cross-border worker you'll likely have to file multiple tax decelerations (one for each country)

Yes, if you have sources of income in multiple countries, you might have to. But if you're (only) working on the other side of the border and this is your only source of income, agreements on double taxation and tax harmonization between the EU member states should actually prevent that. Heck, I even worked in the US once, declared my taxes there and didn't have to do my taxes back home in the EU anymore.


You don't have to use the servers in Sweden if you don't want to.


A bit tangential to the main post, but I'd to share a recent positive experience with Mullvad:

I am a regular user of Mullvad and recently wanted to try a different VPN, that only provides Wireguard configs (i.e. no native app). I used the default setup.

For some reason, my internet connection was flaky, and when it disconnected and reconnected, my traffic leaked.

That never happened to me with Mullvad as the app comes with an "Always require VPN" option out of the box and it has always worked reliably.


On linux you can create a network namespace exposing only the wireguard network device, so that applications in that namespace cannot leak traffic. Setting this up, however, is quite fiddly in my experience.


In addition it is probably not a bad idea to block all traffic on wlo1 / eth0, except that to the mullvad server ip's, through some ufw rules. If you forget to configure the namespace for some applications then, it is highly unlikely the app has internet access (ie, it would need its own mullvad/vpn implementation included).


I have this setup in my homelab, use it to isolate networks by role. The bash script to setup and tear down the namespace is here:

https://github.com/VTimofeenko/wireguard-namespace-service/b...


I would suggest vopono to do this automatically.


It’s easier and more secure to just create a VM that’s bridged to the VPN interface (regardless of protocol) if you don’t use the VPN for everything but the things you do use it for absolutely must go through it.


I think I like this idea the best - simple, effective, and unbreakable due to config changes or updates.

Plus it gives you a psychological separation between "VPN related activities" and not. Or you just do everything in the VM. Adds a layer security wise as well to protect your physical system.

If you wanted to get really fancy you could have a few different VM's and each one on a different companies VPN


Any “always require vpn” option is a game of cat and mouse and is going to leak traffic at some point, whether easily detectable or otherwise. As others have said, you need to set up a secure environment that only has that one and only option for accessing the outside world.


Agree, Mullvad provides really good VPN service. I faced almost zero downtimes / speed throttles. It establishes quick connection with server (maybe because it uses wireguard). Anyway, I'm a regular user and I think paying 5E worth it.


> It establishes quick connection with server (maybe because it uses wireguard)

I'm actually kind of curious about what Wireguard does here. I think Wireguard says it's connected almost immediately even when it isn't, presumably holding traffic back locally while it waits for the connection to be active. I was wondering because I spent some time confused by a non-Mullvad Wireguard connection that wasn't working (turns out the server wasn't available at all) that nonetheless appeared as "connected".


network-manager UI, if I recall, just shows that the WG connection is turned on, not that there are packets coming back. (It's not a client-server, so this "makes sense")


Seconded. Very occasionally I'll have to swap servers in a location but that's super infrequent and it's not exactly a primary tier location that I'm using. One of those things I can generally just setup and leave working.


Can you share the WireGuard config? I suspect it is either missing firewall rules, or they're badly set.


I love the concept and I even wrote a blog post about how to set up a fully pxe bootable server environment using Alpine Linux [1] (which by default boots from RAM) in 2019. I still use it and it's one of those things that makes recovery or testing so much faster because I don't even need a usb thumb drive

[1] https://blog.haschek.at/2019/build-your-own-datacenter-with-...


This is awesome, glad that Mullvad is heading in this direction.

For reference, ExpressVPN (which has been audited by PwC) introduced this in 2019 [1].

Unfortunately, ever since ExpressVPN was purchased by Kape Technologies (they also own PIA, Cyberghost, Zenmate all of which do not have reliable histories); Mulvad has been the clear choice for a while now. They're also the backend for Mozilla VPN (mozilla just whitelabels from Mulvad [3])

  [1] https://www.expressvpn.com/blog/introducing-trustedserver/
  [2] https://www.expressvpn.com/blog/pwc-audits-expressvpn-servers-to-confirm-essential-privacy-protections/
  [3] https://news.ycombinator.com/item?id=26646510


Some information that could be of interest to those running VPN servers.

I live in Kazakhstan and recently our government decided to shut down the Internet. But apparently there were ways to get out: they did not filter two TCP ports. My guess it was some "backdoor" put by employees who had to obey the orders but wanted to provide people some way to get around those blocks. Those ports were used to run VPN software. I used Outline VPN on my VPS and it allowed me and my friends to have a working Internet.

TLDR: allow specifying port and protocol (TCP/UDP) as some kind of advanced option for those users who need it for some reason.

Right now we've got Internet back and it works fine, but who knows when our government will decide to shut it down again.

PS mullvad.net website apparently is blocked in Kazakhstan as well. I know that they block popular VPN provider websites, so that should not come as a surprise, but still. I have no idea whether actual VPN subnets are blocked or not.


I think it is probably more likely those ports were the backdoor the government or their allies was using to function. Everyone always builds in a backdoor for themselves!

When I hear about things like this it makes me glad I have a simple satellite communicator - it will only do short text messages but that is a hell of a lot better than nothing. Of course one could get a full satellite based mobile internet device or phone but the plans on those get quite expensive.


> If the computer is powered off, moved or confiscated, there is no data to retrieve.

Don't forget to add insta-shutdown when any USB device is connected to the system!


And a tilt sensor that cuts power if the chasis is moved


Or disable usb in bios entirely


Disabling USB in BIOS only disables the emulation of classic PS2 keyboards and IDE storage so that old OSes or bootloaders without USB stacks can work with modern equipment. As soon as the OS kernel initializes the PCI bus, USB will work again - however they could go and remove the xHCI modules from the kernel and image.


Mullvad has a custom-built bare metal UEFI implementation based on coreboot, I assume stboot is an evolution of that, which means it takes as close as you can get to full responsibility for initialization of all system components like processor, chipset, Ethernet, USB, everything.

As a result they can absolutely disable USB entirely by never exposing those parts of the device tree to Linux.


x86 devices do not have device trees, and for ARM I'd take a guess and say that as long as the PCI root port is exposed to the OS, a PCI re-scan will be enough to wake the USB chipset.


The VPN provider I use have had diskless systems for years. Is this the same thing?

https://www.ovpn.com/en/blog/improvement-of-the-physical-sec...


It's a trade-off. If you have no disk, the disk can't fail, but the network can, and the remote PXE server can, and the remote SAN can. You can get into a state where you have to pray no servers reboot. Intermittent errors can be real annoying when it makes provisioning fail. (used to work a server farm that'd do server rebuilds over PXE, and ran a few diskless cluster projects)

An alternative is you use a RAID array and mount your disks in read-only mode, or use physically read-only disks and when you have to replace a disk, you pre-mirror the replacement disk. In this way the local disks can be replaced as they fail and there's never a point when the server is at risk of not being able to boot.

......or they could boot from CDROM :)


A network or PXE server can fail regardless, so this are things that always have to be taken into consideration and in those instances then you address those issues. With this type of setup you do not need a remote SAN as it would defeat the purpose of not having external storage that could store logs. Mullvad has servers all over the world, so a temporary failure in one location will not bring down their entire infrastructure.


It's not just a temporary failure, it's potentially the entire AZ going down hard. High Availability network boot without local storage is very difficult/expensive.

They can still use local disks to provision the OS over a network but boot from local storage, and prevent writing to disks from the booted OS (hell, they can completely remove the disk drivers from the kernel!). It just doesn't make sense to ditch the drives from a reliability standpoint. They're going to have a big outage one day just because they didn't want to deal with drives.


Mullvad and similar providers often colocate or rent servers from multiple local hosting providers. A group of servers going down for them would not be a big deal. Network boot is not difficult/expensive. Many of their servers are using 10Gbit uplinks+ so I take it they get pretty good deals for bandwidth. It isn't like Amazon or other cloud providers that charge an arm/leg for egress.

The point of not using local disks is again fairly straightforward, to show that they do not have a stateful storage medium to write logs to. Whether it significantly helps or not is beside the point, they have determined that it helps provide assurance to their customers and additionally showcases a feature for auditors.

Network booting loads the OS into RAM, so even if there was a network outage they'd have to restart the severs to cause a problem. From what I know of most VPN solutions though, again a network outage would only affect the group of servers at that data center which isn't their entire operation.


Every year I feel more and more proud to renew my subscription. What a great company.


This is hardly a new thing in VPN providers though. I know that perfect privacy[1] and azire vpn[2] both advertise this feature already.

[1]https://www.perfect-privacy.com/en/features/without-logs [2]https://www.azirevpn.com/docs/environment


if only there was any proof of this actually being the case and there not being some "accidental" debug log enabled, or some other network level component having "accidental" access to the keys.

There's just no good answer to perfect trust-no-one private internet access.

If you need to hide all of your traffic from other users in your local network, you can accomplish that in a trust-no-one fashion by running your own VPN endpoint on a server you control which provides better privacy guarantees compared to a centralised commercial VPN whose business model will eventually involve selling your data (once user growth stops but shareholders demand continued revenue growth).

But if you need to hide your traffic from anybody but your peer on the internet and you need to hide the fact that you talked to that peer, then, I'm afraid, your out of luck.


> If you need to hide all of your traffic from other users in your local network, you can accomplish that in a trust-no-one fashion by running your own VPN endpoint on a server you control which provides better privacy guarantees compared to a centralised commercial VPN whose business model will eventually involve selling your data (once user growth stops but shareholders demand continued revenue growth).

Well not really. There was a great (german) interview with the perfect privacy founders recently [1]. They seem to be decent guys with close ties to the Chaos Computer Club and I strongly suspect they wouldn't want to work like that.

[1] https://www.youtube.com/watch?v=VMr0gJvI-6I

> But if you need to hide your traffic from anybody but your peer on the internet and you need to hide the fact that you talked to that peer, then, I'm afraid, your out of luck.

Nah, that one is easy just use an anonymous sim card or an open wifi and your good to go.

Honestly these discussions often feel pretty asinine to me. I personally use paid VPNs to pirate to my hearts content, work around my ISPs terrible networking and a little bit of geo-unblocking. Of course you can't use these services to protect yourself from three letter agency type surveillance or equally powerful threat actors but if they are "private" enough to block the music industry and their lawyers from suing you that's a pretty high standard of privacy, certainly more than any ISP alone gives you!


Do you actually pirate music or did you give it as a general example? I feel no need to pirate music today with all the music streaming services especially since I can find all the music I want on all the streaming services which is a world of difference compared to the video streaming services


I stopped pirating music a while ago when spotify became better than what the trackers i was on delivered. That being said, i have recently started looking into it again since spotify is dragging their asses on high quality streaming and their app support on linux started to annoy me. The alternative streaming services barely support linux at all so they aren't really an alternative for me. But you are right it's mostly tv-shows and movies, a few books here and there. It seems I've basically missed the golden age of netflix (or there never was one in germany with their shitty catalog) and stayed on private trackers until now. I suspect it won't change any time soon either with all the fragmentation going on and i absolutely refuse to deal with their stupid DRM measures.


Have you tried youtube music (via youtube premium)? I'm not sure about the high quality audio part because I only listen via bluetooth. The recommendations, general app UX and the fact that I can listen via website on desktop have made me cancel spotify.


For low quality audio it's fine, though there's a lot of music I like missing that spotify has. Quality wise it's all over the place and at best as good as spotify.

Generally I don't care much about the UI of any of the services offered and being browser based doesn't really make it any better for me. I can do that with spotify and most other services as well. What I would like to have is a simple paid service with high quality flacs that has an open enough API that i can use many of the great open source tools available and download music for offline use on my phone (data caps and all) without jumping through a lot of hoops. It's not music management is an unsolved problem and for local music i have tons of great options on all my devices from TUI applications to applications with great desktop integration to great open source phone apps. With Spotify there are at least some projects that work somewhat but not really well and certainly nothing that i can easily integrate into my desktop or phone without relying on proprietary clients.

But honestly before I go around trying endless services to get a decent experience I'd rather just take the red music tracker test [1] and build a local collection that "just works" and be done with it.

[1] https://interviewfor.red/en/index.html


Have you looked at https://roonlabs.com/ ? I'd be interested in your opinion if you have


>> If you need to hide all of your traffic from other users in your local network, you can accomplish that in a trust-no-one fashion by running your own VPN endpoint on a server you control which provides better privacy guarantees compared to a centralised commercial VPN whose business model will eventually involve selling your data (once user growth stops but shareholders demand continued revenue growth).

the privacy protection for most people using VPNs is required against their ISP and other actors looking to analyse their traffic, not users on the local network. a commercial VPN will be better for privacy due to the crowding effects, ie. large number of users sharing the same IP and protects against correlation attacks - it's much easier to trace the activities on your own VPN endpoint back to you. of course you need to trust the operators, which is as different question.


>ie. large number of users sharing the same IP and protects against correlation attacks

Depending on where you are based in the world (see https://www.submarinecablemap.com) realtime throttling of vpn traffic can still identify a user and where they are going in some cases.

You can get a degree of privacy from visiting websites located on servers in big data centres, but nothing a search warrant couldnt find out retrospectively.

Just traceroute your journey inside a vpn to see where abouts you are going when connecting to a webserver anywhere in the world and workout the physical route you are travelling on the cable map.

Obviously the number of languages you speak also restricts where in the world you will be going online to a point and timezones can also make you stand out like a sore thumb if you visit a website when the locals generally arent.

I've identified (US) websites which can workout what DNS server you are using, so in my case, based in the UK if I swap from using a UK ISP dns to using another dns like quad9 in Germany, the (US) websites alter the content you can see, just on that single DNS server change.

There is no privacy!


>There is no privacy!

I think this is a good message. In the same vein, there's no security either. All you can do is make your and your adversaries' life harder, and balance the different tradeoffs.


> there's no security either.

Dont buy that, care to elaborate?


In the same line of thinking as the parent comment, there's no 100% security either. If you loot at IT, everything can be hacked, secrets leak, intelligence agencies hoard vulnerabilities, or even have insiders in security firms or larger corporations.

In the real life, no lock is invulnerable. Most can be picked, frozen, melted, etc and surely have other weaknesses too.

But to achieve their goal, they don't need to perfect. Just reasonably good. And so, I wish for people to be mindful about the nature of these. That they are not perfect, they are not hidder, nor secure. Just, maybe, reasonably so.


Your lock example isnt perfect, it cant fight back.

In IT, you need both joined up offensive and defensive measures which includes self destruct if secrets need to be kept. That is at best a Check Mate.

Take a VPN, in nearly all instances I have encountered the only traffic is genuine traffic, there is no dummy traffic to muddy the waters from external Deep Packet Inspection.

Likewise routing can be used to isolate, I'll give you a real world example which you might be able to relate to.

You are travelling by car from A to B, and you can take a variety of routes to get there. Most modern cars now have built in sat nav, and all you know is when your target is leaving and they will be using the car manufacturers satnav. So you have a window (at the start of their journey) in which to manipulate the targets satnav by giving it fake traffic data to cause it to take a particular route. Ergo you have been able to isolate your target onto roads they wouldn't normally travel. Now that can be done nationally over the radio station network, or nearby using a transceiver SDR in a chase car.

What makes you think the internet is any difference? Business efficiency like JIT is a weakness as we see with the chip shortages and other problems caused by covid lockdowns. VPN companies are no different, they need to maximise profit so they dont add in fake traffic to hide their customers traffic, and by virtue of being able to choose from multiple VPN providers, users self isolate themselves into yet smaller groups. VPN providers should really organise and share networks to further muddy the waters from external entities.


Is no-trust ever possible? I thought people create their threat models and verify they can trust those they have to trust.


> people create their threat models and verify they can trust those they have to trust

What kind of people?

How do you verify you can trust some company?


Everyone does it informally to varying degrees for varying problems (often times subconsciously). E.g. "What do I know about this person? Can I trust them around my kids?"

You verify trustworthiness by research. Who is involved? Do I trust anyone who trusts them? What are their motivations? What would cause them to take action against me? What causes them to protect my interests? What laws are they subjected to (i.e. who can coerce them)? What do they say for themselves? Where do their words fall on the credibility to BS scale? What is their reputation in the community? What do their competitors/adversaries say? What would cause their behavior to change?

I won't enumerate all my research on Mullvad. I can say Mozilla attaching their brand to Mullvad's services helped me a lot (trust by proxy). I'll also say that some of their product decisions give credibility to their anonymity claims. Lastly, I found someone who shared a competitive analysis across many providers. I found the analysis trustworthy. Mullvad has some weak points, but was still the best provider for my particular use case.


That is the great thing about mullvad. They don't have shareholders except the actual owners.

https://mullvad.net/en/blog/2021/9/16/ownership-and-future-m...


which is true until one or more of their owners decide to sell their shares.


my grocer down the road is a nice fella and has tasty vegetables from sustainable sources, but he might get bought up by a big supermarket chain, so I'm not going to buy from him


your grocer does not have the ability to retroactively change the food you've bought from them.

A VPN provider can "accidentally" enable logging prior to the sale


if you trust them now with not logging and selling your data, you should trust them not doing anything to screw you over, and giving you sufficient warnings before a sale happens so you can re-evalute that trust. if you don't trust a provider, you shouldn't use them. "i don't trust any VPN providers" is a reasonable position, but your argument saying "all VPNs will end up selling data" is still flawed.


Wouldn't the ownership of the server be easy to trace back to you?

PIA has been promising a fully audited and verifiable infrastructure in the future:

https://www.privateinternetaccess.com/blog/dont-trust-verify...


>Wouldn't the ownership of the server be easy to trace back to you?

yes. Which is why I said that this helps to shield your traffic from other people in your current local network (think: coffee-shop) which is one use-case of a VPN.

If you need to protect your traffic from anybody but your peer (another potential use-case of a VPN if this were possible) and you even want to hide the fact that you were talking to that peer, then you're out of luck. Period.


You are ignoring the use case of protecting my data from my ISP, who is a known bad actor that wants to sell my data and had the power to strongarm my government into legalizing that data theft.


> There's just no good answer to perfect trust-no-one private internet access.

What about Tor?


I think that if enough exit nodes would be owned by let's say government agencies they would be able to correlate requested domains with actual requester IP.


We all should run a Tor node then to outnumber the bad guys


In addition to the traffic analysis mentioned in another reply, there are ways data can be leaked from Tor. One example from the crime documentary "Hunting Warhead": a white hat hacker managed to locate a darknet server running a forum software by setting his avatar image to a file hosted on a domain he controlled. The forum software retrieved the age via a regular internet route, exposing the actual host IP.

For maximum privacy, Tor should be used with software designed for Tor from the start.


Not sure I would call Tor "perfect", but it's certainly very useful for some use-cases.


https://archive.is/HbjIQ

(it was blocked for me because it's a VPN service provider)


The server configuration (and therefore customers account numbers) is stored in Server OS images I suppose, right ? It shouldn't be an issue as far as inspection is concerned, should it ?

Also, isn't there a law that enforces logs to be kept for n years ? How is it compatible with diskless setup ?


No, they would not need to be stored in the images. They are deployed as part of provisioning package according to the post. If they were part of the OS image, they would have to update the image anytime a new customer is added, which would be far less optimal than having Ansible or whatever push changes. There is no law in Sweden that requires logs, and the point of a VPN provider is to not store logs. If your VPN provider is keeping logs of your connections then you should look for a new provider.


Isn't this the same as just running a boot cd or PXE server and running all the data out of RAM drives? I mean we've been doing this for years on linux as hobbyists haven't we? Or does this bring somethign new to the table?


Pretty similar to the default (diskless) mode in Alpine, though it lacks the tooling to verify persisted data and the sources apkvols can be applied from at boot aren't that well documented.


How do you debug when stuff goes wrong?


The next step: dedicated VPN over FPGAs


*drools*


Wait, Cisco already do this


They do? Do you have any product names you can share?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: