Hacker News new | past | comments | ask | show | jobs | submit login
NordVPN confirms it was hacked (techcrunch.com)
1368 points by afshinmeh 29 days ago | hide | past | web | favorite | 642 comments

> The attacker gained access to the server — which had been active for about a month — by exploiting an insecure remote management system left by the datacenter provider, which NordVPN said it was unaware that such a system existed.

This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?

I work for a web hosting company in the US and at least in our case, it's quite common for remote management to be enabled on pretty much all of our dedicated hardware. However, because of the inherent dangers in opening this up to the public internet, unless explicitly requested by the customer (or Managed Colocation), the NIC used for Dell iDRAC or HP iLO is on an isolated network unique to the physical datacenter. Remote access for our techs is managed through a secured bridge that requires all sorts of security hoops on our company intranet, and remote access for general internet traffic is not available due to the firewall restrictions. While it's plausible for remote access to be gained this way, it is extremely unlikely and would require several exploits at different points along the path.

I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.

IPMI does not have to be open to the internet to be open to a wide audience. Many of these out of band management interfaces are hosted on an internal network, but not isolated by customer.

Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies.

Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.

Which cheap data centers are you referring to? Curious as someone unfamiliar w/ the space.

Generally speaking, there are four (4) tiers of "public" data centers are on the market, ranging from essentially a big room with some alright AC and a line out, to huge, highly secure (cameras, fingerprint readers, SSAE certifications, etc.) buildings with redundant power and HVAC systems.

The higher end ones are usually newer-ish, but there are lot of older "computer rooms" that offer acceptable-level benefits for a reasonable price. Lots of legacy customers and ISPs in these rooms, for the record. You get what you pay for but a lot of older DC spaces are just fine for most users; everyone thinks they need 99.999 but most don't.

There are also re-sellers and managed services companies who take out a footprint in data centers and then lease space in their cabs, sell bandwidth, IPs, etc. Using a series of resellers you can often get around restrictions as to what you're doing -- small fry MSPs don't ask a lot of questions -- but still get a data center footprint. These are sometimes one-man shows, and their quality and professionalism are often sub-par, which is how you end up with default iDRAC creds and the like.

Check out Data Center Maps or WebHostingTalk for some examples.

Source: data center ops manager for several different companies.

My social network for dog walkers definitely needs 5 9's. Thank you for the insight.

I have decided to contact them directly rather than publicize.

Sorry hacker, not today!

Haha! If I was up to no good I wouldn't be using my real name in my handle to ask such questions :)

Maybe it wasn't open to the public Internet, but the VPN exit is inside the datacenter and connects out to the public Internet. Is it feasible that NordVPN provided their customers with a secure tunnel into their own datacenter's management software?

Sounds like an iDRAC exploit (assuming Dell servers).

But, yes, remote management is pretty common in datacenters. The fact that NordVPN wasn't aware of them just shows incompetence.

User root, password calvin. That's the default. And, if I had a dime for every time I've seen one of these in a data center, I'd be a rich man. I have literally begged sys admins to change the default password, but they say, "Why... we're behind a firewall using RFC 1918 addresses. No one can get to these." The rest, as they say, is history.

> we're behind a firewall

This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.

In well maintained networks the management interface (IDRAC, etc.) for each server is placed on a separate VLAN which the servers cannot access. This isn't to say that cheap providers actually do this, or that the VLAN can't be accessed by a compromised technician's workstation/laptop.

So it's a fail-open design, given the rarity of well maintained networks, and the lability and inobservability of said state.

Never trust the network.

Yes, this kind of firewall is always supposed to be between the management hosts and everything else. Only the sysadmins at the data center a very limited set of applications is supposed to be able to access it. The very real risk is misconfiguration.

Depends on what kind. In case of idrac, yes; but it's weird that it was insecure by default in the first place. Usually credentials are configured and provided to the customer. Makes me think there might have been some other interface. Clarification is definitely needed.

It could have been something like https://www.zdnet.com/article/vulnerabilities-found-in-the-r...

There were many IPMI/iDRAC/etc. exploits published in the past few years. Throw a dart at a list of them, and you'll probably find one that was unpatched in most systems as of March 2018.

HPE iLO also had critical vulnerability: CVE-2017-12542.

How the hell do you pwn a server with iDRAC?

If you can reboot it without anyone noticing? Really, really easily: iDRAC gives you access to the local console, like a remote KVM. Reboot into single user mode, change a password, done.

Oh, IPMI and friends are a total mess. Some implementations allow one to take control of a running server remotely especially if they use a shared ethernet for management ( popular in supermicros ). I once had our security geek demonstrate it by taking over the running server, rebooting it using network emulated USB stick, adding a file into /etc and rebooting the server again.

In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.

The first time I booted a server using a virtual CD-ROM (iso on my laptop shows up as a hardware CD-ROM on the server) over IPMI I was simultaneously relieved (because I could fix the machine remotely) and absolutely totally horrified.

iDRAC is a full onboard whitehat rootkit manufactured and supported by Dell. It runs independently of any OS and has control over the system. It is intended to be a substitute for physical access.

In certain configurations, iDRAC gives you an rdp connection. If idrac is left at default, windows admin login not being changed isn't too much of a stretch.

iDrac have a default password : https://danblee.com/dell-idrac-default-username-and-password...

There also quite a number of sysadmins that connect idrac's to the "regular" network, instead of the sysadmin VLAN ...

Pointing fingers without having the details at hand is not competent either.

I’d guess that the DC got owned, no need for iDRAC exploits when lazy VPN company staff never changed the pws.

It doesn't make much sense to me, even with iDRAC/some other console access you don't really have access to OS unless you reboot & go to single user mode etc at which point they should be noticing their servers rebooting etc. would love more info

Just set up your code as a boot-once config and wait for the owner to reboot their machine. Make your code end by booting the installed OS (or even by just rebooting again, most people will just curse about the damn slow server boot process).

You can't do that as you don't have any access until it's being rebooted. It's basically like you're standing in front of the machine so there's not really much you can do when you're just looking at a login prompt, you have to be able to stop grub from just booting with the default options and instead boot up using init=/bin/bash or maybe if the server supports iPXE you can just chain load some payload off the internet.

You can manipulate boot settings using BMC commands. No need to mess with Grub or the running system. Instead, tell the system to boot up from an emulated USB drive (image can be attached from some remote server, often including your web browser).

Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines).

Even SecureBoot won't help as you can just turn it off using the BMC.

See here for a bunch of examples for Dell machines using the BMC's HTTP API:


Why would they notice their server rebooting? And why would they not just assume it was a glitch or power failure?

When someone gets notified by their monitoring system that a server was unavailable (because it rebooted) they might investigate and see that the IPMI logs don't mention power loss

Power failure would require both of the power feeds in the DC failing simultaneously and would be easily verified by contacting the DC and asking if they had any power outages reported at the time. Of course there are cheapskates who don't go for redundant power supplies so it's possible but would be indicated in the IPMI logs

Servers can reboot for any reason. There are tons of kernel issues, especially since Meltdown & Spectre, that cause machine reboots in Linux especially on high traffic machines.

I've worked in production environments with thousands of machines and random reboots are a completely normal event for some workloads. Combination of hardware issues & kernel issues with hundreds of thousands of lines of code makes it inevitable. I would be surprised if NordVPN even noticed and their architecture wasn't designed to automatically start everything at boot.

You can't be perfect at scale - you just need to design your work loads to be redundant and fault tolerant.

It opens an exploit chain, in a normal circumstance you are correct. In a malicious circumstance, it is always feasible irrespective of the likelihood.

It seems NordVPN fucked up themselves are are now trying to avoid responsibility: https://www.theregister.co.uk/2019/10/21/nordvpn_security_is...

""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell.

"We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""

Of course you have to worry about all of that! When you don't self host you have to assume whoever you are renting from hirers the lowest paid employees they can get to manage infrastructure for you. That's how they get profitable. You are not outsourcing expertise.

Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.

Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input).

Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.

> Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.

Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s.

And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.

.. AWS surprisingly doesn't.

Why is this surprising? AWS seem to know what they're doing in general, and this is obviously the right policy in this particular area.

Amazon has security critical functionality on an unauthenticated http endpoint on a link local address. That's pretty damn dumb in my book.

That's a great link, but they're looking at "misconfigurations". How is that relevant here?

> Since the metadata service doesn’t require any particular parameters, fetching the URL will return the AccessKeyID, SecretAccessKey, and Token you need to authenticate into the account.

They talk about the AWS “Metadata Service” Attack Surface and how juicy a target it is. I was just providing support for that opinion.

Yes, software that runs on the instance can learn instance metadata. No, that is not a problem. Running e.g. user-supplied scripts on the instance would be "pretty damn dumb", but no one is that dumb. Any widely distributed software that did something shady with instance metadata would get busted PDQ. Just like any widely distributed software that did something shady with e.g. root credentials, which is about the same threat scenario.

It's crappy design which bypasses important security mechanisms of the OS (lower privileged users) by allowing every application with network access to access such critical functionality. One sane approach would be passing this information to the OS through the hypervisor which then exposes it as a properly ACLed file system.

This is like an author of a website vulnerable to CSRF (because it relies on IP for auth) blaming browsers for allowing cross site requests instead of require proper authentication. Except that Amazon is powerful enough to get away with pushing all the effort onto developers and admins.

You can use iptables to limit metadata access to certain users but that takes effort so no-one does it.

I guess a machine-local service that takes ownership of the metadata service and implements additional restrictions (such as limiting access keys to privileged users) might be doable.

Yes, in some situations it will take effort to not allow network access to untrusted software and/or users. For those situations, EC2 is not a good fit.

How is it “obviously” the right policy? If implemented, console input would presumably be part of the AWS API, guarded with IAM permissions like everything else. If you have full IAM permissions, you can already take over any instance by temporarily attaching its disk to a different instance and modifying the data from there. (That requires rebooting, but so would takeover via console input.) Indeed, I’ve had to do exactly that on multiple occasions to fix broken config files on my personal instance; it would have been much more convenient to have console input.

This "personal instance" model doesn't really fit AWS: if the instance is borked then fail over to another one that isn't. No need for console input.

FreeBSD has an interactive kernel debugger that you can use with a serial console. Super useful to track down some things, not usable on AWS -- you'd need to (somehow) do a core dump and hope you can figure it out from there.

If they're going with a bottom-dollar host, it's possible that the out-of-band server management tools were exposed. It's less likely to be a software backdoor, and more likely to be Supermicro IPMI or other baseboard management controller.

I know that public cloud providers like Rackspace and Azure insert their own accounts and services into cloud servers and VMs mostly under the guise of being able to support said servers and monitor them and their health.

True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.

Very few people go to "true data centers". Those are very expensive because you are buying power, space, cooling and cross connects. Racking machines, replacing hard drives, building a PXE-boot infrastructure, building a remote access infrastructure that bypasses the customer facing network is expensive and time consuming.

>> Is this common practice for data-center providers?

Absolutely. We had similar situation with one of the DC vendors.

So what can you do about it?

Full Disk Encryption is an option, but that’ll entail needing the use the IPMI console to enter the password at every reboot, essentially turning it into a manual operation.

One can do LUKS with access via dropbear in initramfs.

If you're lucky during network install, you'll never need out of bound access.

But rebooting is still a manual process.

If you have physical access twice, full disk encryption usually doesn't help really at all. IPMI seems about as powerful as physical access.

If you just ignore random reboots, sure. But why would you?

I don't think that'd be necessary, couldn't you make changes and then wait for natural reboots?

You couldn't, not via console access anyway.

However, if the attacker gains RCE many IPMI implementations theoretically allow for DMA, but this is a significantly more complex attack to mount in practice with no public PoCs available.

- Bring your own network and limit LOM/IPMI to a private network accessible by VPN only.

- Or, request a private network with no connectivity and a VPN device connected on that private network.

- Or, hardware disable LOM.

There's no reason you shouldn't request an ASA from your datacenter provider in this day and age.

Use AWS/GCP/Azure

doesn't that shift the vulnerability/risk to the configuration of the firewall, security policy, and machine configuration?

Those cloud computing vendors know better than give open access to the word to your infra by default.

On Supermicro hardware and maybe others, IPMI has a very dangerous default setting: if you're not connecting the dedicated IPMI port to a network (typically some closed network dedicated to management), it will use the first ethernet NIC on the motherboard (sharing it with the host), possibly making it accessible through the internet (with default, insecure credentials adding insult to injury) or at least neighbouring machines.

Leaving default creds on IPMI/iLO is not uncommon,some providers don't allow internet access,you have to login on the web console and use a java applet to access it iirc. I can't imagine a well reputed provider exposing ipmi to the internet but the nature of their business means they have to diversify server and network providers.

Not sure if it is still the case but a while ago it was standard on OVH servers for them to put some public keys in their root-like's user authorized_keys. I think they used that to perform tasks requested from the web management system.

They mean IPMI. All servers have IPMI and there’s remote root exploits against many versions of them.

> All servers have IPMI

Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions.

It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).

I'm confused why a factually accurate post was down voted. anyone care to enlighten me?

Don't worry about it. It could have been a mistake or a bot randomly voting. Either way complaining about down votes is against community guidelines.

Typically data centers have compliance requirements like SSAE 16 specifies controls around physical access. Most any major retail data center would have that certification and others.

One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.

Yes. Dedicated servers generally have IPMI / ILO / IDRAC / whatever. It's the only way to scale out management of hosts and provisioning.

To the person / people down voting, can you share your ideas for how you can go about providing at-scale management of large estates of physical machines? I've never seen anything else that's both as practical and as affordable as IPMI / IDRAC / ILO systems that ship with servers, that doesn't introduce weird new failure conditions that can impact significantly more than a single host.

If you care less about the pseudo-anonymous-but-not-really shared-IP aspect of using a VPN, and care more about the this-lan-is-sketchy use case, I have had good experiences with Algo [0]. You can just paste in an API key and spin up your own VPN on something like DigitalOcean. And it uses WireGuard!

[0] https://github.com/trailofbits/algo

This. I can set up and connect to a new OpenVPN instance under my own control in less than seven minutes (5:59 last I clocked) from my phone.

Anyone can do this, it’s not nearly as complicated to launch and secure as some would have us believe. (https://github.com/jenh/sevenminutevpn)

You do lose anonymity with personal VPN, but it all depends on your use case.

But the problem remain the same. Whoever manages the network that hosts your instance will see your traffic...

You choose the business model to trust. VPN serve customers who to pay to be private. That seems like a high value target. ISPs serve connectivity, but apparently in the US, spying on their users is part of their core business and have strong local monopolies. Due to fierce competition and trust being pretty much one of the business requirements (I'd expect a lot more due diligence in b2b), hosting providers seem like the least big evil to me.

There are places that rent boxes for XMR if you want a hosting provider who doesn't have your information.

Any examples?

Yes, it's not a perfect solution. You're basically deciding who you trust the least and avoiding just them.

If you already have a DigitalOcean droplet up and running and you have ssh access, you can use sshuttle [0].

e.g. run this from the command line:

  sshuttle -r example.com 0/0 -x example.com --dns
[0] https://github.com/sshuttle/sshuttle

OpenSSH also includes a SOCKS proxy which you can use with no additional software: https://ma.ttias.be/socks-proxy-linux-ssh-bypass-content-fil...

Whether it grants you any significant anonymity is debatable, but it works well for evading content filters and tunneling your traffic onto a more trustworthy network.

Speaking from experience, sshuttle is way easier and more robust than using OpenSSH's built-in SOCKS proxy.

I use sshuttle all the time when working from "restricted" networks (car dealerships, airports, etc.) For some reason, my local Honda dealer has a guest WiFi that restricts outgoing traffic to a small number of ports, and apparently SSH isn't on that list, so I can't push/pull to GitHub. Firing up sshuttle on port 80 punches right through the filter and allows me to do real work while I wait for my oil change.

Another option is Outline VPN, Jigsaw/Google's open source implementation of Shadowsocks:




Shadowsocks is more resistant to censorship from adverse actors (such as the Great Firewall) than OpenVPN.

Outline's user experience is the best I've seen among self-hosted VPN solutions, as it includes apps for both the server and the client. The server app is suitable for use in organizations, and can manage VPN profiles for multiple individuals.

It's an alternative for sure and has specific use cases, but calling Outline a VPN is disingenuous. It's just a Socks proxy with some obfuscation built in.

Shadowsocks handles all of the use cases of a VPN. When all of a device's internet traffic is routed through Shadowsocks, there is no functional difference to the user. This is the default behavior for all Outline clients (desktop and mobile).

There's also Streisand[0], that gives a lot more options.

[0] https://github.com/StreisandEffect/streisand

Came here to post this. I’ve been using streisand for a long time with no problems. I’ve given out logins to a few trusted friends / colleagues and all have had good experiences as far as I know.

Plus I really enjoyed learning about the in’s and outs of setting it up. I poke around in the VM just for giggles.

Ive done this but have found that most services (Netflix, etc) recognize DO as a VPN. Does anyone know of a hosting provider that isn't blacklisted but I can still setup wireguard on?

Run it on https://1984hosting.com. I've heard it's good although I can't attest personally.

The problem with this is that jumping out onto the net from a VPS-allocated IP causes all sorts of trouble for "normal" internet use. For example you won't be able to use Netflix doing something like this.

I can see why Netflix would try to block it, but I haven't run into any issues with it myself (OpenIKEd on OpenBSD on a $3.50/mo Vultr server as detailed here: https://www.snazz.xyz/how-to/2019/09/13/vpn.html). A lot of websites seem aggressive towards Tor users, but my VPS IP address was treated the same as my home, work, and LTE addresses. Are there any other documented cases I should be aware of?

Vultr is a lesser-known VPS, ymmv. I had issues with several websites using Linode. I don't have a list, but iirc it was some gaming related service that took issue with the IP.

Is Algo really 1 ip = 1 user? I always assumed multiple things could be running on one IP since IPv4 is getting scarce, but bare metal networking is not my expertise.

I use Algo for the exact reason you mention ("this lan is sketchy") and have been pleased, but I always assumed even if my traffic was mingling, one (possibly secret) court order would out me since I paid with a CC tied to my real name.

Nothing on this page or on the trailofbits blog article tells me why I should actually use this. Why should I trust DigitalOcean more than <insert VPN provider here>? Especially when it says "Does not claim to provide anonymity or censorship avoidance" - why would I use a VPN if it can't even attempt to provide some measure of anonymity?

Thanks for this. Just got it up and running in about 15 minutes (most of that was waiting on DO setup and scripts to run).

If you are looking to spin up your own vpn server with wireguard and pihole there is an excellent guide here https://drexl.me/guides/wireguard-pihole-vpn-setup.html

I was getting ready to paste the same thing; the command-line instructions make Algo, effectively, a somewhat technical solution, but it really does just work.

And for those who have used various VPN solutions over the years but not Wireguard: it really is pretty magic. It Just Works, with fantastic performance.

What about the data-mining and selling infrastructure of NordVPN, known as Tesonet? Are those intact? Also interesting to know how their legal departments are doing, such as the Panamanian shell and the Lithuanian headquarters.




Thanks for sharing these. I was familiar with the Protonmail business but did not know this all connected to a bigger picture. I never trusted NordVPN... they spent way too much money on advertising and snake oil advertising at that, focusing on meaningless numbers and distractions.

Hopefully you don't have similar news to share about Mullvad...

The claims about ProtonVPN have been disproven.

I would like to hear more about this. Could you share some information?

Check a few comments down, its discussed here: https://news.ycombinator.com/item?id=21321598

Just search for proton in this thread. They've explained what happened themselves.

Besides, the argumentation from that vpnscam website and its followers reminds you of the typical conspiracy retards that follow Trump.

In no world is it excusable to have your ostensible competitor sign your binaries or certificates. They can make all the excuses they want, but it doesn't dissolve their incompetence, and shows they are unfit for running such a user-critical business.

No third party signed their certificates. Just a contracted employee who worked for Tesonet typed in his company name instead of ProtonVPN. That's just the Android keystore, nothing else. Google supports keystore rotation only starting with Android 9.

It's actually not even a contracted employee actually. It was a Proton employee who in 2016 was getting payroll through another company before we had our own corporate entity. Keystore rotation is still not yet available yet in Android, so the old key (which we solely control) can't be changed or modified. Android actually also hashes with the certificate metadata so even that can't be edited separately.

On principle I am not impressed with what happened and I think it's very sloppy. After the Lavabit fiasco we have to be extra scrutinuous about the leadership in privacy-oriented companies. That said, I still have a few accounts with Protonmail and I think the service itself is pretty good.

Source ?

As a ProtonMail client I’d like to see that myth busted too.

There's a couple ways to look at this. On one hand, there's an anonymous website and hundreds of Twitter bots pushing a story that is demonstratively false (just check public records).

Then, on the other hand, you have Mozilla and the EU (which has access to all European corporate records) vouching for Proton (since they partially fund Proton). We also operate in a highly transparent way, so all information debunking this is actually in public record, details here: https://protonvpn.com/blog/is-protonvpn-trustworthy/

Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. But that doesn't mean Proton on a whole is based in Vilnius.

The people spreading the false information are also falsely implying that Proton's subsidiary controls the Swiss parent company, which is never the case as it's always the other way around (parent controls the subsidiary). And its super easy to disprove because unlike most companies in the VPN space, the directors of Proton's Swiss parent company are in public record, and are all well known people who have been in the public eye for years (e.g. at TED: https://www.ted.com/talks/andy_yen_think_your_email_s_privat...)

Can you explain how Mozilla entering into a partnership is the same as vouching? Did they do any particular vetting or analysis, or was this just a marketing partnership?

You can read about what Mozilla did on their blog post about this: https://blog.mozilla.org/futurereleases/2018/10/22/testing-n...

Quoting from the blog post: "We therefore set out to conduct a thorough evaluation of a long list of market-leading VPN services. Our team looked closely at a wide variety of factors, ranging from the design and implementation of each VPN service and its accompanying software, to the security of the vendor’s own network and internal systems. We examined each vendors’ privacy and data retention policies to ensure they logged as little user data as possible. And we considered numerous other factors, including local privacy laws, company track record, transparency, and quality of support."

It was quite intensive, with on site visits to our office in Geneva and discussions with Mozilla technical leadership.

Thanks, that's what I was looking for!

NordVPN just posted this a few minutes ago: https://nordvpn.com/blog/official-response-datacenter-breach...

NordVPN is down, also looks like VikingVPN and TorGuard were affected as well: https://twitter.com/cryptostorm_is/status/118609795032747622...

The thing that scares me here is that these keys were leaked May 2018, and it's becoming public knowledge now.

Someone found certificates for those three VPN providers and posted them to 8chan with a message like "I don't recommend these VPN providers lol"

The good news is that they're only certificates, and they have now expired, but theoretically they could have been used for the past year without anyone noticing.

During the spate of health care information leakage, someone invented a MTBCA, meaning "Meantime to CEO Apology" for the time between the breach and the CEO apology. At that time, it was running on the order of 8 months.

Now that's a metric

haha this is fantastic.

They wrote:

> We […] started creating a process to move all of our servers to RAM, which is to be completed next year.

What does "RAM" mean here?

I guess that all decryption keys are on ram. If the power is disconnected, then it would need a manual intervention to re-decrypt the data

Likely booting the OS from a RAM-disk and mounting as read-only.

I'm curious about who this cloud provider might be.

The local IP for the OpenVPN endpoint ( listed in the gist) belongs to creanova.org since 2017.

What this article is missing is that the hackers had root access and had NordVPNs private key for their HTTPS cert for several months in 2018. This went undetected for months and they're only now publically admitting what happened due to press attention. Their public response seems to be "it's not a big deal guys, mitm is hard".

> The key wasn't set to expire until October 2018, some seven months after the March 2018 breach


And here's a dump of their logs: https://share.dmca.gripe/hZYMaB8oF96FvArZ.txt

Why isn't anybody in journalism publishing this? Really, they're scammers!

Someone is probably going to ask what other HN users recommend as an alternative. Personally, I use Private Internet Access because they're the only provider I've found with a track record of demonstrably not being able to turn your records over to someone asking for them [1].

[1] https://torrentfreak.com/private-internet-access-no-logging-...

Apparently The Wirecutter now recommends TunnelBear and Mullvad because they post regular transparency reports and do third-party audits. https://thewirecutter.com/reviews/best-vpn-service/

Mullvad is a great choice, but I'm mildly surprised about TunnelBear as a choice. They log bandwidth and they incentivize social media spam.

Citation: https://thatoneprivacysite.net/#detailed-vpn-comparison

I've been using Mullvad for a while now and I have nothing but praise for them. Only complaint is they're more expensive than some of their competitors.

What aspects of a VPN provider would you praise? Customer service, consistency of connection speed? Seems almost like a utility where it's hard to differentiate.

I am not OP, but they helped me with a rare openvpn config for which my forays into the openVPN forums led nowhere.

Out of my 110/12mbit connection Mullvad let's me use 108/11 of that, and even has wireguard support.

I would say transparency/security, quality of the client(s) and customer service, in that order. It's one thing to offer a VPN service, but to make sure you have a nice app on both iOS, Android, MacOS, Windows and Linux seems like quite an investment.

If we're offering recommendations, then I'll go ahead and recommend Mullvad. They've got great clients for most common operating systems, good customer support, good performance, lots of servers to choose from, the ability to open ports, etc.

Something I find pretty neat about them from a technical standpoint is their account creation, user authentication, and payment processes. Sign-up literally takes less than a second, so even if you don't plan on using their service, I recommend you try creating an account.

I've been using Private Internet Access (PIA) since 2016 and can also recommend it from a usability point of view. I'm not a security expert so I defer to others on PIA's security.

In 2015ish PIA got hacked via https://old-support.privateinternetaccess.com because of https://classichelp.kayako.com/hc/en-us/articles/36000646089... and never told anyone.

This bug loudly announces itself on every pageload, it speaks of tremendous incompetence that they ever let this go into production.

The site used to set a cookie that looked like this:

  Set-Cookie: SWIFT_client=a%3A1%3A%7Bs%3A15%3A%22templategroupid%22%3Bs%3A1%3A%221%22%3B%7D; expires=Wed, 28-Dec-2016 23:24:13 GMT; path=/; httponly
Obvious PHP object injection vulnerability that should've been caught by any automated auditing tool.

While the helpdesk software PIA used to use years ago did have that potential vulnerability, fortunately, Private Internet Access never exposed the support desk via plain http, and therefore, PIA itself did not have the vulnerability in its helpdesk.

Hahahaha, this bug was perfectly exploitable via TLS wrapped HTTP (so HTTPS, which is still HTTP as far as the PHP application is concerned).

The SWIFT_client cookie gets passed directly into unserialize(), TLS has literally nothing to do with this.

FWIW rasengan is one of the PIA founders, he should know much better.

This response is so utterly silly I must wonder if this is all just an incredible display of incompetence instead of malice.

Sorry, I glanced at the link you pasted and wrote the response as I knew this was a non issue from the past.

So, I spoke with our internal team and was able to find more details:

- We haven't used that machine since that exploit was made public.

- We were never exploited.

- There was no sign of intrusion of any kind.

- The specific machine was a backup helpdesk test server without any real user data.

Thanks again for bringing this up!

>- We haven't used that machine since that exploit was made public.

So what? You were exploited before kayako patched this bug, it was glaringly obvious to anyone who ever looked at the cookies set by your site.

>- We were never exploited.

This simply isn't true, either you're misinformed or lying.

>- The specific machine was a backup helpdesk test server without any real user data.

The specific machine (Which you took down really fast after I pointed it out! :P) I linked probably did not even exist in 2015, I was talking about your prod env.

I don't have a horse in this race, there's no incentive for me to lie about this. I know what you are saying isn't true.

ExpressVPN - gets the job done, a bit expensive but rock solid reliable, breaches great firewall of china.

The claims there have been thoroughly debunked, most recently by Mozilla and the European Commission as part of their due diligence, details here: https://bit.ly/35RDKzB

Their Google cert literally is "Tesonet" - how is this claim debunked - you can check it yourself.

There's actually a point by point write-up about this on Reddit: https://www.reddit.com/r/ProtonVPN/comments/8ww4h2/protonvpn...

There's a historical, almost accidental connection dating back to the infamous November 2015 DDoS against Proton, but zero connection today, and certainly not in the way it has been portrayed by people seeking to attack Proton. Android certs are permanent and can never be changed so that is why there is still one mistakenly issued Android cert out there today.

For a company based in Switzerland to be "accidentally" connected to a company in another country they claim to have no connection to such that their permanent google cert lists the name of the company they are supposedly not connected with - that doesn't seem odd to you?

That fact that this had to be slowly pried out with changing explanations along the way?

When you say the claim that has been debunked - I expect the claim not to be confirmed.

Given all that is going on with VPNs, your caution is warranted, but one should also critically examine the claims that are being made.

Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on our Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. That doesn't mean Proton as a whole is headquartered there, or that the subsidiary somehow controls the parent company in Switzerland, or that there is somehow data mining going on.

Those are the claims that have been clearly debunked. The fact that Proton has a subsidiary in Vilnius, or the fact that we outsourced our HR back in 2016, are not secrets, and is on our Instagram and the Reddit thread linked above. This is the truth, and this is not some wild EU-funded data mining conspiracy as some would have you believe.

Protonvpn is avoided by people who care about their privacy worse than a STD.

It's against almost all certificate standards for the certificate holder - Tesonet to allow an unrelated third party to use a certificate with their name on it. Or for a third party (proton) to use a certificate that has another companies name on it.

If you would like, I can pursue this issue further given what seems to be confirmation here of a certificate violation.

Do you have a non-shortened link?

ProtonVPN comes from the same people who run ProtonMail, a very well known security focused email provider.

You're being downvoted because people believe in propaganda being pushed by competitors, but ProtonVPN / ProtonMail are very good options. Plenty of links and reports by Mozilla in this thread will lend credence to that.

Aka the conspiracy theory that was ultimately found to be pushed by PIA, one of their direct competitors.

I've got enough HN internet points that a few downvotes will be fine. Thanks!

> Aka the conspiracy theory that was ultimately found to be pushed by PIA, one of their direct competitors.


I've heard this several times but nobody has ever been able to provide a source.

It came from a protonvpn founder, an obviously biased source: https://www.reddit.com/r/ProtonVPN/comments/8ww4h2/protonvpn...

but was conveniently never denied by PIA that I could ever find. If it was a lie, it would be easy for them (PIA) to prove under libel laws in the discovery phase of a trial. I'd argue if it was a lie from ProtonVPN, it would have been in PIA's best interests to clear their name. After all, PIA and ProtonVPN are a few of the only providers who've proven in courts they don't have logs of users. We know they're legit because they said so in court under penalty of perjury. Also, the European Commission has investigated these exact claims, and would have privileged access to a lot of the business documents, and found the claims without merit.

Me? Just a happy protonvpn user who finds the oft repeated shilling for PIA dull. If you really want to hate protonvpn, use PIA, or use someone else. Better, don't trust any of them! Setup algo on a digital ocean droplet of your own: https://github.com/trailofbits/algo

However, this is meant for running over an untrusted network, not for maintaining internet anonymity. Use Tor for that.

ProtonVPN has a large history of being connected to TesoNet, a company providing among other things data mining(!). An extra cherry on top of that is the CEO of TesoNet also being the CEO of CloudVPN, which more or less controls NordVPN.

Now that doesn't mean ProtonVPN is automatically compromised but I feel with stuff like no-log VPNs one should always err on the side of caution.

This has been thoroughly debunked, most recently by Mozilla and the European Commission as part of their due diligence.

ProtonVPN is 100% owned by the company behind ProtonMail, which in turn is funded by the European Union, so this has been verified by the European Commission. Details here: https://bit.ly/35RDKzB

Over the course of the disclosure of the connection between NordVPN, Tesonet, and possibly ProtonVPN, Proton's story kept changing. They said contradicting things multiple times. They locked the Reddit thread. Why did Proton keep changing their story if they had nothing to hide? I will keep reminding this every time the issue gets raised. There is a compilation [0] of changing Proton's responses and them successively admitting more and more things not in their favor. The compilation starts at the part called "Online accusations fly".

[0] https://restoreprivacy.com/lawsuit-names-nordvpn-tesonet/

Both Mozilla and the European Commission have looked into the accusations being made on anonymous websites, and determined that they are false. The EU in particular, has access to records which allow independent verification.

There is also an abundance of public record which demonstrates this is false. The bad faith of those spreading this information is also apparent from the hundreds of fake Twitter accounts used to spread the rumors.

If you are acting in good faith, then we ask that you also take a moment to verify your facts and discover the truth, much of which can be found here: https://protonvpn.com/blog/is-protonvpn-trustworthy/

Can you confirm the certificate for Proton never had Tesonet in it? The number of overlapping coincidences is just unreal.

There's actually a point by point write-up about this on Reddit: https://www.reddit.com/r/ProtonVPN/comments/8ww4h2/protonvpn...

There's a historical, almost accidental connection dating back to the infamous November 2015 DDoS against Proton, but zero connection today, and certainly not in the way it has been portrayed by people seeking to attack Proton.

I mean this[1] is pretty convincing and not directly from the accused company's blog. The only thing it gets wrong is framing ProtonVPN Lithuania as the main ProtonVPN company instead of as a subsidiary.

Regardless of that, there is so much mud being slung I recommend anyone to just search for 'protonvpn nordvpn tesonet', read a few articles on the topic and form your own opinion. Like I said, you can decide if you want to err on the side of caution or if it's a risk you're willing to take.

In case anyone wants VPN recommendations, I have good experiences with TorGuard and Private Internet Access and can also recommend Mullvad. Other people (that I trust) say iVPN and Tunnelbear are also solid.

[1] https://vpnscam.com/nordvpn-protonvpn-proton-mail-owned-by-t...

There's a couple ways to look at this.

On one hand, there's anonymous websites, competing VPN companies, and hundreds of Twitter bots pushing a story that is demonstratively false (just check public records).

Then, on the other hand, you have Mozilla and the EU (which has access to all European corporate records) vouching for Proton, which also operates in a highly transparent way, examples here: https://protonvpn.com/blog/is-protonvpn-trustworthy/

Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. And that doesn't mean Proton on a whole is based in Vilnius.

> On one hand, there's anonymous websites, competing VPN companies, and hundreds of Twitter bots pushing a story that is demonstratively false (just check public records).

I agree, the VPN industry is rife with shady business practices. But the story being pushed isn't 'demonstratively false'.

* TesoNet offers data mining services

* You did contract TesoNet employees

* Due to an error and unyielding policies by Google TesoNet holds your Android app signing keys in name

* There is a lot of intermingling between TesoNet and NordVPN and to a lesser extent TesoNet and ProtonVPN.

Like I already stated, it's very unlikely you are compromised. But unlike, say, a billing company that handles my energy or water provider (where I care much less if they have tenuous links to data mining) my standard is extremely high for a VPN. Internet traffic is supremely personal and for me to trust a company handling that there cannot even be the slightest sheen of misconduct.

For me to trust you you would have to completely cut out your Lithuanian subsidiary and any employees, board members, etc. that were or are related to TesoNet, as well as any reliance on their infrastructure. Obviously businesses don't operate with such 'scorched earth' policies and I don't expect you to gut your company based on a HN comment, but it is what it would take for me and many other privacy-conscious individuals to regain our trust.

Definitely appreciate your concern here, but there's still a lot which is being confused.

Proton does not today, and has never, used contracted (outsourced) employees. As is common with startups, in the past we did not always do all our HR in house (it's all in house today), but employees were always working on Proton and for Proton.

There are no board members, directors, shareholders, or employees, related to Tesonet beyond the fact that a couple employees might have been employed there previously. This in itself is not strange, we also have some employees who previously worked at Google, the ultimate data mining company, but clearly decided they preferred to work for the other side. People can and do change jobs.

Proton has also always run our own infrastructure, and for ProtonVPN, this is publicly verifiable.

So, we don't have to "gut our company" to remove any "intermingling" because there was little to none to begin with, and certainly nothing today.

Indeed trust is super important, but it seems odd to trust anonymous internet accusers or those with a clearly vested interest in harming Proton, as opposed to reputable third parties like the EU or Mozilla who don't have a vested interest here and are independent.

Proton is still to this day, the only VPN company that has an address clearly published on our website, where you can show up, and find company management and board members, and that means something.

Slightly off-topic but I am delighted by the generally non-abrasive way this thread is going. Dialogue is good!

I realized another way that would work for you guys (but is out of your hands) is fighting a court case about this. You'd be legally compelled to tell the truth and very screwed if you deny but then it comes out there is logging or mining going on. It's not ironclad but it is how most VPNs end up being considered 'solid'.

We have indeed retained lawyers to look into our options to fight the online defamation, but its hard to take anonymous accusers to court. However, as we have discussed here (https://protonvpn.com/blog/is-protonvpn-trustworthy/) there is already a lot of ironclad legal evidence.

First, were we to lie in our privacy policy, we would be subject to GDPR fines of up to 20 million Euros, since we have both European customers, and a presence in the EU.

Second, there has already been a court case. We were ordered by a Swiss court to hand over logs, and we stated truthfully (under penalty of perjury) that we did not have the logs requested. This case was previously disclosed here: https://protonvpn.com/blog/transparency-report/

'January 2019 – A data request from a foreign country was approved by the Swiss court system. However, as we do not have any customer IP information, we could not provide the requested information and this was explained to the requesting party.'

I'm not terribly well-versed in the international (or Swiss) legal system but are portions of that request public record, or would it be possible to put portions of it online, verbatim?

It would really strengthen the case to your customers because whilst claiming you had a request when you didn't isn't illegal, falsifying court documents definitely is.

No public indictment was issued because in this case the accused could not be charged since they couldn't be identified. Generally there are only documents if police decide to move forward with a prosecution, which is unlikely since we do not have logs that can identify users.

Anyone can set up an anonymous website and make spurious accusations and/or take money to post glowing reviews. The VPN segment is full of shady tactics like this. Never trust any VPN review site.

Trust serious organizations such as Mozilla and the EFF.

Mozilla trusts ProtonVPN enough to officially partner with them. That means a lot more than some random anonymous reviews.

> ProtonMail, which in turn is funded by the European Union

Wait... that doesn’t sound ideal either.

There are pros and cons to this, we think it's positive (aligns the EU with privacy), but we provided all the details in the below link so people can draw their own conclusions: https://protonmail.com/blog/eu-funding/

Don't use Tunnelbear, they're known compromised. Honestly it's hard to beat Mullvad right now but that does make them a hot target, so keep your eyes peeled and know when to jump ship.

I am surprised why isn’t anyone suggesting Cloudflare’s Warp VPN? Genuinely curious what is the difference. I guess Clodflare one is only for mobile?

Cloudflare's Warp is not an anonymising VPN as far as I know. It is just a way to speed up Internet speeds, especially in poorly connected areas. They make no effort to hide the origin IP. So it is not in the same class as other VPN providers.

This is interesting to read that Cloudfare is suggested here... shows that the term VPN is still thought of as private. (I know it is Virtual Private Network - but the termination is almost never private).

It's anonymous wrt your ISP. Which is what 95% of the complaints here refer to.

> I guess Clodflare one is only for mobile?

Yes, unfortunately it's currently not possible to use Warp VPN on PC. Otherwise, quite good service.

https://airvpn.org/ is also worth mentioning.

- Doesn't work for Georestriction - Shows your IP address (some pages) - You cannot choose your data center - Only for mobile

I've had fantastic experience with airvpn. They're cheap, fast, reliable, and support all the configuration types you could want. I'm not affiliated with them but I'm surprised nobody here has mentioned them yet. By far the best VPN provider IMO.

And it's run by internet activists, support them, support the cause!


Another vote for AirVPN. Good track record, happy customer for 6+ years.

Yeah, they have been around for a very long time too. Used to be the more expensive alternative though, glad to see their prices have fallen a bit.

A good resource for comparing different VPN providers is the VPN Comparison by That One Privacy Guy [1]. I personally use Mullvad [2].

[1] https://thatoneprivacysite.net

[2] https://mullvad.net/en/

IPv6 VPN is available for free from https://ungleich.ch/ipv6/vpn/ if you buy a VM from them.

Haven't tested it, just rembered datacenterlight from a HN thread about buying a mainframe.

Any provider that offers that many IP addresses? I found NordVPN to be the only reliable service if you need to run requests from many IPs from different countries (web scraping).


The thing is, almost all of these providers share infrastructure and IP blocks. Lookup MicFo and the associated lawsuits (not even including their lawsuit with arin). They provided the exact same IP blocked systems to dozens of the top VPN providers.

The reality is, if someone else owns the infrastructure you're just pushing the risk to a different location.

This is only an issue if the keys of the VPN are compromised (like it appears to be for NordVPN).

PIA is also compromised. They installed the known criminal Mark Karpelès as CTO.

Freedome VPN has been similarly been shown to do the same.

Perfect Privacy is pretty good too.

EDIT: I said I used IPVanish mostly because the EFF endorsed them, but someone pointed out below they got caught logging. I suppose that would explain why they're not endorsed on the EFF VPN page anymore. So, I guess time to find a new VPN. :(

IPVanish was caught logging while claiming they didn’t log [1].

[1] https://torrentfreak.com/ipvanish-no-logging-vpn-led-homelan...

Ugh. Good to know, thanks.

Used IPVanish for a few years, it was great the first, sucked the 2nd. I switched. These days I use VyprVPN. I like their Chameleon encryption that hides the VPN (although some data still leaks and some sites still detect it) and the killswitch option (prevents connections out if VPN not active)

I have a slightly dissenting answer to these questions, in the form of an interactive Q&A website:


... this site is awful. It doesn't address the actual reason why people use VPN's. They don't want all their activities to be recorded/tracked by their ISP's (which depending on jurisdiction log everything for at least 6 months if not more) or other actors. And if somebody wants to deanonymize your traffic, they have to go to extra effort, whether it's by exploiting or establishing a relationship with your VPN host or whatever else. Or there are other use cases, like wanting to torrent in a country that is very liberal with serving fines (Germany).

And frankly, his alternatives are just absurd. Tor? Really? Has he ever tried to use Tor for usual daily browsing? Does he expect people to try to use Facebook, Instagram, Youtube over Tor? Really?

How is this not the top comment?

Nobody should be using a VPN provider, full-stop. It is structurally impossible for anyone to verify their claims, they have more incentive to lie than your ISP does, and they're cheap and easy to set up, so the industry is a cesspool.

You should assume that all of them are behaving badly.

You can always setup Algo on your own personal VPS running on Azure, GCE, DigitalOcean, Vultr, etc.


Also, hasn't CloudFlare been audited to verify their no log claims? I know while Mullvad hasn't been audited to verify their no log claims, they have at least been audited to verify the security of their app.


"1. We don't write user-identifiable log data to disk;

2. We will never sell your browsing data or use it in any way to target you with advertising data;

3. Don’t need to provide any personal information — not your name, phone number, or email address — in order to use the App with Warp; and

4. We will regularly hire outside auditors to ensure we're living up to these promises."


> they have more incentive to lie than your ISP does

A lot of ISPs openly collect user data, so I don't know how much that factor matters. And while I can go get a VPN, I can't just get another ISP.

Pretty much a textbook "Lemon Market"


Shouldn't matter to your average torrent user or someone who wants to watch US YouTube. And I guess that's most people who actually use it.

Plenty of people using VPN for: pirating, get an IP from a foreign country to bypass some content limitation etc., so they still need one, secure or not.

>NordVPN said it found out about the breach a “few months ago,” but the spokesperson said the breach was not disclosed until today because the company wanted to be “100% sure that each component within our infrastructure is secure.”

So instead of allowing their customers to do their own damage limitation, they left their customers in the dark and continued to expose them to a breach they weren't sure they had fully contained.

I wonder when that sort of thing will become a criminal offence.

Sorry for posting under top comment, but I think it is very important.

Official response hides fact OpenVPN CA keys also leaked, so attacker could impersonate any other NordVPN server: https://gist.githubusercontent.com/Snawoot/85f77356e229d77aa...

RADIUS secret key also leaked, so propably it is possible to break into EAP session which infers session secret key for StrongSwan.

Also allowing historical sessions to be decrypted.

Also looks like NordVPN has been misleading customers about the number of servers they have (or didn't make clear they were VM/containers).

I don't think they ever wrote anywhere that they have 6,000+ physical servers. Calling a VM (like an EC2 instance) a server is not unusual. For the customers it was important that the resources, bandwidth and different IPs were available. For that it doesn't matter if it's a physical server.

Who has logs of full historical sessions, and the tools to decrypt them? NSA?

Yeah, as far as I understand that's pretty much the purpose of the Utah datacenter.

> RADIUS secret key also leaked, so propably it is possible to break into EAP session which infers session secret key for StrongSwan.

Could you elaborate on this? I am familiar with PKI so the first part makes sense, but I am not familiar with the intricacies of VPNs so I am not sure what this means.

When StrongSwan EAP-RADIUS plugin is in use, authentication delegated to RADIUS server. Actual EAP handshake occurs between VPN client and RADIUS server. [1]

Some EAP authentication methods (namely, EAP-MSCHAPv2 and EAP-TLS) export Master Session Key, which is exported to StrongSwan via MS-MPPE-Send-Key/MS-MPPE-Recv-Key EAP attributes. [2] So, MSK derived on RADIUS side and sent to StrongSwan. Eavesdropper with knowledge of RADIUS secret key capable to intercept and decrypt such EAP payload.

[1] - https://wiki.strongswan.org/projects/strongswan/wiki/EAPRadi...

[2] - https://tools.ietf.org/html/rfc5216#section-2.3

Read up on "Perfect forward secrecy": https://en.wikipedia.org/wiki/Forward_secrecy

Assuming their IPsec was enabled with it (and OpenVPN should be enabled by default), them leaking their keys does not matter. The sessions can not be decrypted even if the master key is leaked.

TLS also has perfect forward secrecy by default.

Impersonation is an issue, but the article stated the CA keys have already been rotated and are out of date.

EDIT: I meant to reply to the post below me, but this is fine. Sorry about that!

> Impersonation is an issue, but the article stated the CA keys have already been rotated and are out of date.

They were not rotated back then in 2018, so we can only guess if MITM had place. Their line of defence appealing to keys which are NOW outdated is just ridiculous.

> Read up on "Perfect forward secrecy": https://en.wikipedia.org/wiki/Forward_secrecy

> The sessions can not be decrypted even if the master key is leaked.

It's not true. PFS provides cryptographic isolation between long-term keys and session key used to encrypt data. Obviously, if MSK compromised it is irrevelant, how it was inferred: with PFS or not.

> They were not rotated back then in 2018, so we can only guess if MITM had place. Their line of defence appealing to keys which are NOW outdated is just ridiculous.

MITM could have taken place anyway because the attacker was on the machines. They did not need the key.

> It's not true. PFS provides cryptographic isolation between long-term keys and session key used to encrypt data. Obviously, if MSK compromised it is irrevelant, how it was inferred: with PFS or not.

PFS implementations have the session key rotated automatically in software and dependent on the implementation multiple session keys are in use at any given time dependent on flow. The PFS session key would also be different for every VPN server in the NordVPN environment. The only possible way to compromise a session on another VPN node (that was not compromised itself) would have been to intercept it at the time of the session being created and MITM by injecting your own PFS session key.

That is why it is called "Forward secrecy": the session can not be decrypted in the future, only in the present.

Unless your assumption is that this is a state actor with the ability to MITM connections in the first place, or a rogue ISP BGP hijacking that would have been obviously seen on something like BGPStream [https://bgpstream.com/], it is safe to say that no other VPN node's traffic was compromised. Only traffic on this single host in this single location.

More reading: https://www.speaknetworks.com/what-is-ipsec-vpn-pfs-perfect-... (Step 5)

"Instead of making use of the DH Keys Calculated during Phase-1, PFS forces DH-Key calculation during Phase-2 Setup as well as Phase-2 periodic Rekey. The PFS ensures that the same key will not be generated and used again."

OpenVPN (using TLS) also uses PFS by default. There is a reason it is called "Perfect."

EDIT: PFS is commonly also implemented by Diffie-Hellmen (DHE) key exchange: https://en.wikipedia.org/wiki/Diffie–Hellman_key_exchange

EDIT 2: I am not defending them as well. I just believe extrapolating the technical details is fear mongering at best. Believe it is best to focus on the facts as it makes for a stronger argument.

I've lost track of discussion and maybe some misunderstanding takes place, but I'll attempt to synchronize.

There are two distinct severe security issues. First one is leaked CA key, which allows to certify any key as a valid key for NordVPN server certificate key. I think it is not necessary to argue about this: traffic decryption to any Nord OpenVPN server became simple as network MITM. Not likely a state-level BGP hijack, but local attack targeting channel of small group of users. Anyway, we do not have cryptographical guarantees since this point.

Second issue is leaked RADIUS key. Why it is a problem for encryption? Because EAP authentication and key derivation runs between VPN client and RADIUS, and VPN server receives derived keys as attributes of EAP message: https://wiki.strongswan.org/projects/strongswan/wiki/EAPRadi...

> The eap-radius plugin does not implement an EAP method directly, but it redirects the EAP conversation with a client to a RADIUS backend server. On the gateway, the EAP packets get extracted from the IKE messages and encapsulated into the RADIUS protocol, and vice versa. The gateway itself does not need special support for a specific EAP method, as it handles the EAP conversation between the client and the RADIUS backend more or less transparently.

> For EAP methods providing an MSK, the RADIUS server must include the key within the MPPE-Send/Receive Keys; Unfortunately, FreeRADIUS before 2.1.10 did not include these attributes when used with EAP-MSCHAPv2.

So, despite session encryption key is not bound directly to long-term secrets which server possesses, they can be extracted from communication between StrongSwan and RADIUS server.

PFS has nothing to do with all of it. In first case it is possible to issue VALID certificate for eavesdrop VPN server and redirect users traffic to it. In second case MSK probably can be extracted communication between VPN server and RADIUS server (I can't say if it will require MITM of RADIUS session or it is possible to decrypt EAP payload with passive sniffing and posession of RADIUS secret key).

What is the source of the gist you linked?

It is copy of snippet linked from 8ch.net: https://web.archive.org/web/20180504001844/https://8ch.net/b...

Hasn't that site been down for weeks now?

It's a snapshot of thread dated by May 3, 2018. Info leaked long time ago, but became sensation just recently.

i find it surprising that none of the threads in this topic mention the very serious threat this breach might have for users in countries like china. the fact that nordvpn neglected to tell its users for months after the breach quite possibly endangered people's lives. unforgivable.

Dystopian and true.

In Australia under the Consumer Data Right it would be an offence to wait before disclosing this. Unfortunately it doesn't cover all industries yet.

I think that is pretty criminal already.


1- Nord falsely blames its server provider.

2- Nord hides it from their users.

3- Nord claims all will be well with an “audit” (again, since they were already “audited”)

This is either criminal negligence, “security theater”, or both.

> Nord falsely blames its server provider.

I don't see anything in the article about those claims being false. Where did you get that?

From the article: "“One of the data centers in Finland we are renting our servers from was accessed with no authorization,” said NordVPN spokesperson Laura Tyrell."

I believe that would be the section they're referencing.

How does this quote demonstrate that the service provider was not at fault?

“We failed by contracting an unreliable server provider...”

They are casting blame on the provider. Providing remote access tools is not a fault. Failure by NordVPN to disable said access is the issue, yet they passed the blame on.

Okay that makes sense. I read differently, like they had fabricated the story about the service provider's management console.

Yeah, that sounds like flagrant incompetence

Calling it incompetence lets them off the hook I think. This was a deliberate choice to keep customers in the dark, which is worse IMO.

Yeah, the word for this is 'negligence'. Sounds like someone should get sued at the very least.

Within 72 hours According to GDPR I thought? “ The GDPR introduces a duty on all organisations to report certain types of personal data breach to the relevant supervisory authority. You must do this within 72 hours of becoming aware of the breach, where feasible.” https://ico.org.uk/for-organisations/guide-to-data-protectio...

Maybe NordVPN would argue personal data wasn't breached?

It's a bad look in any case.

Theis Keys were stolen. All data is exposed now.

Only if someone used it successfully for MITM attacks. We don't know that, they can still argue that in fact no user data got breached.

I don't think they care about GDPR much. They were set up in a way to avoid legal scrutiny (not a bad idea for a VPN provider).

The tinfoil hat would argue maybe this was a leak that happened, but it was shared by design. It’s an HK company with questionable relationships and owners.

> It’s an HK company with questionable relationships and owners.

That seems to be ExpressVPN[1], the main competitor of NordVPN.

[1] https://vpnscam.com/expressvpn-really-based-in-hong-kong/

> I wonder when that sort of thing will become a criminal offence.

If they have EU customers then article 33 of GDPR should see to that.

"In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons. Where the notification to the supervisory authority is not made within 72 hours, it shall be accompanied by reasons for the delay."

Unless the authorities in EU accept the explanation they are in trouble, but I think you shall report even if you think there has been a breach.

They could argue that they don't have proof that the exposed private key led to a personal data breach.

Yeah, I think that's their only option actually.

They would have to be sure that no private data leaked though, right?

Exposing a private key is basically a breach.

"breach" usually has a legal definition, which varies by jurisdiction

> to the supervisory authority

Does this always/necessarily lead to customers/the public being informed? But yeah, better than nothing.

Maybe they only disclosed it publicly but notified customers out of the public eye ?

I was a NordVPN customer 4 months ago and got no notification. I’m hearing about this now via this post on HN. Dropped them around July.

Aight then i'm sorry for their lack of accountability.


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact