Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in the blog post. As for mitigation, we are already testing a patch of the unintended behavior on a subset of our infrastructure. If any of you try to reproduce the blog post's findings you may get confusing results throughout the day.
We will also re-evaluate whether the intended behaviors are acceptable or not. Some of this is a trade-off between multiple aspects of privacy, and multiple aspects of user experience.
Please note that this is my current understanding, which may change. I was only made aware of this an hour ago, and most of that time was spent talking with Ops, considering what to do immediately, and writing this post.
Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away.
You really do provide a reassuring, good service -- thank you.
It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours.
`mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post.
I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.
Carl here (Obscura CEO, one of Mullvad's partners)
This was an interesting finding, though as kfreds mentioned it would have been better to notify the vendor before publishing.
The main finding (IP-position-in-pool correlation between servers) seems to include genuinely unintended behaviour. Given our great experience with the Mullvad team, I'm sure this will be addressed soon.
In general, if you want different "identities", you should make sure to rotate or use different WireGuard keys.
One small thing from the article I'll comment on:
> Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key, which rotates every 1 to 30 days (unless you use a third-party client, in which case it never rotates).
Context: WireGuard is by design[1] a "Connection-less Protocol", there's no concept of a connection, there's only a "re-keying handshake" (key here refers to the ephemeral Diffie-Hellman key, not the WireGuard key) every 2-3 minutes ONLY IF there's traffic flowing.
The above statement is not too surprising if you consider the counterfactual: What would happen if, even with the same WireGuard key, the exit IP were randomized each time you "connect" to the server (say each time there is a "re-keying handshake" or at more frequent cadence (e.g. every 15 minutes) than the WireGuard key rotation).
In this scenario, ~every 15 minutes:
- At the Transport layer, all your in-tunnel connections that are on non-roaming protocols (basically everything except QUIC) would be disrupted, and the connections would have to be re-established.
- At the Application layer, many application-level sessions that treat "same cookie, new IP" as suspicious would trigger logouts, CAPTCHAs, or risk scoring.
Both are terrible UX, and what's worse would also make users much more uniquely fingerprintable ("this person keeps reconnecting from a different IP, they must be using Mullvad").
I just want to say I absolutely love Mullvad! You guys did a fantastic job at designing a genuinely good and trustworthy (as much as possible) VPN vendor. You communicating here is just another data point towards this.
I almost want the people doing the mandatory VPN product placement ("This video is sponsored by NordVPN!") to do Mullvad for once. My jaw would hit the floor from unfamiliarity
If Mullvad was suddenly in that ad scene, I would get worried.
This is not anything specific against Nord, I don't know anything about them. However, at this point, I take YouTube/influencer ads as a very negative signal towards the product being pushed. I am not sure if that's fair, but that's just my gut feeling given the entire YouTube ad scene.
I think it's the cost per viewer, where "scams" are more profitable than a honest business, and that makes my gut tingle. Again, to be fair, I may be being a jerk here with my judgment.
I deeply appreciate Mullvad's thorough approach to privacy and ethics. In this day and age, you all are an absolute breath of fresh air. Thanks for that.
Thank you! As a staunch supporter of Mullvad, you are the only VPN provider I recommend for cybersecurity, especially in today's landscape where VPNs are facing increasing regulatory restrictions. Thank you so much!
It already has official packaging for Tumbleweed, see https://github.com/mullvad/mullvadvpn-app/issues/2242 for the upstream issue. Leap can use the normal Linux application, you will just have to provide the dependencies yourself.
>The Mullvad VPN app is available in our repository for the following supported Linux distributions:
Ubuntu (24.04+)
Debian (12+)
Fedora (42+)
The only thing I see on the issue you linked is a way to jerry-rig the fedora package. When I tried that I kept getting untrusted key warnings. You can skip them of course, but it kind of undermines any type of trust here
It's a skill issue that they decide to not list open suse as a supported distro on their own help page ?
It's a skill issue that the thread has a bunch of different solutions and none of them are definitive and endorsed by the company I'm paying $5 a month too ?
> Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings
How to report a bug or vulnerability
... we (currently) have no bug bounty program ... send an email to support@mullvadvpn.net
I dislike it here because I like Mullvad, but yes, I think it’s fair to go straight to public disclosure.
Someone with likely substantial qualifications put in time to find this. The company is in it for profit (at least partially). What’s fair for the company is fair for the individual. The company can either offer to pay for bugs under the terms they want, hire more security folks to find the bugs themselves, or just accept that researches get to do whatever they want with their findings.
I’d tell Mullvad, but there are companies I don’t respect enough to feel compelled to give them a heads up. Perhaps the author feels that way about Mullvad, it’s entirely within their right to use this to publicly shame Mullvad.
This ought not be considered anything close to common courtesy. This is work. Mullvad is engaged in the business of making money. They should show how serious they are with your money.
Since when do you have professionals giving you examinations out of common courtesy? Out of courtesy can I get a free cancer screening?
If I doctor performed a cancer screening on me, for free and without me asking, then yes — as a matter of courtesy I would still expect that doctor to tell me if he found cancer, rather than reading about it on his blog later.
>Since when do you have professionals giving you examinations out of common courtesy?
Maybe when they decide on their own volition, without any external pressure, to go and poke around your system?
"Hey, I'm a mechanic, I was looking at your car parked out there and noticed something incredibly dangerous that needs immediate fixing. I'll tell you what it is for $1,000."
Even better, the mechanic writes a blog post about the dangers of non-functioning brakes, but doesn't tell the car owner, because they didn't have a sign advertising their "car issue bounty program".
Seems to be a systemic issue with computer guys feeling entitled to financial compensation for strange reasons. See also, people licensing their software as "open source" and then being mad when people make money off it.
Even better, the mechanic writes a blog post about how the locks on that guy's car don't work, and how anyone could just steal it, but doesn't tell the guy because, after all, the guy wasn't paying him to.
Both of y'all confusing individual with corporate.
The mechanic writes a blog post about how the locks on [a car model] don't work, and how anyone could just steal [cars], but doesn't tell the [car company] because, after all, the [company] wasn't paying him to.
Especially, when the car company spends on 'certifications' (security audits, in this case) and specifically markets it as a differentiator. That said, uncoordinated public disclosures in cybersecurity are bad form, given the well-established existing norms & culture; but at least, let's get analogies right.
Obviously there are a hundred variables that differ between the analogy and the actual situation. You changed one that felt important to you (individual/corporation) but there are still 99 that differ. That's what makes it an analogy instead of just being a retelling of the actual situation.
But yes, if you found a general fault in the locks of a certain car model and publicized it without first informing the company and giving them a fair chance to inform the affected customers, people would probably be annoyed with you. Individuals even, not just companies.
"You chose that car that advertises good locks. Guess what, the locks are actually bad and now I'm gonna publish exactly how, to teach the manufacturer a lesson about paying me money".
When their 'common decency' is directly benefiting a money making corporation with shareholders and directors then yes they should definitely get some money out of it.
If you create a 3rd party app to some closed source insecure back end, thats on you for trusting them or not doing your due diligence.
Time and time again private companies have rug pulled things like api access for 3rd party apps (such as twitter/X). Building 3rd party clients for private systems should already be approached with heavy scepticism and always be prepared for the worst.
Most of HN readers/writers are American, of course they won't do anything unless they personally profit off it, the entire culture is built around this mindset. Meanwhile, Mullvad is Swedish, and we tend to assume we all want to help build a better world together. Mix the two, and you get this conversation :)
So what, suddenly Swedes can't live outside of Sweden? Kind of interesting to make complaints about generalizations and in the same comment falling for the same trap yourself.
You identified as Swedish, and then shifted the goal post to the country where you (apparently) currently reside.
What's your point again? Something about all Americans whether they live in the US or not? Are you trying to be incoherent? Daft? Representative of all Swedes by origin?
> Most of HN readers/writers are American, of course they won't do anything unless they personally profit off it, the entire culture is built around this mindset
American culture is highly varied. For some this is true, for others this is wrong and highly insulting.
It's OK for the country to have a pervasive culture yet not every resident or citizen of the country to be a part of that culture, or even actively work against it. If you're not one of them matching that description, it shouldn't be insulting, as it's not about you in the first place.
Maybe not everything is aimed towards you, especially if you don't feel like the description actually matches you :)
That is a lot of words for "my negative steroetypes about you and your country are fine, actually. Don't take it personally, bro. Maybe you're one of the few good ones!"
Every time someone makes a cultural comment here, the reply is always "America is a big country". America can be a big country and still have common cultural elements. It's not inaccurate to say that citizens of a large country mostly share some common characteristics. Those characteristics are what makes them one country.
Discovering a bug that could put people's lives and/or freedom at risk if they don't do something about it makes it okay to go public immediately. That said, by all means notify the maintainer/vendor as well.
It should always be assumed that someone else (if not several someone elses) have already discovered the same flaw and are currently taking advantage of it while users remain totally unaware of their actual risk. By going public immediately, you give as many of those users as possible a chance to protect themselves.
Waiting to disclose something harmful when the users in danger could otherwise take steps to make themselves safe would be like not warning people entering a building not to go in because of a gas leak until after you've contacted the building owner and the fire department has shown up.
> Expecting people to hold off on disclosure of something harmful
That's not what they said though. They said "please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away" (emphasis mine)
I do think hitting "send" on the email to the responsible party immediately before publishing (or at least notifying them as quickly as you can afterwards) is a smart thing to do. I mean, why wouldn't you? My concern was more about the "Not having a bug bounty or dedicated email address does not make it OK to go public immediately" comment. It can sometimes be difficult to track down the right person to notify and so when the risks to people are high enough whichever one you can accomplish the soonest is probably where I'd start.
Depending on the severity of the issue. Emailing support with a draft of the blog post and waiting even a couple of hours for a response so they can fix it first would have been more responsible than dropping the blog post to the whole wide world and catching Mullvad with their pants down.
Why wait for a couple of hours for a response while people who could protect themselves are getting harmed? It's especially true when you don't know if the maintainer/vendor will get back to you at all, or if they even check their mailboxes regularly.
The priority should be on protecting users, and not helping the company responsible for the vulnerability save face, or give them extra time to spin up their PR team, or get a head start on a patch.
When the risk to users is low, or when there's really nothing users can do to protect themselves anyway I'd agree with you. In a case like this where the risk to users can be extremely high, and the moment they are made aware of the problem there are steps the user can take to eliminate that risk, the safety of those users should outweigh inconvenience to the people responsible for the vulnerability
The problem is how do you notify users? What are the chances that a Mullvad user is going to happen across this blog post? Of the entire world of Mullvad users, somewhere between 0 and 100% of their users is going to read it and be in a place to do anything about it. If I were to make up a number though, I'd guess it's somewhere between 1 and 10% of Mullvad users. On the other hand, by telling Mullvad first, so Mullvad can fix their system first, closer to 100% of Mullvad users get the fix before attackers figure out the issue.
Mullvad fucked up. They should been as inconvenienced as thru possibly could be too fix the problem promptly! The issue is irresponsible disclosure hurts more users than it helps.
> What are the chances that a Mullvad user is going to happen across this blog post?
It's not as if the odds of new would-be exploiters seeing it are any better. It helps that the people who are at the most risk tend to have their ear to the ground already because they know what's at stake.
When the risks are this high you have to assume that it's already being actively exploited. That means that already there are more attackers who know about the vulnerability than there are users who know about the mitigation.
All you can do at that point is let as many users as possible know how to protect themselves while Mullvad figures out how to fix the issue on their end, writes and puts out the update, and the remaining users get around to updating their systems. You can't save everyone, but hopefully you at least gave some people the chance to save themselves.
> Discovering a bug that could put people's lives and/or freedom at risk if they don't do something about it makes it okay to go public immediately
The flipside of course is ... does your disclosure increase the risk?
> aiting to disclose something harmful when the users in danger could otherwise take steps to make themselves safe would be like not warning people entering a building not to go in because of a gas leak until after you've contacted the building owner and the fire department has shown up
I don't think it's like this at all. The risk of a gas leak is not increased by telling people about it and can't be prevented after its occurred. To stretch your analogy, I'd say its more like you've found the gas leak and instead of turning off the gas supply are instead running around outside the building shouting about how there's a gas leak.
> The flipside of course is ... does your disclosure increase the risk?
When you've got that much on the line you have to assume that the risk is already present for all users. It's true that there's always a chance that some users won't find your disclosure in time and additional would-be attackers who weren't aware of it already will start taking advantage of the flaw, but the alternative is that no users are safe.
> The risk of a gas leak is not increased by telling people about it and can't be prevented after its occurred.
It's true that warning people not to enter wouldn't make the gas more dangerous, but it limits the death count of the impending explosion. It keeps at least some people from entering the building and walking into a death trap.
There's no way to shut off the gas supply when you can't control what's already running on user's devices and more users are downloading and installing the buggy code all the time. It's really not a perfect analogy. The point is that immediate action will save some people, while waiting around means that nobody has a chance of being saved.
I don't feel like its hard to come up with examples where (I would say) its ethically wrong to disclose immediately. If you spotted a company's mistake that might endanger their user's lives or safety, would you put those users at risk simply because there was no obvious financial reward?
If so, I guess we just have different opinions on the ethics involved here.
You seem to have a very bright line between the acceptable behavior for “no money involved” and “money involved”.
For me, it’s more subtle than that.
Everybody (“almost all software”) has exploitable bugs. Are you a fool for not finding the ones in yours? Maybe. Sometimes.
There is a huge difference between Project Zero finding a trivial vulnerability almost identical to one reported months earlier (close to negligence) and Mullvad having the CEO personally posting a response here in a very calm tone.
> Are you a fool for not finding the ones in yours?
If I have a company which sells a paid product, and my paid engineers do not find bugs then I absolutely do not expect the public to willfully and freely make my product better for me. This is why I would have a bug bounty program as an incentive for the public to help me makle my product better and more secure, like any other company serious about finding security bugs.
If I didnt have a bug bounty program and found out that some black hats were selling backdoors to my system online, I would consider that fully my fault for not incentivizing those hackers against doing so.
I'm not sure what you mean by "Oof". We don't have a dedicated security team because security and privacy are integral to all aspects of our service. It doesn't make sense to centralise it.
As for our support team they are responsive and experienced. Several of them have worked with us for many years and do offensive security research in their free time.
Unlike many organisations we don't see customer support as a cost center, just like we don't see security as a cost center. Our support team represent our customers, and as a consequence contribute a lot to how we prioritise our roadmap.
Clearly the person who wrote "Oof" has never emailed Mullvad support.
Whenever I have emailed Mullvad support I have received a prompt reply from a human being who clearly actually cares about taking ownership of the question and seeing it through to resolution.
I have also witnessed first-hand the support person taking the question to an internal team member where it requires additional input. So there are clear paths for escalation if circumstances require it.
Finally the support mail allows for PGP encryption of communications too.
(I am not a Mullvad shill. Not a Mullvad employee. Just a satisfied customer)
Human psychology is weird and some things are just cultural. If you have the ops team make the security@ email alias just forward to support, you could avoid having to go into all that.
"Just email support@" feels like you don't care. That you do, and that your support team is awesome, doesn't change the fact that there are other companies out there who's aren't. Security people are human with human egos, and they want to feel special, so giving them a special way to reach you, even if it's the same thing behind the scene, makes a world of difference.
"We don't have a dedicated security team because security and privacy are integral to all aspects of our service".
Do you have people whose role is explicitly security? Who are the security SMEs in your organization if not? I personally find the "Security is so important to us that we don't have a team dedicated to it" argument weak, and often results in misaligned incentives - if individuals have to alternate hats from "deliver results" to "properly vet security", the business push to deliver tends to win out. I'd be very curious to hear how you ensure your team doesn't fall into that trap.
Jesus talks and articles have been very helpful to me.
Three years ago I had no idea how the Go runtime worked, but I very much wanted to learn more. I’m not a software engineer in the conventional sense, so reading the Go source was not a realistic option.
Jesus talks and articles inspired me to learn more. Today I feel comfortable with all stages of the general compiler pipeline. In the past few months I have studied the calculi of the lambda cube, Martin-Lof type theory, Horn-clause-based instruction selection, algorithms for register allocation, Milner’s CCS vs pi-calculus, which structures in compilers and kernels map to digraphs, and so on.
I just want more trustworthy systems. This particular concept of combining reproducible builds, remote attestation and transparency logs is something I came up with in 2018. My colleagues and I started working on it, took a detour into hardware (tillitis.se) and kind of got stuck on the transparency part (sigsum.org, transparency.dev, witness-network.org).
Then we discovered snapshot.debian.org wasn't feeling well, so that was another (important) detour.
Part of me wish we had focused more on getting System Transparency in its entirety in production at Mullvad. On the other hand I certainly don't regret us creating Tillitis TKey, Sigsum, taking care of Debian Snapshot service, and several other things.
Now, six years later, systemd and other projects have gotten a long way to building several of the things we need for ST. It doesn't make sense to do double work, so I want to seize the moment and make sure we coordinate.
It sounds like you want to achieve system transparency, but I don't see any clear mention of reproducible builds or transparency logs anywhere.
I have followed systemd's efforts into Secure Boot and TPM use with great interest. It has become increasingly clear that you are heading in a very similar direction to these projects:
- Hal Finney's transparent server
- Keylime
- System Transparency
- Project Oak
- Apple Private Cloud Compute
- Moxie's Confer.to
I still remember Jason introducing me to Lennart at FOSDEM in 2020, and we had a short conversation about System Transparency.
I'd love to meet up at FOSDEM. Email me at fredrik@mullvad.net.
Edit: Here we are six years later, and I'm pretty sure we'll eventually replace a lot of things we built with things that the systemd community has now built. On a related note, I think you should consider using Sigsum as your transparency log. :)
Edit2: For anyone interested, here's a recent lightning talk I did that explains the concept that all project above are striving towards, and likely Amutable as well: https://www.youtube.com/watch?v=Lo0gxBWwwQE
Our entire team will be at FOSDEM, and we'd be thrilled to meet more of the Mullvad team. Protecting systems like yours is core to us. We want to understand how we put the right roots of trust and observability into your hands.
Edit: I've reached out privately by email for next steps, as you requested.
Hi David. Great! I actually wasn't planning on going due to other things, but this is worth re-arranging my schedule a bit. See you later this week. Please email me your contact details.
As I mentioned above, we've followed systemd's development in recent years with great interest, as well as that of some other projects. When I started(*) the System Transparency project it was very much a research project.
Today, almost seven years later, I think there's a great opportunity for us to reduce our maintenance burden by re-architecting on top of systemd, and some other things. That way we can focus on other things. There's still a lot of work to do on standardizing transparency building blocks, the witness ecosystem(**), and building an authentication mechanism for system transparency that weaves it all together.
I'm more than happy to share my notes with you. Best case you build exactly what we want. Then we don't have to do it. :)
I'm super far from an expert on this, but it NEEDS reproducible builds, right? You need to start from a known good, trusted state - otherwise you cannot trust any new system states. You also need it for updates.
Well, it comes down to what trust assumptions you're OK with. Reproducible reduces trust in the build environment, but you still need to ensure authenticity of the source somehow. Verified boot, measured boot, repro builds, local/remote attestation, and transparency logging provide different things. Combined they form the possibility of a sort of authentication mechanism between a server and client. However, all of the concepts are useful by themselves.
Obviously it’s far more nuanced than that. I’d say there are several categories where a reasonable person could have reservations (or not) about LLMs:
Copyright issues (related to training data and inference), openness (OSS, model parameters, training data), sovereignty (geopolitically, individually), privacy, deskilling, manipulation (with or without human intent), AGI doom. I have a list but not in front of me right now.
Yes, and those are interesting topics to discuss. "AI is useless and I refuse to use it and hate you if you do" isn't, yet look at most of the replies here.
> Yes, and those are interesting topics to discuss. "AI is useless and I refuse to use it and hate you if you do" isn't...
Did you read Mr. Bushell's policy [0], which is linked to by TFA? Here's a very relevant pair of sentences from the document:
Whilst I abstain from AI usage, I will continue to work with clients and colleagues who choose to use AI themselves. Where necessary I will integrate AI output given by others on the agreement that I am not held accountable for the combined work.
And from the "Ensloppification" article [1], also linked by TFA:
I’d say [Declan] Chidlow verges towards AI apologism in places but overall writes a rational piece. [2] My key takeaway is to avoid hostility towards individuals†. I don’t believe I’ve ever crossed that line, except the time I attacked you [3] for ruining the web.
† I reserve the right to “punch up” and call individuals like Sam Altman a grifter in clown’s garb.
Based on this information, it doesn't seem that Mr. Bushell will hate anyone for using "AI" tools... unless they're CEO pushers.
Or are you talking in generalities? If you are, then I find the unending stream of hype articles from folks using this quarter's hottest tool to be extremely disinteresting. It's important for folks who object to the LLM hype train to publish and publicize articles as a counterpoint to the prevailing discussion.
As an aside, the LLM hype reminds me of the hype for Kubernetes (which I was personally enmeshed in for a great many years), as well as the Metaverse and various varieties of Blockchain hype (which I was merely a bystander for).
That's a very thorough takedown of something the guy you're replying to never said. The end of their comment was "yet look at most of the replies here".
> That's a very thorough takedown of something the guy you're replying to never said.
Nah. Consider the context:
Aren't we all tired by this anti-AI stuff?
"Look at how I use this cool new technology" tends to be much more interesting to me than "this new technology has changed my job and I refuse to use it because I'm afraid".
[Copyright concerns, openness, sovereignty, privacy, deskilling, manipulation and AGI doom] are interesting topics to discuss. "AI is useless and I refuse to use it and hate you if you do" isn't, yet look at most of the replies here.
This trail of complaints was about the article from which I quoted.
Given the opinion on anti-"AI" articles suggested by that trail of complaints, I'd wager he either didn't read, or didn't thoroughly read the article and the supporting materials it links to. That's totally fine; but do folks the courtesy of going back and reading more carefully (or at all) when someone indicates that one's understanding of the material is substantially incorrect.
> An account-number model like Mullvad's would seem preferable
Thank you! :)
> .. assuming vendor’s TEE actually works
For sure TEEs have a rich history of vulnerabilities and nuanced limitations in their threat models. As a concept however, it is really powerful, and implementers will likely get things more and more right.
As for GPUs, some of Nvidia’s hardware does support remote attestation.
He IS a hacker from the 90s. It’s an assumed name. Plenty of hackers from the 90s have pseudonyms.
> so-called creator of some encryption protocol
All evidence points to him being one of the protocol’s designers, along with Trevor Perrin.
I’ve met both of them. The first time I met Moxie and talked about axolotl (as it was called back then) was in 2014. Moxie and Trevor strike me as having more integrity and conviction than most. There is no doubt in my mind that they are real and genuine.
Interestingly enough, some of the work Trevor did related to Signal’s cryptography was later used by Jason Donenfeld in the design of WireGuard.
> It screams honeypot like nothing else.
As you can see there is plenty of evidence suggesting otherwise.
Edit: In hindsight I regret making this comment. It was unnecessary, but removing it now would look weird.
reply