* The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
* The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223)
* The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner)
* The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server
* Neither servers appear to be hacked
* Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses
---
Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?
XMPP supports channel binding, which is not mentioned in this post but would have prevented this attack. Unfortunately jabber.ru is running server software from 2016, and that old version doesn't support it.
Pinning is problematic these days, because certificates are short-lived and renewed frequently. Users would be constantly asked to accept new certificates on a monthly basis, and they wouldn't think twice about clicking 'Accept' on an attacker's certificate then.
Channel binding works regardless of the certificate, and ensures the TLS stream is terminated by the entity you exepect.
The focus on (lack of) CT/SCT support in XMPP clients puzzles me, because it would not have detected this attack at all, which used valid CT-logged certs from Let's Encrypt. SCT verification would help detect theoretical issuance from another CA if they had a strict CAA record in place (they did not). But even then, channel binding is more privacy-friendly and does not depend on a third party.
Yes, you can pin the public key instead, which is generally more helpful. But most ACME clients (including the "official" certbot) default to rotating the key too. That can be disabled, but it's a problematic default for this use case which means clients can't just enable pinning.
Someone was forced to do it, but they didn’t personally agree with it so they eventually made a “mistake” to tip off the target?
There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.
I mean, I’ve seen the auto renew fail a lot with the certbot. They definitely should have checked it in the renew period to make sure it was working, but I feel for them
>affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223)
Why did they only target the STARTTLS port? On a related note, I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...
>I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...
That is what XMPP clients tend to do...
These days XMPP servers tend to default to requiring TLS on both 5222 and 5223 (Let's Encrypt has changed everything). Prosody does this for example. It doesn't even support port 5223 by default anymore. Port 5223 was never an official port assignment.
So it is very possible that the MiTM was only done on port 5222 because that was the only port that clients were using.
The MiTM attacker can pass through a command stream without STARTTLS. If they intercepted 5223 they would have to do their own client-side TLS handshake with the attacked server, which would look really obvious to anybody doing TLS fingerprinting on the server: all of a sudden, 100% of their clients have the exact same TLS fingerprint.
Stop outsourcing your PKI to ICANN, folks. Domains are not public keys.
They were doing their own TLS handshake - that's how the attack was discovered (the attacker presented a different certicate, which eventually expired, presumably due to negligence). They were decrypting and re-encrypting.
I don't know what jabber.ru's policy is, it's running a very old version of ejabberd. But you would be hard-pressed to find an XMPP server that would allow authentication without TLS. Starttls makes no difference.
It is pretty simple.
5223 port is called "legacy SSL".
So, because it is legacy, clients would, by default, use 5222 + starttls.
I.e. majority of clients connect with 5222.
Also, I've never understood why they've moved 5223 (regular TLS) into deprecated. It was pretty useful to enable SSL on AWS ELB on this port. Which is not possible with 5222, because it does XML stuff before switching to TLS.
I would speculate it is to not keep 2 ports open or something, was a reason to move it to deprecated. + you can do some cleartext communication before switching to TLS (not like it is that useful).
>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt
This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.
No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic.
The same way they are intercepting the jabber port they can intercept any other port for the HTTP ACME challenge. No need to get involved at the name server level.
Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.
Your private keys at rest are / should be encrypted, so it would take a bit more than just reading them.
The next level of mitigating that sort of a thing is to have keys not be on the hosts at all. Enter HSM - Hardware Security Module. A wildly complex topic I cannot hope to cover in an HN comment, but fundamentally, the private keys are not on the same HW as the server software which needs them.
A fundamental property of an HSM is that you, the HSM user, don't actually see the private key. You can ask the HSM to generate one. Derive things from it (in the cryptographic sense of derive). Even prove the provenance of such derived data. But the HSM should not reveal the actual private key.
In the cloud world, the HSM equivalent is known as KMS (Key Management Service), and the Good cloud providers all let you manage your own (with the downside being that you now need to manage your own keys).
Here's a good example why trust is not binary ("you either trust Hetzner or don't"). Quite possibly wiretapping law doesn't require them to facilitate the server breakin.
Not just. The disk itself may be encrypted, the keys may be encrypted. Doing this unnoticed and without rebooting would require modifications to the hardware.
How would you do certificate pinning if you don't control the clients?
My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI.
In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is used for.
Of course, if you do provide your own clients, this just moves the problem further up the chain - in this case the place where customers would download the custom client software would be compromised and a malicious client served instead.
> How would you do certificate pinning if you don't control the clients?
Well you cannot. If you were paranoid, you would perhaps supply a hash through some out-of-band mechanism, which would require manually updating for each new cert.
Obviously most people wouldn't ever want to do that.
These are usually to validate the keys used in end-to-end encryption. Both parties must confirm that they see the same details, which confirms that the same keys are being used on both ends.
Tele communication companies have internal “police groups” (where I am from - I expect it to be the same in Germany) that does nothing but service wiretapping requests from the police. Telcos are required by law to do this.
Them expanding into mtm https wiretapping is a new for me but maybe to be expected …
This list lacks the most obvious one: enable (and ideally enforce) SCRAM-xxxxx-PLUS as the authentication method of choice.
The idea of the PLUS variant is both simple and effective: instead of verifying <user,password> with the help of a salt you are verify <user,password,tls session key>.
That way the authenticating is only valid on a single TLS connection.
Who are the end users of xmpp.ru and jabber.ru? Are they hoping to pick up traffic between Russian soldiers? Spies? I hate mass surveillance as much as the next guy but why target Russian domains specifically?
I can't imagine it is war-related, or at least not in any direct sense, like literally trying to intercept soldiers or spies.
I think the more likely scenario is that someone was to catch carders/botnet operators, since Jabber/XMPP is still very popular amongst people in that scene in Russia; you'll see often see screenshots or logs containing @exploit.in, @jabber.ru, and various other servers pretty much in any Krebs on Security article, for example.
That said, I think mass-surveillance is usually pretty gross regardless of intent.
I don't know what the exact ratio of criminal to non-criminal usage of either server is, but just like how Tor simultaneously has a ton of obviously illegal content mixed with whatever percentage of legitimate use [1], they presumably still do have a fair amount of totally normal conversations going on. And so I'm not an exactly a huge fan of invading privacy in order to spy on random cybercriminals likely far out of your jurisdiction in countries unwilling to extradite in the first place.
And while I understand if Hetzner was forced to comply, unless they wanted to become another Lavabit, as someone who recommends them often, it'll still be a little disappointing, IMO, if it turns out they were aware and allowed or assisted in the MitM attack, because otherwise, I think they're one of the best hosting companies.
Their transparency is great. And I get better uptime, reliability, and support than the overwhelming majority of cloud providers without selling my organs to afford it. And their effort put into turning environmentally-friendly design choices into economically smart choices is commendable.
The "reduce" in reduce > reuse > recycle can be seen in their KISS designs—like the raised, angled roof, allowing natural heat escape rather than loads of HVAC [2]; server chassises that are more bare-minimum structural support and air-shrouds than actual chassises; and stripped-down standard motherboard designs with only the most minimal changes, like 90-degree rotated sockets to allow a single fan to cool the VRM, CPU, and memory [3], rather than a dozen screaming, power-slurping Delta fans for every server.
And their knowledge of "reuse" is evident by their server auctions, where they do Dutch-auction-style rentals of legacy hardware rather than scrapping everything, avoiding spending a load on new hardware while also avoiding the environmental cost associated with manufacturing new everything each generation.
I'd love to be able use them more if only they had dedicated servers outside of Europe.
[1]: "legitimate" is hard to define anyway; one country's journalist is another's "foreign agent", and one country's "freedom fighter" is another's "terrorist".
> I can't imagine it is war-related, or at least not in any direct sense, like literally trying to intercept soldiers or spies.
The Ukrainian military has been using Discord, so it's not totally unimaginable. Are these domains administered by Russians? IMO it would be pretty naive for Russians to be hosting comm servers in NATO datacenters right now. Perhaps related: I interviewed for a job recently where the hiring manager was a Russian immigrant. He was telling me I should go visit St. Petersburg and didn't seem to understand that is not an option at the moment. I was flattered by the suggestion of course and it is a nice reminder that regular people are typically tolerant, but also confused. Do people not understand the severity of the situation or is this behavior of acting like it is business as usual a coping mechanism?
> I can't imagine it is war-related, or at least not in any direct sense, like literally trying to intercept soldiers or spies.
Perhaps a majority of war between technologically developed countries these days occurs on the internet - just because no soldiers or spies are involved in something, that doesn't mean it isn't direct war. Example: propaganda around Russia/Ukraine last year and Israel/Palestine this year
They point out that "CT is optional", but isn't that ultimately a decision by the XMPP clients? What stops them from requiring CT now that browsers effectively do? What trust store would they be relying on that issues certificates that don't work in the browser?
Right, they could do this and this seems like a good idea to me. The problem this require-CT behaviour isn't switched on by default for basically any TLS library, so an XMPP client developer would have to go out of their way to switch this on, and I assume most or all (currently) don't. That can change of course.
How problematic is that in real life? All CAs worth their salt are already publishing all certificates to CT logs because most TLS servers are browsers and most clients are browsers.
Maybe there's a CA somewhere that has opt-in CT logging enabled, but I don't think they're very popular. For example, even the rogue CA certificates were published onto the CT log (because Let's Encrypt does so by default).
I think a CT check shouldn't be a very problematic change in practice, as long as there are reasonable exemptions (i.e. don't require CT for certificates imported into the CA store, allow the user to disable the check, etc.). For server<->server communication things become more difficult, but the entire network would be better off if a major server like ejabberd were to require CT.
Thinking laterally for a moment regarding the big picture here, why do we still rely on data centres.
They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points.
And the more we break away from data centres and clouds, the more the internet infrastructure will have to work the way it was designed instead of having to flow through these crazy aggregation points that are both serious points of failure and major security risks.
> They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points.
Yes, but then you need backup power, someone to replace disks / hardware if things break, proper security for compliance reasons, cooling, noise. Once you set up all these things you just invented a data center again.
I don't see how getting rid of data centers makes any sense.
You can get all of that in your own business premises with a fiber uplink. But at the point you're staffing IT personnel and managing server racks, I suppose you may as well call the location a datacenter, albeit a private one.
Regular domestic connections often block ports (especially email). In many countries, ISPs won't sell you a static IP (your dynamic address might not change ever, but you don't control rDNS records).
These kind of measures force people to move onto a rented server instead. Often ISPs rent servers themselves. The conflict of interest here is hard to ignore: if ISPs make it easy and convenient to host from home, their business of renting servers suffers.
All the people working in the datacenter have that level of physical access.
Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen:
- unlocked racks
- doors open
- temporary network cables and keyboards, monitors and mice attached to running systems
- systems logged in left unattended
- floor panels raised up and left open unattended exposing cabling
- meet-me rooms with interfaces exposed (gear in racks without doors)
DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18...
Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
* Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now.
* Require them to sign into and out of the site.
* Annotate which systems / compromises are in place.
- That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc)
Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
> realistically you just have to let them in anyway
No, you don't. If they have a warrant then you need to let them in for the purposes specified in the warrant. Otherwise you're free to tell them to piss off. Unfortunately you're also free to acquiesce to any of their demands.
This kind of passive, default-compliant attitude from service providers, while understandable from a "path of least resistance" standpoint, is exactly the kind of behavior that allows the third party doctrine to circumvent so many of our basic rights. As a service provider, often the more difficult path is to challenge authority, rather than to cooperate with it. And unfortunately that means that most service providers will simply cooperate.
> No, you don't. If they have a warrant then you need to let them in for the purposes specified in the warrant. Otherwise you're free to tell them to piss off.
Any lawyer will tell you - if law enforcement attempts a warrant-less search, you tell them you do not consent to it, but you do not attempt to physically stop them from performing it. Tell them they are unwelcome and to come back with a warrant, but if they insist on entering in spite of that, you let them in.
If you unlock a door for someone but simultaneously say “I don’t consent to you passing through it”, the first act does not cancel out the second. Whereas, if you don’t unlock it, if they really want to go in they’ll knock it down, causing damage in the process. Unlocking it for them is about avoiding damage to property, it is not a form of consent if accompanied by a clear verbal refusal of consent
Non-compliance with a law enforcement order is a good way to get shot (in America) or arrested (in most countries) even if there is no legal basis for the order.
Latter is not correct.
It's well known difference in Russia between companies that willingly cooperate with government agencies informally, and those who just provide information upon formal request according to law.
How do you know for sure the people who “just provide information upon formal request according to law” aren’t covertly engaging in informal cooperation?
If one morning the CEO gets an unexpected visit at home from a group of FSB agents asking for some favours, is the CEO going to say “no”? And if the CEO says “yes”, are you going to hear about it, or are they going to let the CEO continue that pretence?
Western CEOs don’t have the same worry about “accidentally” falling out of hospital windows.
Actually, that's happen to some people that I know.
Roem.ru site (small but ifluential at time) recieved official, but illegal request from high level FSB agent to disclose commentators identities. They send formal complaint to a FSB own security and to public prosecutor office.
Former officially warned FSB to stoppes illegal actions.
Funny thing: 7 years later FSB agent was convicted for being CIA asset.
I highly doubt it is that simple for LE to enter a DC without a warrant signed by a judge, but insiders have all of that access and plants in DCs can and do happen.
I was present when Dutch LE seized a bunch of servers on behalf of an FBI liaison officer in NL and everything went 'by the book', there is no way an LE officer without a signed order from a judge would have been granted access.
You would be 99% wrong. Even if law enforcment presented proper paperwork, every colo I have ever used would call and verify the paperwork. They might not call me, but they sure as hell would call their own lawyers. Once law enforcement is on the other side of the cage, important customers who pay real money could get compromised.
There is a massive difference between getting physical access to your server in a data center and coughing up everything about your server by simply emailing a minion in a cloud provider.
Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.
And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same.
Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target.
And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal...
... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal.
All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in.
I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead.
No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS.
I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way.
And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8].
> Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.
Amusingly, I read about an incident like this on one of those forums (probably ServeTheHome). It apparently happens so often that Hetzner's control panel has a special state for it. Server status: "seized by law enforcement" and the power-on button is disabled.
Next time you try that trick, use heptane (sold commercially in North America as "Un-du") instead of isopropanol. Hot knife meets butter... and it evaporates cleanly and the sticker retains most of its stickiness!
You can try to see if your provider supports "opaque" types in which you can submit the binary encoding of the record without them knowing what it is. Barring that you'll need to request support for it. It's an increasingly popular record type so not supporting it isn't terribly great service nowadays.
I'm not familiar with the discussion that went into the design of CAA (it was probably discussed at some point on the relevant IETF mailing list, if you want to go digging).
I have found that some providers that say they don't support it just mean in their web interface. It may be worth testing if you can set up a hidden primary server and have your provider just be a secondary to it. That is how I managed to get around that restriction with Linode.
Yes, but if you can serve multiple certificates on one endpoint (think SNI) then you can add your own self-signed or private PKI certificate to be able to check if all your requests are being intercepted by a lazy adversary.
The Hetzner one is a physical server. You would need to stage a "power outage" and backdoor it, which is probably not that easy - e.g. planting a kernel module which survives kernel upgrades and is pretty advanced at hiding itself (the article talks about analyzing raw memory dump).
Secure boot on a cloud machine is pretty useless, there's nothing stopping the hypervisor from injecting code into the running machine. Theoretically virtual machine memory is encrypted, but you'll just have to trust the hypervisor's word for it. You can try to verify the boot chain all the way to the hardware keys, but if the hypervisor just replaces your `JNE` with a `NOP` you'll have a hard time automating your protections.
I suppose you can transfer the keys out of the machine over the network (and hope the hypervisor doesn't replace the socket buffers just before transmission) and verify them off site, but guest machines will always be just that: guests on a host that has all the power.
That is not the case for the latest CPU extensions for encrypted VMs, AMD SEV-SNP and Intel TDX, which are designed to allow remote attestation based on a key hidden in the CPU that the hypervisor does not get access to.
The hypervisor only ever sees the VM’s memory in encrypted form, and it’s integrity-checked by the CPU to prevent replay attacks.
I'm not sure truly verifiable "Secure" Boot is actually realistic in most scenarios, though, right?
I'm not super up-to-date on what's being offered right now, but I'm not sure if there is a way to have a proper trusted execution environment on most Intel or AMD offerings; I thought Secure Boot on AMD64 platforms generally rely on TPM or something like SGX for validation, with the former having seemingly a dozen different ways to be tampered with, and the latter being discontinued and being vulnerable to several different attacks, including DOWNFALL.
I think EPYC and Sapphire Rapids have some sort of Trusted Execution Environment stuff with SEV-SNP and TDX, maybe? But I don't think either option is really feasible for people paying Hetzner-like prices for hosting; Hetzner's newest Xeon offering is seemingly Cascade Lake, and the only EPYC offered is a single-socket Rome 7502P with 128GB DDR4 for 142 euros, which seems very hard to justify, given they also offer a 7950X3D with 128GB DDR5 for ~25 euros less.
Even then, I don't think I could put my confidence in a machine I don't own, didn't setup, can't physically inspect, don't know where it came from, whether the firmware has been tampered with, etc -- especially if it is something as complex as x86, where there is seemingly at least one new horrific hardware-level vulnerability that crops up every generation or two.
EDIT: I forgot Hetzner also started offering Ampere Altra servers for 200 euros. I think those have TEE of some sort with the TrustZone stuff?
Not sure how secure that really is, though; I haven't really looked into the ARM offerings as much as I should have, mostly since, if you don't want Apple, I'm not aware of a good middle-ground between a cheap SBC and a $3,000+ Ampere server, outside of jerry-rigging some second-hand Gigabyte Cavium ThunderX2 nodes off eBay.
Only if secure boot was enabled by a trusted party on trusted hardware.
If you enable secure boot remotely without physical access to the machine you can't be sure it was actually setup in a non-compromised way. For example the machine could be running a custom backdoor-ed TPM, BIOS settings, ...
What is the fundamental technical solution for these kinds of attacks from nation-states and other powerful actors? Is there anything which can satisfy the nerdy dream of software which is freely distributable, yet completely safe from censorship, intervention or mutation by anyone other than its author?
Are properly built P2P systems over the current internet infrastructure a good enough answer? Or should we rolling out a second low-bandwidth global mesh internet working over radio?
I just hope Freenet 2023 evolves to have a zero-friction DX for P2P applications.
A quick warning on hetzner. I needed a personal bare metal machine so signed up.
I was travelling and on an IP in a distant land so their sign up asked for secondary verification via PayPal. All passed and now it’s should get a server?
Nope - next day their support emailed telling me they would not approve my account without… no word of a lie here… either 1: a fax of my passport info page or 2: a scan and email containing the same.
I refused reminding them of GDPR and that email is at best opportunistically encrypted and at worst open to interception.
They replied stating they believed they were GDPR compliant because all they do is use the passport to verify the account and delete the document. They also said I could hide anything sensitive other than my name and date of birth!!
I suggested the process is not GDPR compliant as anyone could intercept unencrypted emails and that they should talk to a lawyer if they did not believe my assertion.
Within a short time the server was approved and online. I queried if they would revise their process in light of our interaction. They did not address the question.
I tried to sign up from The Netherlands for a server hosted in Germany.
They asked me to email them a copy of my ID card for verification. They refused to have me as a customer even after I provided it. I guess they didn't like the look of my face? They refused to tell me how I could still proceed, and I would've happily paid a year in advance or something. Heck, I could've probably visited their office in person.
I did try to register from a Protonmail account because I self-host my email and don't want to run into a chicken-and-egg problem (need to mail them because my server is down, can't mail them because my server is down...), so perhaps that was the issue.
Because Hetzner refused my business I was forced to look for another provider, and I've been using OVH without any issue for a few years now. Quite a shame though, because Hetzner definitely seems to provide a superior service - provided you can get them to take your business!
I absolutely hate any process that requires uploading my passport or other identity documents. I always redact as much as possible and then cover the photo with red text saying "FOR $BUSINESS IDENTITY VERIFICATION ONLY." Sometimes they push back on it, but usually it's acceptable. The worst is when their automated system rejects it.
On email: "While encryption is not required, it is up to every organization to develop a rationale for developing the most appropriate data security practices."
Fax is an accepted GDPR-compliant form of "secure" communication. Yes, many service providers need to ascertain your identity. In my experience, passport photo verification is normal at Hetzner, Hetzner is very aware that not everyone wants to meet the requirements for their service, and they will happily refund you if you decide not to proceed.
This passport verification is required in Germany for any Internet connection. You need to do the same if you sign up for a cellphone plan in Germany. It's a surveillance state.
They had the option to send it encrypted with PGP. But yes, this reminds me of communist countries where you had to leave your ID at the hotel upon check in. The Stasi mentality lingers on and accomplishes nothing.
I think it is less Stasi and more so 30 euro dedicated servers with unmetered gigabit lines are ripe for bad clients.
You've got the general issue of abuse and fraud that all providers face. But I think there are two issues that make it worse for companies like Hetzner, OVH, and other low-cost providers:
1. Chargebacks are a big deal, both in terms of being cut off from payment networks but also the fees imposed, which are especially harsh if your margins are likely razor thin sometimes; looking at the server auctions right now, it's kind of wild that Hetzner manages to give you a place in a datacenter, 3700X, 64GB RAM, 2x1TB enterprise SSDs, and unmetered gigabit on a decent network for 33 euros while still making a profit.
2. I would imagine that attempting the proper credit-card theft kind of fraud is also more of an issue for low-cost providers, not only because of #1, but because I think you'd manage to keep and abuse servers bought with stolen money for a lot longer; I think legitimate owners of said cards are less likely to notice 30 euro charges every month compared to being robbed blind by unexpected AWS fees.
I've had to deal with anti-fraud paranoia from OVH, BuyVM, Hetzner, and many others, likely all for the same reasons as Hetzner.
Both Hetzner and OVH refused to provide me service without photo ID or a passport. BuyVM refused one of my Jordanian friends entirely unless he paid in crypto. And while minor in comparison, I've had to change my PayPal email to match my account email on BuyVM despite it literally previously being paypal@myfirstandlastname.tld.
Not meant to be a dig at BuyVM, btw, even though the crypto bit may seem harsh. I really like them; freedom to host pretty much anything that isn't straight-up illegal, even Tor exit nodes, the support is good, they're often around and transparent in the community chat, and it's hard to beat free BGP announces, up to 10Gbps speeds, anycasted VMs across Europe and the US for <$10/month, and optional $3/month DDoS mitigation -- especially since it's the kind of small enough, tight-ship type of thing to where you can moan about poor routing or custom mitigation filters and potentially have someone actually try to take care of it (and it's also hard to beat for abusive or awful clients too, hence the trouble).
They provide great service, even if only to use as a DDoS-mitigated tunnel for more powerful servers elsewhere, or as a CDN, etc.
BuyVM comes off as one of those sites that's explicitly set up to host illegal content, but with plausible deniability. I don't understand how they're still allowed to remain operating. Governments usually err on the side of arresting innocent people.
P.S. Hetzner gigabit is not actually unmetered, but the limits are vague, but high (>100TB/month)
Plenty of modern hotels in Western countries require you to submit your passport to reception, who then scans it and keeps a copy of it. In fact Marriot recently suffered a data breach where the attacker obtained the photos of these passports.
Of course, that's not mutually exclusive with Western nations having "The Stasi Mentality..."
I know. Most of them have a fill in the blanks sheet with name address and document s/n.
I can understand scanning documents if one rents a vehicle, although for hotels I fail to see why the form shouldn't be enough. It's not like they don't have cameras at the reception desk and I don't pay with a card under my name, they can also check my ID and fill in the form themselves if they don't trust the customer. Why should I trust them to properly handle copies of my ID? They are not a bank operating under strict regulation.
The thing is back then they were physically keeping your ID until you've paid in full and checked out. Last time it happened to us in Serbia, in 2019. The receptionist regarded us with suspicion. Brought back memories from the '80s. If you've played Papers Please you know what I mean. Now about 8 Schengen Area states have introduced border checks. Great.
Interesting. That didn't happen to me when I stayed a month in an Airbnb in Serbia, although the host did ask me for a copy of my passport. (Not that I'd expect it from an Airbnb host even if hotels were doing it... just an additional data point).
For renting a vehicle (or more likely, something like a bike or moped), I can understand why they take your ID as a form of collateral in exchange for the material goods they're lending you. But for a hotel there's no real reason to hold onto it.
Out of curiosity, did you acquiesce and leave your passport with them?
> Don't use Cloudflare or similar services. See my article here for an explanation on why. If you use a service like this, you're basically already MitMing yourself.
I wish more people would realize that when arguing on the internet about CAA, DNSSEC, NSA, etc. that none of it really matters. We willingly allow a government aligned entity to unwrap 20% of all TLS connections on the internet and peak inside.
Cloudflare is horrible for privacy. It is also a bit of a sovereignty issue for European countries to have all their citizens web habits to be MITM by a forging power (no matter how friendly they seam).
Edit: not even going in to the sovereignty issue of having an American private company effectively decide your internet regulations.
Which lets be honest isn't a problem that 99% of the sites using Cloudflare need to solve. Nobody is going to waste energy and time to attack your blog with your vacation photos.
The huge wave of DDoS extortion attacks that happened 3-4 years ago was mostly enabled by "booter" services that themselves hid from law enforcement behind Cloudflare.
Beat me to it. More often than not, every time I run into the run-of-the-mill booter or other super blatantly illegal things, at least some portion of the infrastructure is behind CloudFlare.
I don't have enough fingers and toes to count the number of abuse reports I've personally filed that seemingly go nowhere.
OVH at least takes down the booters and other proper malicious stuff. However, copyright infringement reports go ignored to the point to where I stopped bothering--not that I had any right to care when I torrented loads, but it was a little annoying that ~$10 software, with no DRM, regional discounts, and a dozen ways to pay still ended up pirated to the point to where it wasn't worth selling
(fun story: I still have email records from when a pretty beloved multi-billion dollar tech company launched their ROG-clone sub-brand and pirated our XenForo plugins to use on their site instead of ponying up $10 -- maybe one day I'll have to ask them for a laptop).
CloudFlare, on the other hand, at best, just passed on whatever I send in, and whoever their actual providers were never cared either, so nothing gets done. It can feel like rackeetering; sign-up for protection from the people attacking you ...whom we also protect and refuse to take down.
That said, I think 99% of small sites, personal blogs, etc, really don't need any of the services offered; the most useful offering for most people is literally probably DNS.
Game servers and gaming-related communities, or at the least the ones I am most attracted to, are just about the only thing I think requires DDoS mitigation regardless of size. Our little Garry's Mod TTT server with a max player limit of like ~24 people would get attacked a couple of times a week at least.
And good lord, the RuneScape community is another breed of toxic. The way I learned to program was by reverse engineering the game, and we'd work on private servers that were essentially from-scratch MMORPG servers that just emulated the RuneScape mechanics and protocol.
One aspect of the game, though, is PvP -- when you kill another player in the wilderness, unlike in a lot of other MMOs, you get most, if not all, of their items they have on them.
I could perhaps understand DDoSing another player during combat in order to steal their gear. But what made me stunned beyond belief was that when I hosted a "spawn PKing" server (e.g., PvP-only, but you don't have to work for your gear; everything is free - it's just for fun), they'd literally lure each other into TeamSpeak to grab IP addresses and DDoS each other for.... nothing :)
(and so you can imagine also just how often we'd get hit with massive attacks ourselves)
DDoS protection is standard among hosting providers now, including budget ones like OVH.
The fact that Cloudflare is allowed to continue hosting websites which are obviously illegal, some notorious, is deeply strange. As I wrote in my article on the subject, it makes no sense when you consider the way the US responds even just to copyright infringement; see how they nuked Megaupload's business without trial because they saw them as knowingly enabling piracy. However, it's a known fact that US authorities will keep illegal or disreputable services up if they see them as a source of more intelligence. I can't really see any other explanation for how Cloudflare is allowed to host some of the sites it does without pressure from the US unless it's basically funnelling all of the data to the NSA.
In this aspect, Cloudflare should be viewed similarly to an ISP. Why are ISPs allowed to host illegal sites? Well, they aren't supposed to pay much attention to what they're hosting - it's not their job. But they aren't supposed to protect what they're hosting, either. If they get a court order asking for the details of the subscriber hosting some website, they turn those details over. If they get a court order asking to turn off the service, they will. Governments are fine with this, because they can easily get the details upon request.
Cloudflare should be viewed the same way - they shield you from DDoS, not from the government. They allow everything to be hosted until proven otherwise. Cloudflare doesn't have to police what's hosted through it, because the police can do it easily enough.
There are lots of pirate websites, explicitly designed for piracy, but saying the opposite on their terms and conditions page to create a little plausible deniability. I can't tell you if Megaupload was one of those and I don't know what evidence the government had.
> Cloudflare exists out of necessity for the most part.
I agree with this, there don't seem to be that much self-hosted software that someone could (easily) setup for the use cases that Cloudflare serves.
> The alternatives to shield from large scale DDoS are all US American too.
Not only that, but the WAF functionality is also pretty useful. To be honest, the same applies to something like wanting to have CAPTCHAs on your own site - not that many options out there.
As far as I can tell as a hobbyist, if you wanted to host everything yourself:
- dealing with load: at best you can probably just run multiple nodes with round robin DNS and something like HAProxy, or even just live with Nginx/Apache/Caddy, though all of those would crumble under attack; probably with some resource limits in place so the software getting overloaded just crashes it (with automatic restarts) and doesn't grind the entire server down to a halt
- WAF: you could get a basic WAF running with something like Apache2 and mod_security, or if you can compile it for Nginx and get it working (a bit annoying to do, also apparently slower than Apache2 version), or something like Coraza (still new), but even then you need sets of rules, OWASP has some, but they're not updated as often as whatever Cloudflare uses, so the effectiveness of it all is debatable; there's also something like fail2ban, but some people really don't like it for some reason
- there's also additional stuff you can use, like LibreCaptcha for CAPTCHAs, something like Keycloak or Authentik for SSO and managing your users on prem (especially with mod_auth_openidc), stuff like Matomo Analytics instead of GA, Uptime Kuma for uptime monitoring, even your own self-hosted mail servers if you feel brave; but all of those take effort and need maintenance
And even then, certain things are not an option - you won't be shrugging off huge DDoS attacks and you probably won't be running your own CDN (easily), unless you have bunches of money to spend and the know-how. So of course people would rely on external orgs for whatever they can.
As a hobbyist, dealing with load consists of upgrading your $5 VPS to a $10 VPS or even a $50 dedicated server from Hetzner(!) - note that *no* other provider has dedicated servers at this price point.
WAFs are heuristics at best. If what you're running on your server is actually secure, you don't need a WAF. If it's not secure, the WAF is guaranteed to let through at least one attack.
CAPTCHAs are difficult. Try to avoid depending on them, but it's fair to use a third-party service if you need one. hCaptcha is pretty easy to integrate right now.
OVH absolutely does have dedicated servers at this price point, and below - check out their Eco range (previously SoYouStart) or even Kimsufi. Leaseweb often does as well, although they have not been as good a deal recently from my perspective (and can cost more for transfer).
If someone tries to send you more traffic than your link supports, your only way to survive it is if your provider can filter it. Cloudflare will actually do a decent job of that even on the free plans.
"Survive" is hyperbolic. If someone DDoSes your $5 website and it is down for a day until you sign up for Cloudflare, you do not literally die. You do not need to sell everyone's 24/7 browsing history to Cloudflare to keep your heart beating.
Most sites don't get DDoSed. https://immibis.com/ has been running without DDoS protection for a long time now. It's as simple as nobody caring to do so. Why would they? What's in it for them?
And if someone does knock it offline, I still don't care. I can wait until they get bored. The site isn't important to me, either.
And if I really do care, Cloudflare encourages people to sign up whilw they are actively under attack. Of course, it costs money, because you aren't paying with your access logs all the times you aren't under attack.
There are lots of reasons to not use Cloudflare, but many of those given in the article don't hold up. For example, Cloudflare does not set a cookie for all connections, discrimination against Tor users, CAPTCHAs and WAFs are all configurable.
* The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
* The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223)
* The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner)
* The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server
* Neither servers appear to be hacked
* Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses
---
Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?