> When the ISPs discovered that the IP addresses belonged to Cloudflare, arms were thrown up in despair.
The level of desperation the ISP’s engineers have felt in front of such incompetence must have been through the roof. I am getting tired of our politics here in europe: tech literate people in governments are put to work on surveilance stuff, never for actual policymaking.
Surely they should have contacted Cloudflare and asked them to sort out the issue with their network ...
CF will probably retort {common carrier, we are American and you are not etc}. Austrian policy makers get upset and ... THIS NONSENSE IS STILL NOT SORTED.
The internets are somewhat broken, quite badly. We all allow ourselves to end up in a series of virtual walled off silos - Facebook, Twitter etc, run by some pretty worrying monster commercial companies, whilst living within quite disparate physical societies. I think that the fediverse is a possible contender for the way forwards. I dumped Reddit for Lemmy and that seems to be working so far. I know a lot of 'X'iles are finding a home with Mastodon.
I worry about quite a few networks. One of the tools in the box is the IP block list. With the rise of the hyper-scalers, the IP blocklist is becoming increasingly useless. Same with SMTP filtering for spam. You can't block M365 or Gmail en-masse (tempting for my home setup, though!)
I recently gave CrowdSec a run at work (I will stick with it but with care). Great idea for the 2010s but not so much now. I got it to watch HA Proxy logs (proxies on site Exchange) and it pretty soon decided that my real users should be banned because of how Outlook works, or rather how the auth that Outlook uses works. Outlook doesn't just use Kerberos, it also uses EWS for the Addressbook and the good Lord knows what else. So you get a connection to a endpoint from Outlook without auth creds which generates a 401 (failed auth) error, then Outlook tries again, this time with creds and it works OK (200). To a CrowdSec agent those repeated 401s look like bots pissing around.
That is one reason why we cannot have nice things.
I'm a Brit (yes we Brexited) but I am still very much a European and so is my little country, like it or not - I have seen shed loads of number plates from across the EU trundling along the large A30/A37 crossroads/roundabout called Yeovil in Somerset. Mostly lorries and quite a few cars and campervans.
So I am from country X (GB in my case, not that X!) but I engage with (w,x,y,z) societies on line. Now whose laws apply? In general it seems we muddle along effectively but should there be a formal internationally accepted agreement?
401 does not indicate anything apart from authentication failure. Bear in mind that I am trying to parse logs to decide what is happening.
I'm no expert but I suspect that http works a bit like this:
C: hello web server
S: yes I exist and I speak the following languages: lol, rofl, powershell (on thursdays)
C: cool, let's talk lfor and I will sacrifice my first born child and give you my wallet
S: no chance, but I speak lol and thank you for your wife
C-S: witter on in "lol" for some time.
<cat memes flow regardless>
Kerberos has overloaded how auth should work and inadvertently fucked up log processing. Actually I think it is really NTLM. It goes in with auth instead of hello.
Foreign licensed vehicles can operate and drive in the UK for up to 30 days without applying to the DVLA (DMV) for UK registration.
However the UK government has never subscribed to any of the available EU crime and general population administration data sources which has always left the UK vulnerable to overseas criminal activities concentrating here because there's obviously no reciprocating return of information to the continental authorities either and so the UK has become the best place to let the law slip if you're up to no good in Europe.
> CF will probably retort {common carrier, we are American and you are not etc}.
Considering how quickly they reversed course on their grand statement about having principles, any such retort would probably be about as effective as they predicted in that statement.
.
> The internets are somewhat broken, quite badly. We all allow ourselves to end up in a series of virtual walled off silos
The problem with the Internet is that between nat and annoying rules and asymmetric connections, most home connections aren't suitable for serving things.
The big services being centralized isn't a problem with the Internet, which can be seen from how often non-Internet things consolidate the same way.
>The level of desperation the ISP’s engineers have felt in front of such incompetence must have been through the roof.
If I was an engineer there I would block the addresses and call it a day. What do I care? After all it's customer service who will deal with the angry calls.
Yes, who would also yell at you when customers can't access their cat pictures.
Of course it could be an accident, and you didn't know. But if it was clearly malicious compliance, personally and I think I speak for many, I wouldn't continue to have on the team somebody with these kinds of traits.
I don't think I understand. A court orders you to block certain IP addresses: you have no options. If that block disallows access to unrelated pages there's nothing either the engineer or the boss can do. Should the engineer lose any sleep over this? Am I missing something?
Courts don't work like computer programs, if the court order would have clearly unintended outcomes then you have some option to in good faith not comply with the order, have your lawyers raise it with the right politicians or courts, and not end up in legal trouble.
The job if the engineer who notices this is to raise it to their boss and legal counsel and let them decide whether you should still execute the planned block or not. If they decide you should then you do, if they decide you shouldn't then you don't.
> The job if the engineer who notices this is to raise it to their boss and legal counsel...
...shouldn't the legal department be involved way before the issue reaches the engineer? Why should the engineer ever care about it in the first place? Chances are these orders do not come directly to their email.
In this case the court said to block illegal content, and here are the IP addresses where that content is served which should be blocked. It passes the sniff test of the lawyers as not much different than blocking domains of piracy sites. They pass the requirement on through the engineering org, it eventually reaches an individual contributor who is tasked to make it a reality.
Then that engineer looks at the specific IPs and realizes that some of them are cloudflare ips and the collateral damage of blocked content would be significant. Their job is to then escalate and pass it back to higher managers and the legal department to confirm they knew about that implication and want to move forward.
The legal department at the isp having specialists means that they could understand this situation in a single email from the engineer, not that they obviously have the foresight or capability to even find out that some of the IPs might be for cloudflare. Or sometimes the lawyer might even assume that's not a big deal but higher managers can realize that it is a big deal and then discuss options.
All that I mean is that even at FAANG companies with extensive and competent legal departments its not correct to just blindly execute a change like this if you think it looks wrong; it's not at all a safe assumption that what you know is somehow a subset of what they know.
you are missing the experience of an asshole manager pressuring you to 'just fix it' in an enraged state accompanied by their phone constantly buzzing from the sweet notification sound of getting bombarded by emails/tickets.
for context, i didnt experience this yet. I hope i never do.
The association of IPs to individuals and particular services is the original sin of the Internet. At Cloudflare, we’re working to fix that sin. That doesn’t mean governments don’t have the right to regulate content on networks in their boarders: they absolutely do so long as they follow principles of Rule of Law. But blocking on an IP can never be transparent — and therefore violates the Rule of Law — and will always cause collateral damage. Terrific that the Austrian authorities recognized that.
> The association of IPs to individuals and particular services is the original sin of the Internet. At Cloudflare, we’re working to fix that sin.
... Being able to talk to someone without an intermediary is a problem?
> That doesn’t mean governments don’t have the right to regulate content on networks in their boarders: they absolutely do so long as they follow principles of Rule of Law.
Pretty sure governments have the right to do whatever they want regardless. It's that pesky "sovereignty" thing.
I wasn't able to npm install something because Cloudflare decided the IP I was currently assigned has to solve a captcha before downloading node packages.
Isn’t it funny how quickly they went from “we won’t block for any reason” to where they are today?
And the funny thing is that their blog post complains of the following:
> In a deeply troubling response, after both terminations we saw a dramatic increase in authoritarian regimes attempting to have us terminate security services for human rights organizations — often citing the language from our own justification back to us.
Well, yeah. That’s what happens. It’s just a lot easier to justify something you wanted to do already, isn’t it?
This is one of the useful tactics that unions will use when they aren’t legally allowed to strike for one reason or another, they will engage in large scale “work to rule”, which is pretty much what it sounds like, you do your work following the rules to the letter and no further, stubbornly by the book no matter how inconvenient or unproductive or disruptive the rules are when you suddenly start following them exactly.
It’s particularly effective if the rules are a large part of their collective grievances with management, as it highlights to all involved that the current rules need changing, and can sometimes result in fairly quick and positive changes since it demonstrates to management that the current “rules as written” are not the most fiscally effective set of rules for running the business.
Especially if you stop every single step to go back, make sure that every single previous step has been correctly completed, assign roles for the following steps, have a discussion on workers rights and the equal share of labor to make sure that the workload for each of those steps is distributed equally, and so on and so forth.
See also "Simple Sabotage Field Manual", by United States Office of Strategic Services, written by US military for use in occupied countries in Europe.
Ah, malicious compliance, my "last trick in the bag". When I'm tired of arguing, I do exactly what I'm told. Nothing opens eyes quite so well as getting exactly what you asked for.
Plenty of corporate web filtering solutions block on IPs, and the adoption of ESNI will only make it worse. If you manage an important website, moving out of a bad neighborhood (Cloudflare) before you end up the victim of blocking is probably a good idea.
I hope important websites do NOT move out of the public neighborhood (CloudFlare) to discourage honouring block requests (and ultimately also these requests from being requested) from incompetent regulators.
Fortunately, CF is great at providing DDoS protection and adaptive WAF services; which should incentives other website owners to keep using CFs services. And even better, now it cannot happen anymore because it violates (whatever is left of) net neutrality- awesome.
Have you read the adorable article? I’ll quote it for you in case you missed it.
> “With regard to the blocking of access to the IP address 190.115.18.20, the Telekom Control Commission found a violation of Article 3 Paragraph 3 of Regulation (EU) 2015/2120, because the IP access block poses the risk of ‘overblocking’ any website content.”
Layer 7 gear has been around for a long time. At an ISP level I’d be shocked if they weren’t running big F5s or something similar that can handle this properly. Actually do so is another matter entirely though, and I’m not even considering throughput.
Layer 7 gear has also been pretty irrelevant for a long time and not really something an ISP wants to waste money on when all it needs to do is deliver L3 fast and cheap. First payload encryption, then protocol encryption like the aforementioned ESNI or QUIC, and now DNS encryption. Even if you invest in it how much good is it going to do? Most ISPs have boxes that can handle volumetric attacks and the minimum required logging for legal compliance.
> According to reviews conducted by local telecoms regulator TKK, the IP address blocking violated net neutrality regulations and will no longer be allowed.
Thank god for net neutrality. We of course have that here in America right? Cus freedom!
> Montana Governor Steve Bullock (D) signed an executive order requiring internet service providers (ISPs) with state contracts to adhere to “net neutrality.” The state became the first to enact such measures in response to the Federal Communications Commission's (FCC) decision to repeal net neutrality rules last December.
but they could have. still could. and you might not even know. and the fact that one of our political teams has been fighting SO HARD against net neutrality is a big red flag.
protections are important even if someone's not actively and noticeably abusing people right this minute.
There's been exactly zero problems in 20 years (the idea was introduced in 2003), yet somehow "still could. and you might not even know.".
No one is fighting "so hard" this is a dead topic.
T-Mobile offers to zero rate YouTube if you let them throttle it - it's your choice. Would that be legal under Net Nutrality? Would they have to get permission from YouTube?
>No one is fighting "so hard" this is a dead topic.
AI was a pretty dead topic after the 80's boom as well. Hell, we can argue VR is still a "dead" topic, but we know sometime in our (millenial+) lifetime that we're going to have major landmark cases over VR/AR tech.
I'd rather patch loopholes before they get abused at this point.
> And despite not having net neutrality in the US not a single one of the fears came true.
That same set of scoundrels[1] that CF backtracked on defending keeps having routing issues (on top of their ddos issues).
Supposedly they (or someone related?) recently complained to a state AG under a state-level net neutrality rule. Not sure if it's been long enough to know if anything'll come of it.
These are BGP blocks, right? If Cloudflare's ISP doesn't have a route to the system in question, then it can't proxy the site anymore.
Maybe Cloudflare has an Ethernet cable between the server in question and itself, but that seems unlikely to me. There is probably some ISP in the middle that will play ball, right?
Cloudflare owns/leases fiber to form its own international network. A user's request hits a cloudflare server. if that server needs to contact the origin, it can exit from anywhere in cloudflare's network. If the origin is in the country which is wanting the block to happen they'd just ask the hosting provider to take the service down, otherwise CF would almost certainly be able to reach it from somewhere in their network.
You seem to be assuming that cloudflare's CDN is located in the country doing the blocking.
I'd imagine that CF's first act would be to remove any of the "offending" content from the country, at which point BGP blocking would no longer do anything.
I noticed earlier today that thepiratebay.org seems to be down for the first time in a while. It's just a cloudflare error page at the moment. Really curious what's going on there.
What I don't get is how DNS blocking and other DPI is legal while IP blocking isn't. Sure, you can make and argument that the ISP-provided DNS is a service they provide so you can compel them to not resolve certain addresses, but inspecting and modifying other DNS packets on the network seems no different to inspecting and modifying HTTP packets to inject crap or throttle streaming - the exact things net neutrality was meant to protect against.
Read the article or the sentence - the difference with DNS is that it's not as likely to result in blocking innocent third-parties. Services don't typically share DNS names, whereas IPs are typically shared or reused constantly.
I did, but that still doesn't explain how that relates to net neutrality. I guess the court might have used the likelihood of false positives as the factor that decides whether net neutrality or copyright law pervails in a given situation, but this isn't mentioned in the article.
I think it makes more sense if you consider the Directive is about "universal service and users’ rights relating to electronic communications networks and service", rather than "net neutrality" (which is a very ISP-oriented term).
> [Austria's] Telekom Control Commission found a violation of Article 3 Paragraph 3 of Regulation (EU) 2015/2120, because the IP access block poses the risk of ‘overblocking’ any website content.
Since the blocking was illegal per the European Union's regulations, presumably any other EU nation would have to enact similar decisions if this blocking was done elsewhere in the EU.
I always assumed businesses such as Cloudfare or Google would have a good reason to be the main IPFS or tor node hosts, since that can help improve the backbone infrastructure for this side of of the internet, and can also help catch bad actors if there are any there (easily turn in all logs to three letter agencies if they knock, or they can simply offer it up for anon tips).
It was started by the Navy way back in the '90s for secure communication online. Then the source was released like a year after being made public and development was funded by the EFF until the Tor Project itself was officially founded.
They still do (or did?) get grants from a few federal agencies though. But they had no strings attached. This is probably what you're thinking of.
Source: Hung out with a few Tor devs in Berlin a few years back. Including He Who Shall Not Be Named.
I think they meant it's believed that most/some exit nodes are run by US intelligence to spy on people. Not that development has anything to do with them now.
That would be incredibly wasteful. It can use many IPs per host when using geographically local endpoints, and relatively few customers are large enough to need IPs to themselves.
The EU engineers should just say "we followed the lead of Musk's X and we've taken the liberty of removing the IP/DNS blocking features from our network equipment. Sorry!"
IP address blocking may have been banned, but DPI to block domains based on SNI is alive and kickin'.
I've seen people talk about encrypted SNI for a very long time now and it's still not working; someone must have dropped the ball pretty hard regarding that
The evolution of ESNI into ECH has been slow but basically encryption of just the SNI would not play nicely with the rest of the protocol. So more had to be encrypted but that meant a deeper interaction to analyze.
It's also dependent on a new kind of DNS record and the original design wasn't great for DNS load balancing. Some tweaking had to happen there too.
The level of desperation the ISP’s engineers have felt in front of such incompetence must have been through the roof. I am getting tired of our politics here in europe: tech literate people in governments are put to work on surveilance stuff, never for actual policymaking.
The shit show continues.