> Although terminating the encryption was necessary to inspect the traffic and effectively mitigate the attack, the use of HTTP Pipelining required Google to complete relatively few TLS handshakes.
Ouch. I mean, congrats for allowing attack traffic to reach the servers, but you know what... once a request is recognized as "bad", then there are hight chances that other requests on the same http connection will also be bad perhaps?
This is exactly, why a more appropriate mitigation is to issue connection-close (in http1.1 world) or GOAWAY frame (in http2 world).
With that in hand, it's possible to bounce the connection back to the bot. Sure, the bot will retry which will put pressure on TLS. But then, if a source IP is trying very hard to establish many connections it's another strong sign of malicious traffic. In TCP you can rate limit SYN packets and _not_ allow more than X new connections per second, quite easily. Another mitigation is to allow the bot to send http requests, but just not answer them. Allow them hang on the connection, burning the bot's resources (memory, concurrent connetions).
Anyway, the point is: requests-per-second is not a perfect metric for L7 / application attacks. A smart application shoudl be albe to push mitigation up to lower levels (like syn rate limiting, or TLS client-hello signature rate limiting), and _not_ intake large amounts of traffic. A better metric is bot count (or number of unique IP's seen), which was shared in the article and is in mid-range.
To recap, a well designed system should _never_ need to handle 47MRPS. It should be able to packpressure it.
Google doesn't terminate TLS on "the servers". It gets terminated on the frontend load balancers (GFE), which is presumably where the attack mitigation happened.
I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules, but for their network vs an appliance.
Sadly no, would have loved to, but had to go through the server host which makes sense as they are the direct customer (albeit maybe not as well invested in the specifics or diligent). I've used MT with 2 different server hosts (one an early partner of yours that moved to Path, and more recently one that still uses MT). Early days, MT had a few leaks (IoT device attacks), which I'm happy to say seems to be patched up so kudos to your team. This intermittent random UDP issue is more recent while I was trying the second host, but couldn't notice a pattern or replicate, so I ended up rolling my own with tc, port knocking, and the largest pipe we could find.
Edit: For others, our problem was probably an edge case, and our use is probably not common. MT seems good compared to the alternative solutions I know so looking forward for an opp to try this if I have ever a need.
Useful contribution, defending a cesspit of hate like Kiwifarm. Or Stormfront. Cloudflares stance is extremely admirable but there should always be a limit. Things like Kiwifarm don't need protection.
Infrastructure platforms shouldn't be the deciders of what gets hosted online. Sure, those sites aren't nice places. But I'd much rather have governments take action on them if they are illegal than have CEOs in mega corps decide in combination with the court of twitter.
Services like domain name providers and ddos protection are so essential for putting content online that they should be regulated as utilities. It would be absurd for a power or water company to cut service because they decided something legal but undesirable was happening at a building just because enough people tweeted at them.
There are some services that should be allowed to pick and choose what to provide. A forum like HN or reddit for example is perfectly justified removing whatever they want. If you don't like it, go somewhere else. But once you get to the point where your domain name and phone number are getting canceled, its no longer possible.
The current playbook for when you want to take a site down is to sign up to it and post something illegal like a bomb threat. Then immediately screenshot it and post to Twitter. Have all of your followers retweet the screenshot to the cloudflare twitter account and the company will take down the site quickly.
Doesn’t matter how quickly the sites moderators take your post down. Small site owners are expected to react within seconds while big tech is allowed days.
Sigh. You don't have to use this. Large companies that want super tailored DDoS protection can. Everyone else can just use our standard DDoS protection which is included with every plan and works: https://blog.cloudflare.com/26m-rps-ddos/
Not commenting on this particular case, but I know in smaller dev shops that time spent working on one feature is directly taken away from time spent on other things, and so if the new feature isn't useful to you, it has also taken fixing and dev time away from ones that might be. I also suspect that many people have seen the trajectory where apps get more and more features and don't focus on their core competency.
Again, I don't believe this is the case here, but it might generally explain the phenomena/misplaced snark.
Of course I'm not using it myself. But the other half of that problem is that this will inevitably be another way that cloudflare breaks my experience as a user and I can't opt out of that.
I am running quite a different browser setup from most regular web users.
I am using Firefox with multi-account containers and uBlock Origin enabled, and I also have an OpenVPN client running, the amount of captchas and distrust I receive from Cloudflare (and their customers, unknowingly I suppose) feels disproportionate.
It's funny to me that these companies most likely spends a lot of time and energy on how to optimize their websites and purchase flows, and they might not know how often Cloudflare puts up barriers and destroy their hard work.
I see a lot of sites (including those behind Cloudflare) being blocked when using Cloudfare's Warp VPN. 403s, captchas, etc.
It would be nice if Cloudflare could put abuse control measures on the front-side, and stop the abuse before allowing it to exit their VPN. E.g., if Cloudflare detects abuse with their existing anti-DDoS, anti-scraping, etc. measures, and the client is on Warp, instead of creating a poor experience for every other Warp user sharing that exit IP, just block/aggressively throttle the bad actor's traffic at the VPN terminator.
They have struggled with credential stuffing, captchas were the only way to stop our login pages from being bombarded at one time. They are getting better with it though but I would rather not lose a tool of last resort.
Other services made GA today, at the time of me writing this post:
- Cloudflare One [1]
- Cloudflare WAF [2]
- Domain Scoped Roles [3]
[0] https://blog.cloudflare.com/welcome-to-ga-week/
[1] https://blog.cloudflare.com/cloudforce-one-is-now-ga/
[2] https://blog.cloudflare.com/account-waf/
[3] https://blog.cloudflare.com/domain-scoped-roles-ga/