Guess with this I'll now find out, as crapping on Krebs' site is practically a right of passage when you've got a botnet now.
Today, I am happy to report that the site is back up — this time under Project Shield, a free program run by Google to help protect journalists from online censorship.
Google Ideas is 5 years old now. I'm not aware of it having started as a 20% project. [disclaimer: I work at Google, but not on Jigsaw.]
We were considering using Project Shield relatively recently (as in, some time this year), but at that time, the info about how it actually worked was... well, non-existing on their website. IIRC, their SSL support is also fairly new, which made it pretty unusable until very recently.
As another poster indicated, if you want someone to terminate SSL for you, you're going to need to hand them a key. We encrypt ours at rest, provide the same security and care to your secret material as we do Google properties, and as you can see with our Customer Supplied Encryption Key support for Persistent Disk and GCS, we care a lot about letting you control access to your data. If you don't mind me asking, to whom are you comfortable uploading your keys to?
Disclosure: I work on Google Cloud, so I'm actively trying to take your money in exchange for our services.
Didn't Cloudflare invent Keyless SSL to solve this problem? https://www.cloudflare.com/keyless-ssl/
> Note: Keyless SSL requires that CloudFlare decrypt, inspect and re-encrypt traffic for transmission back to a customer’s origin.
That's not particularly different (to me), but I have a different threat model. Again, it comes down to what scenarios you care about and what you're comfortable with in exchange for <something>.
Even initiating tons of sessions is likely to mean that the key server is going to be busy. But if you're really concerned with sharing you key with us, I agree CloudFlare's Keyless SSL provides a real service that does a lot for you without handing the key over explicitly (you just have to keep doing your part).
I don't see how they'd be able to effectively stop Layer 7 attacks without being able to see unencrypted requests.
Project Shield welcomes applications from websites serving news, human rights, or elections monitoring content. We do not provide service to other types of content, including gaming, businesses, or individual blogs.
I guess Brian’s a reasonable exception.
Disclosure: I work on Cloud, but I'm not a networking expert.
You can block the attack based on IP, request headers, rate control, geo, etc. The problem with some ddos attacks is the size of the request body. That's what usually overwhelms a server.
Ddos attacks do knock out the servers, but since the requests are distributed, then you don't see that. Because so many servers are getting knocked out, then real paying customers can't serve their content. I'm guessing that's why they pulled the plug.
Google's infrastructure is so large that knocking out a few servers won't impact much, they just spin them back up.
Krebs' article describes two attack methods, fake HTTP requests and a simple volumetric flood of GRE packets. L7 mitigation clearly isn't part of GCP's service. The GRE packets should be a different story -- if the customer isn't using GRE, they should be able to drop this traffic easily using the GCE firewall or GCE load balancer.
The vulnerability report at http://www.securityfocus.com/archive/1/532239 suggests the GCE firewall is, or at least was as of May 2014, ineffective against such an attack (even if your firewall rule set will drop the problem traffic). But it also says the GCE load balancers do not have the same problem. Krebs didn't mention how much of the 600+ Gbps was GRE, but let's say a GCE load balancer was configured to pass through only TCP/80 and received 100 Gbps of non-TCP traffic. Would the load balancer function as intended and drop all of this with no interruption to legitimate TCP traffic, would Google null route you, or something else?
We can get all the reassurance from an account manager that they'd never stop our service. Real life doesn't always play out that way though...
We generally don't talk about DoS attacks at Google (sorry). But I can say we're familiar with them, and that Cloud uses the same front end and networking infrastructure as the rest of Google here. We do understand (per project, the unit of billing) if there's tons of ingress/egress from a single project, and your intuition is right: it would be wise to respond to "Hey, is this 1 Tbps of traffic legitimate?" quickly ;).
Disclosure: I'm an engineer in Cloud (mostly GCE), but since I'm not an account manager I refuse to reassure you ;).
I'm a game server operator. I'm not currently running anything, but I probably will in the future. We used to receive volumetric L3/L4 floods all the time from booters. The good thing is these were always just dumb kids, not sophisticated attackers, so they would be easy to filter with a simple ACL (often they were targeting some random port we don't use, so a simple rule set of "accept whatever UDP and/or TCP ports we use, drop all else" would be effective). There were a few providers who would set up these ACLs, but they were only effective up to a few Gbps and would null route past that.
If I get back into this, I will give GCE a try with GCLB in front for, effectively, ACLs. In my experience it's a near certainty that any reasonably popular game server will be attacked (very often as retaliation for banning someone for cheating). I'd imagine your mitigation capabilities for a small-time game server customer paying a few hundred a month would be different than a large corporate client paying 5 figures, but I'd still like to try GCE for this as it seems like the best bet (besides OVH).
Presumably these giants can easily handle such traffic as long as someone is willing to pay for the privilege? 665gbps seems tiny in comparison to the capacity someone like Google might have at its disposal but I'm speculating as I haven't seen anything detailing their network stats.
To give something of a concluding statement to this waffle I guess I have respect for Google in running this public service type protection for sites that have a strong enough 'public good' element.
So yes, 665 Gbps is well within our network capability.
Disclaimer: while I work on Google Cloud, I'm no networking expert.
Reading between the lines of what you said it's clear just how big your network could be!
Disclosure: I work on Google Cloud.
Today it's different, not only do they have a massive network, but they probably also have firewalls which drop the attacks from the start.
Stuff is going to get even worse when IoT devices begin using IPv6. By design, devices are publically reachable and not hidden behind a NAT router, which makes RCE exploits way, way easier.
I'd take a guess that loads of IoT devices have "backdoors" like open SSH/telnet with insecure default passwords, too - the same shit that hit el-cheapo routers, for example.
Is that true? I just setup IPv6 at home yesterday and I don't see the difference from IPv4 in terms of reachability. The default policy on my firewall for incoming traffic for both IPv4 and IPv6 is drop.
Yes, NAT can give you a pseudo-firewall in that LAN devices aren't given publicly routable addresses, but I have no idea why anyone would leave their IPv6 network completely open "by design".
IoT devices like surveillance cams, VoIP babyphones etc. have two options:
1) depend on third-party servers for operation, so no outside-to-device-initiated communication is needed. Downside: costs money to run the servers, and just imagine the sausagefest when a camera cloud server gets hacked. Upside: you can firewall off your private network as you like, and the device will still work.
2) allow the owner of the device to directly connect to the IoT device via IPv6. Downside: everyone and their dog can access (and exploit) your devices, provided that they know the IP address. IPv6 addresses are long and pretty random, but e.g. if the MAC address is used for assigning the IPv6 address, not so random any more. Upside: no SPOF/centralized node that turns your device into a brick if the operator shuts down.
Basically, the only way to "securely" operate IoT devices is inside a separate VLAN. FritzBox routers can do this, but not many others, e.g. because the router can't independently manage the Ethernet ports, because it's cheaper to put in a GBit switch than to choose a SoC that's beefy enough to drive four GBit ports individually. I put "securely" into quotation marks, as a hacked surveillance cam or babyphone is an open invitation to any hacker.
Not sure that word means what you think it means.
This is a major pain to configure, and way beyond the capacity of even most IT professionals. One of the problems is that you do want to allow some devices (phones) from the 'normal' network to (selectively) be able to connect to devices in the IoT vlan.
I spend a solid day trying to set this up once (and on a 'real' switch, not a Fritzbox which I have too but only use as modem) and I'm not saying that I'm that good a networking guy (I mean, that I wasn't able to it working means I'm not) but I do know more than the average internet installation guy who would be the only hope for 'regular' users to set up their networks properly.
Some things just do not make much sense without being able to control them remotely. A separate VLAN for devices would only allow them to communicate with another which probably is not what you want. BTW. i could not set up VLANs on a fritz box with FritzOS last time i tried. However nearly all routers with OpenWRT will do without a problem.
Although this wont be practical for any device that is not a phone or a (desktop) computer.
Unfortunately, this belies the meat of the issue with IoT. After you've bought the router, there's no reason for your router's manufacturer to keep updating the firmware and fix bugs that allow an XSS attack, and even if the manufacturer does upgrade the firmware, there's no no way for the manufacture to force the firmware to be updated on all of the devices that have been sold, hence installed devices with older firmware that's been exploited and is now part of the botnet used to attack Krebsonsecurity.com
Separating the IoT stuff (at least the stuff that needs connectivity from outside) into its own VLAN at least prevents a hacker from gaining access to the rest of your home network (e.g. NAS devices or your normal computers).
While this would as you said protect my other networks to some extend it does not solve the problem. I will not ever trust these devices to get secure enough to expose them directly to the internet.
This is why i am thinking the only way to operate these devices securely is to require some kind of transport protection in form of an IPsec tunnel for example. This would allow me and anyone with the right set of access to control the devices without making them accessible to anyone else.
If home routers would encourage the use of such tunnels it would be a normal thing to have a link back to your home network (or maybe a separate IoT network) which could be properly firewalled...
Also, you still have to receive the traffic even if you then ignore it; if your pipe is smaller than the attacker's, it may be enough to overwhelm you.
Provides the context if a little light on technical details.
It isn't for me. I get "This site can’t be reached - krebsonsecurity.com refused to connect."
$ dig krebsonsecurity.com +short
$ dig ns krebsonsecurity.com +short
However if you query the Whois, you'll see that the site changed its name servers from Prolexic's to Google's ones today:
$ whois krebsonsecurity.com | grep "Name Server"
Name Server: NS-CLOUD-D1.GOOGLEDOMAINS.COM
Name Server: NS-CLOUD-D2.GOOGLEDOMAINS.COM
Name Server: NS-CLOUD-D3.GOOGLEDOMAINS.COM
Name Server: NS-CLOUD-D4.GOOGLEDOMAINS.COM
$ dig krebsonsecurity.com @ns-cloud-d1.googledomains.com +short
# echo "188.8.131.52 krebsonsecurity.com" >> /etc/hosts
Now browse to krebsonsecurity.com et voilà!
Also many ISP's servers ignore TTLs for certain changes that happen infrequently (like NS changes) and wait a few days. So they might refresh the A record according to the TTL, but they'll still be pulling it from the wrong name server.
As to your point about "propagation", argumentum ad populum applies.
The process of spreading to a larger area or greater number; dissemination.
72 hours is indeed a comfortable upper bound ("up to"), that we would give to customers so as to make absolutely sure every cache down the chain has refreshed its records, and is not the cause of the issue at hand when name servers were changed. 24 hours is common delay.
Perhaps it is a bit of a misnomer to use it for cache timeouts, but it isn't particularly wrong or confusing unless you have a very pedantic reader.
Am I being a bit pedantic? Sure, I'll admit that. But the reason I'm being pedantic is that this mischaracterization implies that there's nothing that can be done to prevent it. That is absolutely not true and if DNS is handled properly, you can get 99.99% or more of the world resolving with the correct records in under 1hr, with most of that being in under 15min. Excusing this as DNS propagation and not handling just looks lazy to me.
But what do I know, based on all the down votes apparently folks disagree with me so I'll gladly bow out from the conversation.
$ dig www.krebsonsecurity.com | grep IN
;www.krebsonsecurity.com. IN A
www.krebsonsecurity.com. 267 IN A 127.0.0.1
Edit: works here now
1) Put the site behind CloudFlare.
2) Wait for an attack...
3) Force all users to go through a capcha before accessing the site.
Note: The capcha setting can be enabled with 3 clicks in cloudflare UI and it takes 2-5 minutes to propagate. (Yes, I speak from experience)
Cloudflare, in their defence, say they don't censor/check/approve sites and that's a good thing - after all, sites like wikileaks should be allowed protection.
sometimes you can get the HTML to load from :80 but the CSS breaks. HTTPS doesn't seem to work at all.
At 650 Gbps (81.25 GB/s) he's looking at $1.625/sec ($5850/hr) in cache egress fees alone.
I would go CloudFlare, which is flat rate.
Disclosure: I work on Google Cloud, so I'm after your money (but not via having you pay for egress to DDoSers).
"Project Shield is a free service that uses Google technology to protect news sites and free expression from DDoS attacks on the web."
$ nslookup krebsonsecurity.com 184.108.40.206