Today may be a good day for you to donate to the Internet Archive. If you rely on the Wayback Machine or enjoy the game archives or simply appreciate the amazing preservation work the non-profit is doing, kick them a few bucks.
Naturally their own donate page is offline right now, but you can donate yourself via check or wait for the site to come back.
Internet Archive
300 Funston Avenue
San Francisco, CA 94118
I had the privilege and joy of meeting several of the team at the Internet Archive who actively police for this kind of content as best they can, while maintaining the openness of their service. They're cheerful, honest, hardworking and energetic people. There's few groups I'd feel so happy to support.
Maintaining an open and freedom-loving service in the face of abuses is incredibly hard. It's an incredible disgrace how badly these attackers have misunderstood their targeting.
Another comment from them: "Site is down for hosting a mayor [sic] amount of ISIS stuff."[1] They also tweeted about bringing down several other supposedly ISIS-related sites. I'm guessing they're not aware of what the Internet Archive does...
Why should it have to delete it in the first place? There's just so much wrong with that reasoning, it's hard to know where to start.
If Archive.org deletes it, it will simply be uploaded elsewhere. You cannot prevent uploads internet-wide. It's simply not possible.
And why are you even trying to remove it at all? Why are you not investing effort into educating people on why it is propaganda, and why it is false? Your real problem isn't that the materials exist, it's that people believe them!
With these two problems combined, it's not that hard to understand that by removing the materials, you lose your chance at actual damage control. You just push it deeper into the darker corners of the internet, where you have no ability to educate users at all, and where you completely lose control over it.
You cannot prevent this, but you can and should make it harder.
There is a certain group of things that should be hard to share (not to mention illegal, but execution videos and calls to kill infidels are already illegal to share), and archive makes it easier to share.
You can and should educate people, of course, but you should make the propaganda (again, all the war videos, all the execution videos, all the cut heads and shot disbelievers) harder to get.
The widely shared IS propaganda has been cited as a factor why ISIS is so popular lately. Why so many people are radicalising. The lenient approach archive.org is having to this is helping them and they have big role in destabilizing the current world, if only indirectly.
Yes they are deleting some of the videos but don't delete copies that are uploaded seconds later with the same names. YouTube and others are much more proactive with deleting jihadi propaganda. Archive.org is jihadi youtube.
I don't like it personally. But yeah DDoS is not the solution I guess
Well I'm glad you have the objective viewpoint on which kinds of propaganda are OK and which aren't.
If ISIS recruiting materials shouldn't be hosted, then neither should military recruiting materials from any other nation or political group.
Of course, the real solution here is to not try to police political speech on the internet and let people decide what they believe for themselves. If people are deciding to join ISIS, our job is to communicate why we think ISIS is wrong, not to subject the internet to moral censorship.
Have you actually read the comment you're responding to? You cannot both remove the content and educate users. You have to pick one of the two, and "removing the content" isn't going to solve the problem.
You are completely missing the point, and just repeating what you said before.
EDIT:
> The widely shared IS propaganda has been cited as a factor why ISIS is so popular lately. Why so many people are radicalising. The lenient approach archive.org is having to this is helping them and they have big role in destabilizing the current world, if only indirectly.
This is horseshit. As I've already said, the problem isn't that the propaganda exists. It's that people unquestioningly believe it.
I'm sure that it "has been cited as a factor", because that's a far easier way to look like you're dealing with the problem, than to actually try and solve it. That doesn't make it correct in the slightest.
This might be a good opportunity to mention IA.BAK[1], an effort to back up the Internet Archive. Currently, the entire Internet Archive is stored in one physical space. IA.BAK allows users to store copies of the archive which can be used as a source for data recovery in case of a disaster at this location.
Good point. Future versions of the project could theoretically work with encrypted blobs, with the keys (which are significantly easier to keep in multiple physical locations) only known to the Internet Archive. This would probably require some architectural changes to the Archive and might be harder to scale, plus one of the goals of IA.BAK is to not be dependent on any Internet Archive resources other than the actual files.
Definitely something to think about down the road, but in the meantime there are plenty of public datasets to back up.
DDoS is making it so small players (and very large ones like the IA) can't run web sites anymore. It's time to stop bullshitting this problem and pretending it's going to go away or that you can sign up for a magic cheap plan somewhere and all the problems stop happening. We really, really need to start getting active about solving this problem.
That means building DDoS protection into the nature of the web and all hosting by default (AWS and GCS I'm looking at you), and that means actively prosecuting DDoS attackers.
That also means, if you're a cloud DDoS provider, not helping to protect the sites where these attacks are marketed.
> That means building DDoS protection into the nature of the web
No, you'd have to build it into the nature of the internet, specifically into all the users of insecure machines and all the builders of core routers. One good thing that could be done is to have an organized response to any DDOS and to make it so easy to trigger this response that victims will lose very little in terms of downtime.
Commercial responses to DDOS (CloudFlare for instance) are useful, but only as a stop-gap measure, essentially the fact that there is a business opportunity here documents a huge flaw in the way the internet currently works.
Remember that hosting providers are as much (or even more) on the receiving end of these (for instance: when a customer of a hosting provider gets DDOS'd all the other customers will suffer too), and signaling they are part of the problem is akin to victim blaming.
It's not victim blaming to insist that supercorporations like Amazon and Google provide the same proactive DDoS mitigation services that many other providers do. If OVH can do it, they can too. OVH can't even figure out how to do recurring billing properly and they can do it.
It is victim blaming when people tell you it's your fault you're getting DDoSed because you can't afford several thousand dollar DDoS mitigation, it is victim blaming when someone's server simply gets null routed because they're being attacked, and it is victim blaming is when AWS hands Greatfire a crazy bandwidth bill because China attacked them.
Even if it didn't, they're at least trying. That's a much better policy than burying your head in the sand and pretending this doesn't exist like AWS and GCS do. Seriously, try to figure out what their policies are on DDoS, they don't even mention it anywhere.
Their active ignorance on this issue combined with their astronomical egress prices make them dangerous to use for public facing web sites of any serious caliber, as I've stressed multiple times on HN. I'd consider using them for behind-the-scenes stuff, but for front-facing sites? What's going to happen to your site when you get a 100GBit attack? Are you going to get charged for the bandwidth? Is your site going to get null routed? Zero information provided. Good luck.
I've gotta say, I just read through the OVH DDOS mitigation docs, and for me the upshot is that if my servers were targeted, I'm in for an unspecified increase in latency, which for a rather large cross-section of use cases is one and the same as the service being unavailable due to the direct action of the DDOS (which is kinda still what would be happening).
It's also not something you can disable; it's mandatory, so as to not impact other customers. I (my employer) use AWS pretty heavily, and we've never noticed our ability to access our resources being impaired by a DDOS on other nodes, so... is it not rather obvious that AWS (and in general, the set of "cloud providers without constant outages on their status pages") have basically the same setup - network is rationed, a given resource will only be allowed so much of the pipe, and your service is unavailable until and unless you have the capacity to handle it. It's not as if it could be any other way.
The other danger seems to be in unexpected bills. Does this happen to people? Are there companies going bankrupt because they didn't notice a DDOS for a couple weeks? I've had AWS support reach out to me several times - without ever having had a paid support subscription - when my costs have gone up, just in the course of adding new functionality and expanding capacity. It's not like you set up one instance and can start egressing petabytes.
AWS does monitor their infrastructure. All accounts have limits from the get-go; you can't run up a bankruptcy-inducing bill without your phone ringing. And in all cases, your exposure below that level is rather directly under your control.
If I happen to have a use case that involves a bunch of random hosts sending me a lot of data, I expect to be able to pay the bill for the bandwidth and the capacity I provisioned to handle it. Or, if I didn't, I max out the bandwidth reserved for my resources and I don't end up seeing all the traffic. I don't see why it'd be helpful (or even meaningful) to differentiate between DDOS vs. legitimate traffic. Wouldn't a published policy on any cost or functionality difference when a DDOS is occurring provide a malicious actor with information they would need to avoid triggering that clause, so as to maximize the damage they're able to cause to your business?
I guess I'll likely change my tune if and when I'm targeted and have to deal with a DDOS (and this sentence is why i went to the throwaway, which is a shame; I'd be happy to put my name to the actual content of this post. But fate is not to be tempted.). In the meantime, though, I'm skeptical of the need, and even the desirability, of service-provider policies that have anything to do classifying the "nature" of traffic flows. It just seems completely out-of-scope for them, and in-scope for myself as the person doing the provisioning.
It works reasonably well against typical DDoS attacks. I've read that booters have started to show warnings that they're ineffective when used against IPs belonging to OVH. It probably doesn't do much against resource exhaustion attacks at the application layer (that's more of a job for WAFs/rate limiting, anyway.)
How do you do that? The nature of DDOS is to send multiple requests from multiple locations. It's not easy to counter this if you can't scale up as big as the DDOS is. If you start banning IPs, you are going to also ban legitimate users. (search for "tor and cloudflare")
DDoS attacks are not the result of millions of people individually attacking a site, they are almost always the result of small criminal organizations that are controlling botnets that do the attack on compromised systems for-profit. They offer these attack services via market web sites. It's highly likely that this person used one of these markets to execute the attack against the Internet Archive.
It's not easy to go after them, but neither is it easy to go after CP peddlers, and yet we still go after them and arrests are made. It's one part of a many-faceted strategy that will be needed to get some control over a problem that is clearly getting worse.
Nope. Archive.org has a part where you can upload videos, it is separate from web archive.
ISIS uses archive.org almost exclusively for sharing their propaganda videos (all the executions, war videos, etc.). It's not the web archive they are using for this, it's the video archives.
I am not sure why are they doing that; probably because the barrier for using it is low and they don't delete the ISIS propaganda so much as other websites (like youtube).
Other website jihadists love to use is justpaste.it for text
To you who are encouraging a proliferation of ignorance and punishing the innocent many for the crimes of a guilty few: this is their goal and you are helping to realize it.
Most of these services have some kind of fair-use policy. Nothing that you would notice as a small to medium-sized user, but if you push the kind of bandwidth that I imagine the Internet Archive would, or if you're suffering from a large-scale, targeted DDoS, I'm pretty sure you'd be "encouraged" to switch to the Enterprise plan, which starts at $3k/month if the numbers posted in various discussions on this topic can be trusted.
True, good point. In their case, Sucuri actually offered to help for free via twitter, which is nice. CloudFlare has project Galileo, which they could apply for and get free DDoS help.
The importance of the Internet Archive is such that we can not allow it to ever go down.
I see a lot of reasons not to use these providers, but privacy and integrity is not one (and I find it a bad argument against them).
Every traffic passes through so many hops, routers and networks that adding a secure, well tested and privacy-paranoid provider, like CloudFlare will not affect the privacy or integrity of your connection.
Yes, they have to SSL terminate your connection, but does so every proxy that they decide to use (nginx, elb, whatever). Plus, the security that a company like CloudFlare has to protect that data is a lot higher and more strict than the majority of sites out there (including the Internet Archive).
It is also frustrating that people would downvote me above for suggesting to use them, when in fact it is the only way to handle a large scale DDoS uness you have an insane pipe with hundreds of G of uplink.
Having access to the plaintext traffic of 5% of all web sites on the internet makes them a huge target for anyone from hackers to nation states/intelligence agencies and what not. I have a lot of faith in their security team, but this level of concentration is not healthy for the internet as a whole.
There's a huge difference between having access to the plaintext of encrypted connections and merely seeing connection metadata, as would be the case with "traditional" DDoS mitigation services that do not act as a proxy. Reverse proxies are certainly not the only way to handle large-scale DDoS attacks, though I would agree that, leaving aside the privacy concerns, those services are hardly competitive in comparison to CloudFlare, especially on price and UX.
CloudFlare doesn't provide mitigation for $20/month. Unless you sign up for their Business plan at $200/month (or even Enterprise, in some cases), you won't get mitigation worth crap.
A lot of people mistakenly think that CloudFlare provides "real" DDoS mitigation, but that's just not true. In fact, I'd wager that CloudFlare has mostly outlived its utility nowadays, and is primarily a risk rather than an asset - for it to work, you must compromise your users' privacy, unlike competing mitigation services.
"Saving bandwidth" is hardly a valid argument anymore either. How much is your users' privacy worth to you? Traffic pretty much costs between $0.50 and $5.00 per TB nowadays -- would you essentially break TLS for everybody to save a few bucks a month?
As for Sucuri - only their Business plan covers Layer 3/4 attacks, and it's not entirely clear to me what they mean with "site". Is it a single subdomain?
If you want actual, real DDoS mitigation, then just pay a real mitigation provider like Voxility, X4B, Level3 (formerly Black Lotus), and so on. There are quite a few.
EDIT: Yep, Sucuri classes a "site" as a "unique FQDN", more or less. It's not clear what the bandwidth/traffic limitations are. It also seems to me like it only does HTTP-based traffic, much like CloudFlare.
1TB is between $90 and $250 on AWS. Azure and GCP are even worse. However, I don't think that this is a huge issue, as attacks generating large amounts of outbound traffic charges can usually be mitigated with some effort.
That depends on what you expect to get out of them.
* If you're looking for DDoS mitigation, use basically any of the providers that offer it on a network level - it's really not necessary to have access to HTTP traffic to mitigate attacks. Also, don't run Apache - it's notoriously vulnerable to various low-bandwidth attacks.
A non-exhaustive list of "real" mitigation providers (note that I am not making any particular recommendations):
- Voxility
- X4B.net
- Akamai (formerly Prolexic)
- Level3 (formerly Black Lotus)
- Psychz Networks
- CNServers
- Sharktech
- OVH (not as a separate service, but their entire network is covered)
You'll want to avoid anything HTTP-specific (as it will be prone to the same privacy issues as CloudFlare), and opt for layer 3/4 mitigation only.
Another option, if you're running at a larger scale, is to purchase mitigation appliances and just set up your own mitigation infrastructure. This will not be cheap and require some serious connectivity, but beyond a certain point it'll be more cost-effective.
* If you're looking for a WAF, run one on your own backend server(s) and/or loadbalancer(s). There's no benefit to doing this remotely, really. Even something relatively simple like ModSecurity will cover a wide array of problems.
* If you're looking to save bandwidth: don't bother. Traffic costs virtually nothing nowadays, and saving a few dollars by having another provider cache your assets hardly outweighs having the privacy (and potentially security) of all your users compromised. If you find traffic to be expensive, you should probably look for a different provider - some providers (like AWS) notoriously overcharge for it.
* If you're looking for better performance: CloudFlare doesn't really provide that to begin with. There's an extra hop for non-static assets, and depending on the location of your server and users, it can actually slow things down. If your performance is really critical to the millisecond - and chances are, it isn't - look into hosting providers that offer anycast.
* If you're looking for DNS hosting: plenty of options. Many providers offer it for free if you host with them, Hurricane Electric offers it for free regardless of where you are hosted (http://dns.he.net/), and if you need an SLA, there's Rage4 and Route 53. Pretty much every DNS hosting provider uses anycast.
* If you're looking for magic SSL/TLS: Use Caddy (https://caddyserver.com/), which is a HTTPd that will automatically set up and renew certificates for you through Let's Encrypt, as well as greatly simplifying TLS configuration. It's essentially zero-effort.
Trying to outsource this to a third party (like CloudFlare's "Universal SSL/TLS" does) defeats the point - it means that the third party can see all of your traffic, all the while providing a false sense of security to your users; they see the padlock, but their traffic is not secured end-to-end.
In short: the only correct place to terminate TLS is on your own servers.
* If you're looking for something else that CloudFlare offers: Feel free to describe it, and I'll suggest an alternative.
Most of these providers you listed have many issues, more expensive and do a terrible job protecting web sites.
For websites, like the Internet Archive, you do not need a layer 3/4 mitigation only provider. CloudFlare, Incapsula, Sucuri and others do that very well (I know from experience by working with them).
Well, I am still looking around so no fixed ideas yet whether to use or not use whatever CF offers, but since in my experience CF is the one that stands up the most in whatever stuff I read, I thought it would be very nice to have alternatives for research.
I'm guessing it's never come up before, because why would anyone ever DDoS the Internet Archive? They have no ideology, make no political statements, back no movements. They have no real money to extort.
I suppose the lesson here is "never underestimate the stupidity and destructive power of wannabe vigilantes."
When things don't make sense, often it is because information is missing and/or misdirection is in play.
For example -
This could be a cover for an operation that Internet Archive is not aware of, that is possibly tied to the Internet Archive truck theft (and recovery) that happened a few months back.
The truck's contents (drives, network gear, h/w's firmware, etc) was the way to get a payload into the internal network. And the DDoS attack is now covering whatever is really happening with lots of noise.
This is pure speculation on my part, but is much more reasonable than the pretense that someone would be stupid enough to DDoS the Way Back Machine because a few sites it auto-archived had some links to terrorists. In combination with someone also being stupid enough to steal a flaming truck (https://blog.archive.org/2016/04/20/truck-and-back-again-the...).
I think you're vastly underestimating the arrogance and stupidity of internet vigilantes. Remember when Reddit fingered some random dude for the Boston Marathon bombing, leading to massive harassment of the family of a guy who, it turned out, had killed himself a month previous?
Naturally their own donate page is offline right now, but you can donate yourself via check or wait for the site to come back.
Internet Archive 300 Funston Avenue San Francisco, CA 94118
TIN: 94-3242767
https://archive.org/donate/ http://archive.is/UBX0Z#selection-2697.1-2701.24