
Malicious attack on Wikipedia – what we know and what we’re doing - app4soft
https://wikimediafoundation.org/news/2019/09/07/malicious-attack-on-wikipedia-what-we-know-and-what-were-doing/
======
jedieaston
Remember: there are BitTorrent links that the Wikimedia Foundation gives out
of SQL dumps of Wikipedia and the other projects. You can have a copy in case
this happens in your country:
[https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh...](https://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get_it.3F)

Also, the Kiwix project has a hotspot project that allows you to host ZIM
files (dumps of Wikipedia and other CC licensed content, like TED talks and
StackOverflow) on a Raspberry Pi, allowing you to share it with others. Setup
info here: [https://www.kiwix.org/en/downloads/kiwix-
hotspot/](https://www.kiwix.org/en/downloads/kiwix-hotspot/)

~~~
Avamander
I'd actually love to see a fully working IPFS fallback for wikipedia when
regular hosting doesn't work. Would it even be possible with ipfs?

~~~
buildbuildbuild
IPFS has a Wikipedia mirror but it is fairly out of date since it is dependent
on the Kiwix archive.

[https://github.com/ipfs/distributed-wikipedia-
mirror](https://github.com/ipfs/distributed-wikipedia-mirror)

------
judge2020
Someone claimed the attack on twitter with some details (DDoS) - and proved it
later by stopping the attack for x minutes then restarting it at a specific
time.
[https://twitter.com/fs0c131y/status/1170093562878472194?s=20](https://twitter.com/fs0c131y/status/1170093562878472194?s=20)
\- the attacker also went on to DDoS the twitch ingest servers (not twitch.tv
itself) knocking some big streamers offline.

~~~
dangxiaopin
It looks like a volumetric attack from this tweet. Wikipedia needs to use
Verisign BGP mitigation. They create GRE tunnels to your routers and are
capable of handling 2Tbps. During an attack, you make a BGP announcement and
the traffic goes via Verisign scrubbing/tunnels. No application changes are
required, no Matthew Prince selectively and benevolently enforcing CF
neutrality. It's used by large banks.

~~~
lkoolma
After working with a few large corporations and their DDoS protection
solutions, I did not have a good experience with Verisign, and they were not
able to handle attacks or get things working. However, I have great
experiences with Akamai and Cloudflare. I trust the people at Wikimedia will
choose wisely. I would I have learned that Verisign has one of the worst BGP
mitigation/scraping solutions out there. There are a few alternatives that
have more experience and provide much better uptime, include solutions from
Cloudflare and Akamai.

~~~
dangxiaopin
Any serious mitigation solution must be BGP based, not proxy. Besides its
technical merits and convenience, it also minimizes the risk of a benevolent
controller (e.g. Matthew Prince of Cloudflare) ruining your company, because
it becomes your upstream provider only during the attacks. Otherwise the GRE
tunnels are not in use. The IP addresses are still yours always.

We used Verisign for mitigation of a 44Gbps volumetric attack and it worked
very well. We also evaluated Neustar, but Verisign's infrastructure seemed to
be more robust.

~~~
snazz
A proxy is a perfectly acceptable “serious” solution for this type of problem,
as well as nearly all of the rest. Wikipedia is not the kind of website that
would warrant being removed from Cloudflare. What’s wrong with having an
upstream provider for caching close to the user and other features when you’re
not under attack?

~~~
SahAssar
> What’s wrong with having an upstream provider for caching close to the user
> and other features when you’re not under attack?

The problem is that you are basically mitm:ed all the time.

~~~
acdha
That’s not what MITM means. I get that you don’t like Cloudflare but voluntary
use of a CDN isn’t a MITM any more than, say, Amazon is a MITM because you
host on EC2.

~~~
SahAssar
Cloudflare is in between the client and the server, decrypting, rewriting and
(if set up right) re-encrypting the request/response. It masquerades as the
server by presenting a proper certificate for the domain even though it is not
the entity that is actually controlling the domain.

That to me sounds very much like MITM, although it is not a MITM _attack_
since the entity controlling the domain opted into it, so basically it is
voluntary MITM.

Using a VPS like EC2 is a different story since the decryption happens within
the layer that you control. Of course you need to make sure that you choose a
vendor for that layer that you trust, but on EC2 the traffic that amazon sees
is encrypted with keys they don't have and decrypted with keys stored on a
layer that I control. Amazon could read out the memory of my EC2 to get the
keys but their business depends on not doing so, so in this case either I have
a vendor that always will decrypt and read traffic (Cloudflare), or a vendor
whose business depends on hypothetically being able to but not doing it. There
is a clear difference to me.

That is the same for most CDN's (including CloudFront and all the other major
offerings), so I'm not trying to single out Cloudflare.

~~~
acdha
If you don’t trust Cloudflare, don’t use them but there’s no meaningful
security distinction between what they do and what AWS does: in both cases you
have a vendor with the capability of violating your security and a promise
that they won’t abuse that access.

This is why having a threat model is so important: it keeps you from wasting
effort on things which sound like security but aren’t actually changing
anything meaningful.

~~~
SahAssar
There is a security distinction, and this has been shown by for example
cloudbleed. Every step that has access to plaintext data is a potential attack
vector and might be logging/leaking information.

There has also been times where cloudflare (when setup improperly as I
mentioned in the previous comment) has misrepresented the security of a
connection, as shown by
[https://www.theregister.co.uk/2016/07/14/cloudflare_investig...](https://www.theregister.co.uk/2016/07/14/cloudflare_investigating_mystery_interception_of_site_traffic_across_india/)

------
softwaredoug
Just want to mention, WMF has a very small but elite team of engineers. Amazed
they maintain an Alexa top 5 site with many orders of magnitude less
engineering staff than Facebook or Reddit. I think they must count ~100
engineers?

I can't imagine what such a small team must be going through with a major DDOS
- wish them well in their efforts!

~~~
chillydawg
It's because they're just serving a big site, not running the world's most
sophisticated surveillance and ad serving machine. Serving giant websites
isn't all that hard if you're just spewing out SQL queries into html
templates. It all scales in all directions with a properly thought through
architecture.

~~~
blauditore
> Serving giant websites isn't all that hard if you're just spewing out SQL
> queries into html templates. It all scales in all directions with a properly
> thought through architecture.

No.

1\. Your comment makes it sound like Wikipedia is just, or mostly, serving
read-only content, which is far from true. Yes, static read-only content is
significantly easier to serve than dynamic, editable one, but Wikipedia is the
latter.

2\. Claiming it's easy to build something at this scale is "isn't all that
hard" just makes me think you've never done anything similar. It reminds me of
devs saying they could re-build MS Office over a weekend. It's just ignorant
of the software's actual complexity.

I'm not associated with Wikimedia in any way, but have worked on large-scale
software projects before, and things are quite different from, say, websites
only serving 100k monthly active users.

~~~
abraae
I've never heard anyone in my life say they could rebuild MS office in a
weekend.

What, in your opinion, would be the work needed to go from a 100k monthly
active user site to a wikipedia scale site - that would be comparable to
rebuilding MS office?

~~~
z3t4
The core parts of Office could be done on a weekend, but in order to get the
same complexity and incompatibility it would take several "codemonkeys"
several years to achieve.

~~~
journalctl
Silly Microsoft wasted hundreds of people and decades of time. Why didn’t you
tell them?

~~~
z3t4
Software projects are usually 90% done in 1% of total time taken. And if you
just solve the problem with duct tape eg. a shell script, like piping stdin to
a file, or contenteditable=true in HTML, you would have a very basic word
program, and if you take that route you will probably have the essential
features done over a weekend. But going from that to a full Office clone would
take years. The real challenge in development though is to solve real
problems, eg not make solutions looking for a problem, and not implement new
features ( implementing only features that solve real problems).

------
vortico
Just like trying to set your local public library on fire. There are always
crazies in the world.

~~~
dleslie
There was a string of arson attacks on little free libraries in Metro
Vancouver; eventually a pair of teenage boys were arrested.

I suspect that the sharing of knowledge and encouragement of developing wisdom
is, to some, a threatening prospect. Perhaps they have experienced learning
difficulties and are struggling with shame and frustration, or perhaps they
disagree strongly with the concept of an intellectually liberated population.
Libraries are, after all, a pillar of liberalism.

~~~
inimino
That's sad to hear, I always love coming across the little free libraries. I
would guess they were just bored and angry teens, likely not making any deeper
statement but just expressing their anger and frustration and willingness to
break the rules. Also, it's fun to watch things burn, and they come with
built-in kindling. Hopefully some judge will make them rebuild what they
burned, that would be fair and give them more appreciation for the work of
others that they destroyed thoughtlessly.

------
Theboda
Apparently this group is behind it. Also attacked WoW and twitch servers..

[https://twitter.com/ukdrillas](https://twitter.com/ukdrillas)

~~~
Nextgrid
Part of the liability should be shared with the people owning the compromised
machines these crazies are using for their attacks, otherwise attacks like
these will never stop as long as enough free “ammunition” is being left around
by incompetent people who can’t be bothered to secure & monitor their systems
properly.

Edit: in reply to some of the (valid) counter-arguments, I'd like to say that
there are indeed many issues that will need to be considered before passing
such a law - this is just an overall idea. In addition, my intent isn't to
punish the occasional kid doing something stupid and leaving a misconfigured
device, it's to punish companies selling/deploying obviously insecure devices
at a large scale, like ISPs deploying cheap shitty outdated network hardware
or the countless resellers white-labelling insecure network cameras. Currently
there is no penalty for manufacturing insecure hardware and this situation is
the consequence of that - I'd like to fix this problem. We have regulations
that (mostly successfully) prevent companies from selling hardware that blows
up and destroys your house, why can't we have the same for networked hardware?

~~~
throwamay1241
Can't wait to tell Gran she's legally liable for a DDoS because her unsecured
IOT washing machine best buy sold her caused the internet to cave in ;)

~~~
ohazi
Skip Gran and sue Best Buy and the IoT washing machine manufacturer.

~~~
BetaDeltaAlpha
There are exactly zero big box retailers or lobbyists that will abide that.

~~~
yorwba
Big box retailers seem to be able to comply with regulations mandating
physical safety. Digital security requirements could be enforced by a similar
system.

~~~
majewsky
Because "physical safety regulations" is something that the majority
understands, so it's hard to argue against that in public. With digital
security, most people lack the mental models to follow the discussion, so it's
really easy for lobbyists to tell them flatout lies about how those damn dems
are out to take their smart lightbulbs away from them.

------
jwildeboer
That post neither says what they know nor what they’re doing IMHO. I hope
there will be a more detailed post Morten soon.

~~~
jwilk
_post-mortem_

------
lolc
Note: The short blurb doesn't say anything about the nature of the attack.

------
ga-vu
These are the clowns, btw:
[https://twitter.com/UKDrillas/status/1170221000363061250](https://twitter.com/UKDrillas/status/1170221000363061250)

~~~
ineedasername
And... they've been booted from twitter.

------
jacekm
On a side note, I was surprised to learn, that Wikipedia does not have a
proper status page. status.wikimedia.org redirects to grafana dashboard and it
too was down yesterday.

~~~
lukejduncan
I wouldn’t expect a site to have a publicly available status page or anything
like public grafana boards. Isn’t that what HTTP error codes are for?

Can you share examples of where this is common?

~~~
detaro
HTTP status codes tell you that something is broken, but likely not details.
Or nothing at all, if all you get is a timeout because the service just got
DDoSed.

Many services and sites have them, a few random examples:

[https://status.flickr.net/](https://status.flickr.net/)

[https://www.vimeostatus.com/](https://www.vimeostatus.com/)

[https://3down.mit.edu/](https://3down.mit.edu/)

[https://www.githubstatus.com/](https://www.githubstatus.com/)

------
ineedasername
Does anyone know the nature of the attack? If specific countries were
targeted, or if it was widespread but only crashed some smaller sites with
less resources/redundancy?

------
irrational
Are there non-malicious attacks? Doesn't the word "attack" imply
maliciousness?

~~~
marble-drink
"Attack" is a technical term and doesn't imply malicious intent. Many attacks
are accidental or the result of negligence. Some are performed by researchers
in purpose but without malicious intent (e.g. the attacker wants to prove that
something is possible without actually doing any damage).

------
sametmax
I don't understand the goal though. What to gain ? Training for another big
target ?

~~~
throwamay1241
\- Advertising for potential DDoS service buyers

\- Bragging rights

\- Experimentation

Edit: Also potentially political or personal. Eg Posting something that
offends 8chan||nation states etc.

There's quite often blackmail involved (Pay us $x BTC and we go away).

Cloudfront or similar should offer DDoS protection for free as a gesture of
goodwill, it's good bragging rights for CF so everyone wins.

~~~
MayeulC
> Cloudfront or similar should offer DDoS protection for free as a gesture of
> goodwill, it's good bragging rights for CF so everyone wins.

Well, it is still a lot of wasted resources (bandwidth, energy, compute) for
everyone involved (ISP, CF, attacker, defender, compromised machines), so I
wouldn't be so quick to say that "everyone wins".

~~~
throwamay1241
Consider it a 'cost of doing business'. Everyone gets DDoSed, I'm pretty sure
that's one of CloudFronts primary service-based solutions

------
lemiffe
I appreciate them making this statement. I had issues accessing Wikipedia from
Turkey a week ago and I assumed censorship, but this week I have also had
issues accessing it from Poland and I started suspecting something was amiss.

------
Semaphor
Huh, so it wasn’t my network having issues. The weird thing was. That at the
same time google.com (though not google.de) stopped working. Weird
coincidence?

------
snazz
DDoS, I’m assuming? It was intermittently offline yesterday, which was
annoying, but I didn’t assume malice then.

------
ngcc_hk
Read through it. Any more info who is doing it. Not for accusation but
prevention.

------
cybersnowflake
Maybe the damage will randomly improve the political articles and knock down
some of the fiefdoms for a change. Couldn't get much worse...

