
Bringing Down Millions of WordPress Sites - jermaustin1
http://jeremyaboyd.com/that-time-i-brought-down-millions-of-wordpress-sites/
======
joshfraser
Back in 1999 when I was learning to code (PHP obviously), I was stoked to
discover the mail() function. Up to that point I'd only been doing HTML and
JavaScript coding so my newfound ability to send emails was a pretty big deal.
I asked my friend what would happen if I put the mail call inside a
while(true) statement. He said it would run until the server hit the 30 second
timeout. I asked how many emails my server could send in 30 seconds but he
didn't know. I decided to to find out, directing all the emails to my hotmail
account. Little did I know, that my ISP & hosting provider had disabled the
default 30-second timeout so the script kept running. A few hundred thousand
emails later I was having second thoughts about my poorly planned experiment.
The next day I got a phone call from my ISP. Turns out Microsoft perceived it
as a DDOS attack and blocked all traffic from my ISP. And so for a day
basically no one in my city could access their Hotmail accounts or any other
Microsoft properties. Oops.

~~~
disordinary
I was on a Cathay Pacific flight a few years ago and was looking at all the
games they had in the in flight entertainment system. This was an older plane
with really bad games, one of which was a number guessing game. I was bored
and so wondered what would happen if I entered a negative number as my guess.
The entire in flight entertainment system on the plane was down for a couple
of hours until the crew figured out how to restart the system properly.

~~~
grzm
I think you mean Cathay Pacific?

Edit: Glad to help! I was going to delete after you corrected, but it looks
like I can't :/ Interesting note: Google doesn't even give me the "Did you
mean" option when I tested "cafe pacific". Must be a pretty common mistake.

I've always wanted to fly Cathay Pacific, as I've heard the service is
fantastic. Never had the opportunity yet, though.

~~~
disordinary
A lot of those big airlines are a mixed bag. If you get one of their newer
planes they are world class, but we were on an aging 747, it creaked and
groaned, had the aforementioned in flight entertainment system, and the
toilets stunk up all the rows near them. But some of their planes are
beautiful and the service is top notch. This was a few years ago but even in
economy they gave everyone a little pack with toothbrush, ear plugs, travel
socks, and those masks you put on to sleep when it's light.

I've had a similar experience on Lufthansa where one flight was an amazing
plane with a bar downstairs, and all the mod cons, and then I had a flight
from Frankfurt to LA which was 12 hours without any back of seat in flight
entertainment (it just had a few screens hanging from the ceiling). This was
only three years ago.

That's why I try and fly Air NZ wherever I can, it's not the best airline in
the world but it's consistently good across the whole fleet.

------
themodelplumber
What I found horrible was kind of an opposite experience with a bug. There was
a developer who built a really powerful project on an aging framework. You
could do amazing things with this software; the results were always
impressive, but the underlying framework and its tools made the experience a
bit less than "slick" on modern computer systems.

So the very nice community that had risen around this software kept shrinking
due to competing projects drawing people away. Eventually, things got really
stagnant, but the developer just didn't seem to notice or care and kept
shipping code, slowly as usual. Finally he pushed some big changes to the
software, made an announcement, and members of the community _really_ woke up.
"Finally, the features I've been waiting for!" "Wow, thanks for squashing all
these bugs!" These people had been subscribed to the mailing lists for over a
decade and thought it polite to reply and express appreciation and admiration
at this recent--and huge--development.

Six months later, the developer posted to the general list. His message was
brief and to the point. There had been a showstopper bug, he said, and today
he was providing an updated release with the fix.

Looking closer at the announcement, I realized that the bug meant the latest
version of the software _could not perform its intended function_. At all.
Completely broken.

So: No one--not a single person (self included)--except the main developer had
even given his big release a try. And _he_ hadn't even tried it for ~6 months!

This made me feel really awful for the developer. Fortunately, today people
still use the software and have since shared their results & appreciation. But
I will always wonder if the developer felt that experience as hard as I did,
or even worse.

~~~
Declanomous
Well, if it was on Linux, it could have been that people were excited for the
release, but it wasn't available on their distro's repo yet. I've definitely
been really excited for a feature and then discovering it doesn't work once
it's available in my repo 6 months later.

As much as I like the repo system (and C), I was pretty impressed by an
experience I had a little while ago. I found a bug in Atom after a new
release. I was able to track down the cause in about an hour, and the person
who made the commit that caused the bug fixed it an hour later. The bug fix
release was available within 24 hours or so. I don't know JS or Coffeescript,
but it was verbose enough that I was able to track down the bug and get it
fixed without having to wait for the bug to be accepted upstream, or download
and compile the package. Compiling a package isn't hard, but I only do it for
the software I _really_ want the most recent version of.

------
mmaunder
Timthumb is how Wordfence came about because my own site was hacked via the
vulnerability when it was an 0day and that's how I got involved in WP
security.

[http://markmaunder.com/2011/08/02/technical-details-and-
scri...](http://markmaunder.com/2011/08/02/technical-details-and-scripts-of-
the-wordpress-timthumb-php-hack/)

Props to Ben. When I worked with him to fix timthumb I don't recall him
mentioning that it was someone else who wrote the bug.

There have been plenty of vulnerabilities just as bad or worse since - and it
keeps our site cleaning team busy. Besides Timthumb, the other top WP
vulnerabilities/exploits are Revolution slider, Mailpoet and Gravity Forms.

You rarely see a site hacked via an old timthumb these days but it still shows
up. Most hacked sites are from the other three newer remote code exec vulns I
mentioned.

~~~
OSButler
One big problem with Timthumb was that several themes came pre-bundled with it
and to make matters worse, they often changed the name of the file, so that it
wasn't enough to just scan for the filename but also for variants and file
contents.

At the time it was the exploit that kept on giving, since you would suddenly
find another timthumb related file on the server that was being targeted,
followed by another, ... In the end, a lot of the hosts simply blocked
requests to timthumb related files, in order to try and stop it on the server
level and not the individual account level.

As you mentioned, these days the sites that I'm being contacted about that
have been defaced or were hosting abusive content, are usually related to
severely outdated plugins. I haven't seen a timthumb issue in quite some time,
but had to restore a site that was affected by the original(!) Revolution
slider exploit recently, due to having a several years old version of it
installed.

------
Falkon1313
My worst was probably when I rebuilt an old, broken recurring billing system.
Unfortunately, in the real world, the retry on failure limit didn't quite work
as intended. If a charge failed or was declined, it was supposed to try again
a couple of times over the course of several days and send an email to ask the
subscriber to enter new payment info. It worked as expected in testing.

Unknown to any of us, the code that triggered it on the production server was
being triggered every few minutes by an outside system instead of once a day
by the server (as it was supposed to be). So after deployment, thousands of
failed transactions were being retried over and over every few minutes. People
woke up to scores of emails about the transactions failing and called their
banks to find out what was going on. Meanwhile the company that ran the system
got some rather unfriendly calls from the banks as it must have looked like
they were trying to do something nefarious.

The good news was that no one was overcharged, it was only failed transactions
that were repeating (and repeatedly failing). Tripped a lot of alarms though.
I added safeguards against the code being triggered by external systems and
also explicit chronological limits that didn't rely on the server
configuration. It was a learning experience.

~~~
jermaustin1
A very similar thing happened to me on my first payment processing system.
Thankfully it maxed out at 3 tries then sent a final email that the account
was suspended until the payment method was updated. The paypal portion worked
as expected... they have better devs than I ever will be, but the
Authorize.Net portion would fail out within 3 hours.

The bug was simple to fix, the production database's schema failed to update
in the deployment. Once we figured that out it was 10 minutes to rerun the
deploy script.

------
SparkyMcUnicorn
This might be considered along the same lines of "unintended use of code".

When I was around 13 I wrote a bash script that would automatically backup
your browser data to an FTP server of your choosing. This included your the
passwords database, and I made it work with just about every browser possible.

I released the script on some IT forum (I forget which) and it was a big hit
for some reason, but then someone re-released it on hackforums and told people
to hardcode the server credentials instead of using a settings file. Tricking
someone to download a tiff, exe, whatever would upload their password
databases from all their browsers in a matter of seconds.

It got quite a bit of attention (many thousands of views) and I've always felt
bad about the amount of credentials that were possibly stolen.

------
bandrami
It was 1995, and everybody was excited about this new technology called BGP. I
was a young sysadmin on his first job at a NOC.

...

Ten minutes later, most of Fairfax County is stuck in a routing loop, and
we're getting angry phone calls from NOCs in California, the UK, and I think
even Hyderabad. While I didn't _actually_ take down the Internet, it gave me a
lot of appreciation for how vulnerable its glue is to human error.

------
keyle
I get contract work from time to time about making a wordpress based site for
customers. And they all have custom requirements on top. Granted I haven't
done it for years - it's probably better now - but to me, it's just a shinier
Joomla.

I couldn't help but facepalm at this article. I understand deadlines and all,
but get a friend developer to overlook your code before releasing in the wild,
because you know the caveat of "hey I haven't tested it properly" isn't going
to make anyone itch.

~~~
mercer
I've found that the Advanced Custom Fields solves most of the custom needs I
get from clients.

I'm not a fan of Wordpress' innards. I usually avoid themes or plugins (other
than ACF and a few 'crucial' ones). I'm not a fan of the theming system to the
point where I often override or circumvrent the out-of-the-box queries and
templating rules. I use my own get_partial() function that properly isolates
my template variables and have even used React for the view layer once.

But I still use Wordpress quite often because 1) so many of my clients need
_exactly_ the features that wordpress offers, 2) they're familiar with WP and
often insist on using it (and it saves me support), 3) I've found that the
CMS-part of the whole thing is by far the most work, and building something as
slick as WP' admin interface is just too much reinventing the wheel, and 4)
there's still an advantage to being able to just run the site on 'typical'
hosting (speed, pricing).

Furthermore, the inconvenience to me is a huge advantage to the client,
because if they need small changes or additions they're now free to hire any
(shitty) 'webmaster' to make css changes or install crappy plugins that
magically offer them the features they need. These webmasters are a lot
cheaper than I am.

All that said, I generally prefer the kinds of projects where I don't feel
Wordpress is the right solution. But for a number of reasons the WP projects
are still the most common and relevant ones right now.

~~~
addedlovely
You'll probably really enjoy Timber for themeing
[https://wordpress.org/plugins/timber-
library/](https://wordpress.org/plugins/timber-library/)

Integrates nicely with ACF as well.

~~~
scarecrowbob
I do a lot of WP work.

Personally, I like things like Timber, but IMO, you really want to avoid
getting very far out from the most basic stuff if you ever want to hand the
project off to rando WP folks... I stay away from anything even as complex as
sage/roots for those reasons.

Though I don't even like ACF, as I have a bunch of snippets for building post
meta interfaces and ACF is large enough that I haven't read all its code. I'd
rather just write 200 lines into an MU plugin.

------
drinchev
For me, the biggest Wordpress fail, is a file called xmlrpc.php. This file is
just a menace and should not exist in the default installation.

My first two things I do when I install a new wordpress server is add this to
`.htaccess` :

    
    
        Redirect 301 /xmlrpc.php http://127.0.0.1
    

Otherwise you will have a very easy DDoS vector that can bring your website
down, immediately after some bot finds out this file exists on your host.

~~~
rmccue
WordPress 4.7, due in a couple of weeks, includes a brand new REST API which
means we can finally start removing some uses of XML-RPC and maybe even
someday disable it.

~~~
mtberatwork
One of the major issues with XML-RPC functionality in WP is that it's enabled
by default and opens up a wide target for brute force attacks[1]. I'm assuming
this will not change with their REST API. Let's face it, how many WP sites out
there really need POST/PUT/DELETE endpoints to be enabled for their family
blog?

[1] [https://blog.sucuri.net/2015/10/brute-force-amplification-
at...](https://blog.sucuri.net/2015/10/brute-force-amplification-attacks-
against-wordpress-xmlrpc.html)

------
tracker1
I used to run random feed attacks on the aol/yahoo account fishers...
basically feeding and submitting their forms with random names/emails, etc...
so that they'd get so much bad data the good would be harder to sift through.

Then, iirc around 2001, there was some change in the law that made that action
potentially a felony (regardless that my target was also probably illegal), I
stopped doing it...

Overall more of a grey hat thing, but it was cool to play with... would set a
new set targets almost every day, as I received fishing emails.

------
jaredmiwilliams
I previously worked for a state government, and owned a project that served
districting information. It was used by constituents to find their elected
officials and to display different types of districts on maps.

I pushed out a bug that took down the mapping functionality. Normally this
would not have been that interesting, but at the time the state was in the
middle of a heated redistricting debate. Some newspapers around the state
picked up on the outage and attributed it to shady dealings from the majority
party at the time.

Slightly different, but it's not often that I have the opportunity to check in
code that will lead to a conspiracy theory.

------
mvindahl
Far smaller scale but this one still makes me giggle:

Worked at large enterprise, at small project team working on the successor for
a dated customer support system running in production. Apart from that vision,
we had little idea about the domain and about what we were doing. Another
team, located in France, was running the existing system, and they were
required to cooperate with us. In reality think that they were quite
unimpressed with our efforts.

Anyway, we wanted to import their data into our system -- partly for indexing,
partly because we saw this as the future -- and we managed to obtain
credentials to use their SOAP API. There was a _lot_ of data, and it was
paginated by the API, for obvious reasons. Hence, we wrote a script which,
when our system was spun up, would import the data from the legacy system by
querying all pages of it in rapid succession.

During development we imported from a test system of theirs which contained
only a small amount of data but for alpha testing we wired it to their
production system. After about 15 minutes we got a phone call from France
demanding to know why on earth we were forcing the company customer support
system to its knees in a DOS attack. By that time I think there was no doubt
in their minds that they were dealing with clowns.

------
vlunkr
This is the scary part about dependency-heavy projects. Sure, you trust the
developers of Rails, but what about that gem 5 levels down your dependency
tree? Maybe you've seen the name as you glance through your lockfile, but you
don't know what it does, when it was last updated, who (if anyone) maintains
it, how competent they are, or what processes they have in place to keep bugs
out.

~~~
ianlevesque
[https://github.com/rubysec/bundler-audit](https://github.com/rubysec/bundler-
audit)

------
jermaustin1
Anyone here ever checked in a bug that infected large numbers of users?

~~~
dotancohen
In fact I have. I hit Enter before completing the WHERE clause, and because
I'm OCD I close parentheses and quotes before filling them in, and put the ;
at the end of SQL statements before filling in the clauses.

Well, I _did_ a long time ago. Now I treat the semicolon key like I treat the
Enter key. Lesson learned.

~~~
jermaustin1
I am also much more cautious now. When I write my UPDATEs I leave out the SET
until after writing WHERE clause. And for DELETEs I actually write it as a
SELECT first, then once all the rows I expect are accounted for, switch SELECT
to DELETE.

Since I use MSSQL, the semicolon isn't required, only a new line, so I
actually start every .sql file I write with 3 new lines and an open multi-line
comment this way if I accidentally hit Ctrl-E without selecting the EXACT
statement I want, it throws an error.

All of these things to combat my stupidity.

------
beejiu
On a much smaller scale, I published a small change to a WordPress plugin I
develop called Thank Me Later. Unfortunately, I had used a closure which was
only supported by PHP 5.3 and above. At the time, there were still a large
number of users using WordPress with PHP 4.something. Being such a small
change that couldn't possibly cause any harm, I got on the train and was away
from the computer. An hour or so later, I opened my email inbox full of
messages like "Help! My website is down", etc. I estimated I knocked at least
100 websites offline.

------
crisp
Reminds me last summer when I was working as a junior software developer in
this rather small or medium sized IT company. We used our custom framework
written in PHP, and the funny part was that the framework didn't have a single
test. Everything had always done by testing manually and hoping for the best.
Had I known this beforehand, I would have applied somewhere else.

Anyway, our customer was (and still is) one of the biggest wholesale in some
part of the Europe. We had developed online store for them like ten years ago,
and the project had become more or less a legacy project where everything was
outdated and nothing ever worked. If the customer asked some changes or
updates, they were always implemented without proper refactorization nor
testing. So you can imagine, if everything is done like this for the past
decade with this custom framework, the code base just becomes too complex to
maintain, not to mention developing it any further.

So, I had worked in the company for about 3 months when I am given this
project. The former developer had left huge changes unfinished without too
much commenting nor even briefing me about them. Took me like 3 months to get
myself familiarized with the project, finish up the changes and carry out the
rest of the requests. The summer ended and everything seemed to work fine in
the local as well as in the staging environment. It was Thursday morning I
pushed the changes live and everything crashed. Quick rollback, some research,
couple fixes and a new try. Same thing. New rollback, some research, some
fixes and push. Still broken.

Took me about 10 hours with the help of 2 developers to find out what was
wrong. The framework was fundamentally broken in one part, it somehow didn't
include files correctly. Still to this date, we don't know why it worked both
in local and staging environments but not in production. When I got this
fixed, new problems arose: customers had lost products from their carts,
logins didn't work, the site was slow... After banging two weeks my head to
the wall, everything finally worked. I was so relieved.

These problems affected thousands of users throughout Europe, and I can
imagine how pissed some customers might have been. I was so ashamed on behalf
of our company, our "framework" and how things were done that I quickly left
my resignation.

If I learnt something, discuss the tools and how things are done in the
company before applying for the job.

------
sschueller
Does anyone have a good guide/article on hardening wordpress from an hosting
perspective? What is listed at wordpress.org is too thin for me. What tools
are available that can be run on the server instead of in wordpress (WordFence
alternatives that I can run on the CLI via cron etc)?

For example I have fail2ban rules to catch break-in attempts etc.

~~~
nisa
From a hosting perspective you want to isolate Wordpress as much as possible -
i.e. putting it in a namespace with php-fpm (easiert way is using firejail)
and running PHP as a seperate user. You also want to only give it the minimal
required access rights to the database.

You also need to keep up with security updates and run a supported PHP version
- there are still crazy amounts of PHP < 5.6 installs out there :(

From Wordpress.org own stats:
[https://wordpress.org/about/stats/](https://wordpress.org/about/stats/)
[http://php.net/supported-versions.php](http://php.net/supported-versions.php)

That's 70% of Wordpress installs that run on EOL and unsupported software!!!

Besides that update your kernel as soon as possible (reboot or hotpatch e.g.
ksplice) and keep userland up to date. There are tons of local root exploits
for Linux that help an attacker to attain some serious persistence.

Some other ideas I've read is keeping most of Wordpress readonly for the PHP
user and use wp-cli ([http://wp-cli.org/](http://wp-cli.org/)) or similiar
tools that update from the shell. You can do this some group membership
trickery.

It's also possible to almost completely forbid access to web clients to wp-
content/plugins and wp-includes if your plugins don't do something stupid. But
that's error prone and something you should test. For a simple site with a
theme you control and few or none plugin it should work fine through.

As a general rule be very conservative regarding plug-ins and themes and spend
some time to look at what comes with them.

------
agumonkey
Spin it: how I created more IT jobs than the Obama administration.

~~~
Roritharr
I like that idea, at our Job we sometimes Joke that "Taskrunner" should be a
Job description for some people.

------
user5994461
Disclaimer: Title is misleading. This is not a 0-day nor a new vulnerability
on WordPress.

The author is just talking about some old stories that happened years ago.

Everything is fine. There is no emergency and no need to go update your
wordpress setups ASAP. :D

------
cdevs
Apparently this years theme was for everyone to keep telling me database IDs
are unique that aren't unique...

