

Ask HN: Have you ever had one of your sites get hacked? - vaksel

How did it happen? How did you recover? What did you change so it wouldn't happen to you again?
======
modoc
Yes. Many years ago a server was gotten into by a brute force ssh script. The
cracker then used a local exploit to gain root access and run all sorts of
unpleasant software, downloaded my databases, etc...

I ended up just getting a new server, as I couldn't track down all the things
he'd done. I started learning about server security. I reduced the number of
people who had ssh accounts, and installed stronger password rules. I use
iptables to block brute force SSH attempts. I use logcheck, chkrootkit, and
other tools to constantly monitor my server.

~~~
pert
I'd say the the single most important thing that you've mentioned there is
'logcheck'. If you can remove all of the login spam (by moving ssh to a non-
default port for example), then watching your logs becomes a reasonable task
and will alert you to any specifically targeted attacks.

------
RobGR
These are the ones that I know about.

In 2001 I was running a redhat 6.1 server at my house, and the ftp had a known
vulnerability in it, but I didn't use ftp so I ignored the update. Then
someone was over at my place and trying to retrieve an ancient mail archive
from an old account, and asked me to turn it on briefly so they could upload
it to me; in the couple of hours it was on, a worm installed a rootkit and a
IRC based control thing. None of it worked because it was compiled against
different versions of libraries; it hung with a full cpu load, which is how I
discovered it. I was using TripWire at the time so I was able to identify all
the files that changed. I tracked the attack back to someone else's Redhat 6.1
running on a residential DSL line in S. California, which had a web page
advertising an auto salvage place on it, and I got no responses to my friendly
emails. I started paying attention to my TripWire reports after that. I
cleaned up by deleting individual files ( I saved and examined them with
various tools.)

I had a customer in the VoIP / Calling Card area who was using a server
provided as surplus by someone who feed him overseas calls, and he kept losing
his overseas termination contracts and he suspected this dude had put a
backdoor on the server and was finding them and cutting out the middle man,
and he was right, a disguised sshd listening on a weird port. The customer
decided to leave the backdoor open and pass only selected traffic through that
box.

A different customer was traveling and from a university computer cluster
ssh'd back into the server I built for him from a machine that had a trojaned
sshd. The guy came in and trojaned / rootkitted that box, and followed my cron
job of a key-based ssh login that scp'd the backups off that nightly back to
one of my personal machines, and somehow upgraded priviledges to root. I
discovered it because I happened to be looking at top when he started
compiling some hacker tools on my box. He left bundles of php code in various
directories hidden by having a name of three spaces and etc, the code was web
scrapers of email addresses and phishing pages that looked like the login to
various bank sites.

By this time I using integrit instead of tripwire, and on debian. I
disconnected all the infected / suspected servers, booted them from various
livecds, and scanned them for rootkits; then I wrote a few lines of perl that
re-installed every installed debian package, overwiring files, and listed
every file on the system that was not from a debian package. That was how I
found the directories with 3 spaces for a name and etc.

That guy kept coming back for 6 months or so -- every couple of weeks, there
would be a single login attempt on some server I owned, using the distinctive
username that was part of the cronjob backup. I would track down the ISP or
administrator of the IP he came from and report that the box was probably
hacked, I was mostly ignored but I did get a few notes of thanks and I did
have a conversation with some college kid running his frat's web site on how
to keep his linux secure.

Several times I have helped people who used one of the cheaper web hosting
places clean out little snippets of php code out of all their files, that were
probably put there for SEO / scamming purposes by an employee of the hosting
company or a customer of them that had figured out how to escalate privileges.
I think that is actually pretty common, but most people never notice because
they only upload files, to their site, they never download and their integrity
months later.

There are probably a few that I have forgotten too. I have a usb thumb drive I
keep a collection of rootkits, phishing code, etc gleaned from break-ins of my
computers or obtained from friends.

I had two or three compromises of client machines that all fell into the
catagory of a Redhat 9 box with the original, un-updated, vulnerable samba
running; the machines were never intended to be connected to the internet, but
were connected briefly to dialup allow remote support. In those cases we wiped
the boxes clean.

------
dpifke
When I was a young sysadmin at an ISP in the mid-nineties, we had a shell
server that would be exploited fairly regularly. This was in the days before
automated updates, when the standard Sun delivery method for patches was a
mailed CD-ROM. (Online patches were available, but you had to seek them out,
and even then they occasionally lagged behind the exploit code being posted to
bugtraq by a few days.)

A big part of "securing" this particular machine was to separate it from the
rest of the network so that even if compromised, it had no more access than
the outside world. I set up an RSA private key for SSH authentication so that
I wouldn't have to provide my password to a possibly compromised sshd. I
backed up the tripwire database to removable media and checked the integrity
of the machine's binaries regularly.

Shell access as a standard feature of dial-up internet access died right
around that time, and I don't think many ISPs were sad to see it go. That and
Usenet were huge time/resource sinks that I'm not sure provided a lot of value
in the end to our average customer.

~~~
jacquesm
There is nothing that will teach you security as fast as having a number of
tech savvy users on your machine with user rights.

Privilige elevation is so much harder to protect against than getting people
on your box to begin with.

------
dmix
When I was 17 at my first job developing web sites, I set up a PHPBB forum for
a customer. Brazillian script kiddies used a security flaw to gain access to
one of our entire servers and ruined a bunch of sites.

Great beginners lesson. Keep your mainstream opensource apps up to date.

~~~
ja2ke
I was going to tell this exact story, including PHPBB and a Brazilian script
kiddie taking down a ton of sites. I wasn't the one who installed the BB
though. I just lost a bunch of data. Was it the LFNetwork server? :)

------
bjw
Yes, due to the first OpenSSH remote exploit. With some many servers to manage
one of them was forgotten and hacked.

Our recovery was a bit complicated, because we didn't trust any of our boxes
after that. So, we shut down the hacked box, mounted somewhere else, removed
all the data and reinstalled. We reinstalled all of our other boxes too...
Lots of work and lesson learned.

To secure our systems now we take multiple approaches:

-Different operating systems across our network. For example, our web server runs on OpenBSD and our database on Linux (RHEL). We also have Debian and FreeBSD.

-Reduce the functionality of each system and who can access it. Plus, never allow SSH from one critical system to another.

Added monitoring. We can't be 100% safe, so we started using:

-<http://modsecurity.org> to filter web attacks

-<http://ossec.net> to monitor logs and file changes

-SeLinux enabled on all our Linux systems

-systrace on all our BSD boxes

-<http://sucuri.net> to monitor our DNS, Whois and site.

------
olihb
Only once.

I had an asterisk server with polycom phones on a network not connected to the
internet and because I'm lazy all the phones had 1234 as their SIP password.
If you're inside the building, you don't need to hack the server, just use the
phone in front of you...

Fast forward a couple of years and now some of us work at home or telecommute.
So I had to configure an asterisk server hosted in a datacenter. I used the
same config files for the autoprovisionning of some of the phones.

Well, one of the account got exploited(SIP scanner that tried simple password
on common extensions) and called thousands of numbers in Colorado(from Québec,
so long distance) to fish for credit card numbers. Our termination provider
saw that these calls didn't correspond to our normal usage pattern and
deactivated our account until I called them.

In all, it only cost us 10$ in long distance fees because of the audit server
of our provider(unlimitel.ca). Now all the phones and servers in our business
have strong passwords. And I had no excuses, the phone are auto-provisioned so
phone users don't even see their SIP password...

------
jacquesm
yes.

I run a filedump as part of all the stuff we do, one day some smart alec
figured out that I had forgotten that if somebody uploaded a .php file that it
would execute instead of being downloaded... Very very stupid of me.

The next step was to upload a nice little backdoor filemanager package written
in php and run it from the download directory (this must have been harsh
because the url switches after every download). Eventually they got access to
the password hashes, but since I found out about the hack within 24 hours I
just changed the passwords on the system after plugging the hole.

Obviously the way to handle that was to refuse files whose exension is .php.

I still can't believe that it was that easy to get in, and I feel totally
stupid for not having foreseen that.

~~~
pwmanagerdied
It's possible that you've still missed something, unless you're certain that
.php is the only extension which your server executes (.php5 often works, and
some servers will also run files with .cgi or .py extensions, among others).
Blacklists are never fun.

The solution which occurs to me is (assuming you're on Apache) setting the
.htaccess file in that directory to cause all files to be downloaded,
regardless of their type. I haven't tested it, but I think this is the setting
you'd need to make:

    
    
      ForceType application/octet-stream

~~~
jacquesm
you had me paraonoid enough there to check :)

It's ok though, the fix I applied was a little more elaborate than indicated
above.

thanks anyway!

~~~
TimothyFitz
It's _really_ hard to harden yourself against code getting interpreted by the
browser (and thus allowing someone to steal cookies and impersonate other
users). Here's a long article on exactly how complex it is (as known today):
<http://tstarling.com/blog/2008/12/secure-web-uploads/>

Personally, I just don't trust web uploads on the same domain. Put them on a
subdomain or better yet a completely separate domain, preferably served out of
something set up to _never_ execute any code whatsoever.

~~~
jacquesm
Excellent read Timothy, thank you. I have changed some of the code because of
this article. I also checked the history of all uploads to see if any of the
holes mentioned had been used, and apparently there was at least one instance
of somebody trying the .js route but they got hung up on the fact that the
file names are changed after every download.

------
kngspook
Yeah, I had someone alter the HTML on a static webpage at some point to make
it link out to some sites in China.

It was a simple site, so I just scanned all the pages to ensure the rest
weren't tampered with (it was just index.html).

Then I switched webhosts (from iPowerWeb to NSFN+WebFaction) because iPowerWeb
was incompetent. (And not just because of this incident; they had 3 hour wait
times when I called, weekly unplanned downtime with both their web and email
servers (which they wouldn't even admit), lost a valuable business domain
which was snapped up by some fellow in India 2 minutes before I found out
about it, and billing problems on top of it all...)

------
ethanmullins
Once in 2007 when I was 14,I was running a RealVNC server on a computer with
the default port forwarded. Around the same time, unknown to me, a buffer
overflow exploit was released for the particular version of RealVNC that I
had. Some kid eventually found it, and proceeded to do stupid things on my
computer (IM my friends, change my background, etc). Luckily nothing of any
importance was lost.

Another time, in 2006, I had a shell account on my friend's Linux server. At
the time I was using it to host a Wordpress blog. My friend was running an
outdated version of PhpMyAdmin, and it was exploited. The guy who got in
dumped all of the databases, and on top of that, e-mailed us screenshots of
him doing it. :/ Thankfully, all of the users had recent enough backups to
deal with.

------
dannyr
Yes. Mine got a SQL Injection attack so the hacker was able to run scripts
that changed the database. I thought I plugged the holes to prevent it but
apparently, I missed one.

I also did not make a fuss about it since the hacker likely wants some
attention. I was backing up my db regularly so I just restored from a backup
when it happened.

------
eli
I set up a sendmail that was hacked to send Chinese spam. Lesson learned:
don't set up a sendmail server unless you really know what you're doing. I
think I nuked the whole server and started over.

Edit: In fact, we ended up moving those domains over to an Exchange server to
let the Microsoft guy run them :(

------
rs
Back in the day, I ran a Mandrake linux box from home on an old laptop
(student days, wahey!) which used to host my site and emails. One fine day, I
realised that it was getting a little sluggish in terms of performance.

On a closer look, running "top" showed a process called "z" that was eating
the process up. Yup, only called "z". It turns out that someone was running a
script looking for awstats vulnerabilities, managed to find my installation
and installed "z". "z" opened a back door and a small file management utility.
Thankfully, nothing was deleted. I think the intruder couldn't get out of the
www-data account. I just reinstalled the whole thing from scratch.

The worst part of it was that the vulnerability (and the solution) had been
out in the open for months!

Lessons:

1\. Always keep an up to date system

2\. Subscribe to all announcement/security mailing lists for sites/packages
that are deployed to the public

------
pert
I got burned about 10 years ago by 'samba' not doing what I thought I'd told
it to. I used the 'bind interfaces only' configuration directive, which I
thought would prevent it from opening any ports on my Internet interface. I
was wrong.

Unfortunately I also had no idea about the concept of security patches and, to
this day, I still have no idea how I should have gone about getting security
updates for Slackware (I switched to Debian and never looked back). The result
was some script kiddie got root and started to use my box to start scanning
for more vulnerable samba installations to break.

My response was to unplug all of the network cables and have a poke around to
see what he'd been up to. I took a full backup of the box and then re-
installed it from scratch as I couldn't trust it.

I learned that you should always look at what ports you have open (`netstat
-lpn` is my favourite command for this) and that there are some times when a
firewall might be of use (I'm not a fan of firewalls on anything other than
gateway boxes).

------
trafficlight
If you said no, you haven't been doing this long enough.

~~~
arantius
How long is "long enough"?

I've had a dedicated server running my (personal) web site, mail, and a
variety of other services, directly connected to the public internet, with no
firewall, for five years.

My domain is #95k in alexa, #69k in Compete and #35k in Quantcast. Just to
say, while it's no heavy hitting destination site, it's not just a tiny
personal site either.

I've never been hacked. I just don't set up insecure services.

------
paraschopra
Yes, last year my blog got hacked by a script kiddo. The blog software I was
using had just been found vulnerable and almost all blogs using that software
got hacked that day.

I had one old backup but lost a few months posts. Was terrible but got over it
soon.

------
Jem
About 6 years ago I was running a "top list" script that was exploited. They
planted a backdoor in an obscure folder, so that even after I removed the
script they got in again twice. Took me a week to dig out every dodgy file
they'd put in.

Also had my very first PHP script exploited (2004ish). I was threatened by
some jerk who told me if I didn't remove it from my server he'd exploit it. I
didn't, he did, lost months worth of work.

Been nothing since, touch wood.

------
drinian
Yes. It was an up-to-date install of Wordpress, about two months ago. Luckily,
the domain was being scanned for vulnerabilities by Duke University OIT, and
they let me know about it; it was non-obvious that the hack was there. There
were certain URLs that had lists of keywords, mostly pharma scams.

I don't use Wordpress any more.

------
DanielBMarkham
Yes. Many years ago I ran my own Windows Server network out of my home office
-- Exchange, SQL Server, SMS, etc. It was all part of learning how to use the
tools and staying on top of the technology. At the time I had a T-1 leased to
the office, which gave me tremendous speed (relative to everybody else in
town, that is)

The problem with this plan was that Microsoft is a place full of really smart
people making really complicated software. _Each_ of these server programs was
it's own world of consultants and gotchas -- it was too much to manage while
doing other "real" work. Patching and tweaking alone could have took all of my
time.

I guess it took about a month for me to figure out that my Exchange Server had
be pwned and was sending out spam email for somebody with a Russian IP
address. I figured it out when I went to the box for something else and
realized the CPU was pinging out at 100%. A little digging around and wham!
Guess what? You've been punked.

I played around with cleaning it up for a couple of days but it just wasn't
worth the hassle. I learned a lot about Exchange Server databases in the
process, but I have no desire to be an email technician. And with Microsoft,
everything is tied into everything else: it's never just one complex product,
it's fifty.

So I shut the entire thing down. Swore that if I needed server or storage
stuff I'd let somebody else worry about the details. It was one of the
smartest things I've done as far as network administration -- understanding my
own constraints. Now if I want Microsoft Server programs, I just rent them off
the web. But I've found that, with the exception of SQL Server and IIS, most
of the MS Server products are simply too overblown to make it worthwhile for
an advanced hobbiest to play around with.

As a humorous side note, the next New Year's Day I got an email from some
Russian guy who wished me the best for the coming year. We exchanged email
pleasantries on New Year's for a few years after that. I always wondered what
happened to him. I was hoping for jail, but he's probably got a villa and big
ship in the Med somewhere.

Punk. (grin)

~~~
sho
Are you sure your Exchange server had actually been hacked, and wasn't just
relaying? That happened to me too, _ages_ ago, but I wouldn't call it a hack -
just a bad misconfiguration, got me blacklisted at the time. In fact it was so
long ago I can't even remember if it was called Exchange or something else;
this is NT 3.51.

Ah, NT, how I don't miss thee. Although, to be fair, I hear the MS toolchain
has improved greatly in recent years.

------
sho
Yes. I made the mistake of providing accounts for friends, and one of them had
a common account name and an easily guessable password - it was bruteforced,
the server rootkitted, and used as a 0-day ftp drop. I only noticed when
bandwidth usage went through the roof.

I didn't trust the server at all after that, of course, so nuked it and
reinstalled from scratch. Sans my idiot friend's account. Luckily it was a
standalone and not part of the network or i would have had a _lot_ more work
to do; once something gets inside you just can't trust anything. Needless to
say I would never have been giving away accounts on a big, fast, well
connected work machine to friends though .. _coughs_ .. anyway.

Was a good lesson about not trusting others to know what they're doing, taking
steps to minimise bruteforcing (different port, limited tries) and rootkit
detection, which is pretty hard. In fact it can be almost impossible to even
_know_ you've been rootkitted if you do not take proper precautions in
advance. I only knew because they'd blown their cover with the bandwidth; that
was a bit of a wakeup call. Nothing like having one of your boxes subverted
before your eyes and you don't even notice .. that was a nice feeling, not.

Still, glad it happened. Getting hacked is like catching chickenpox; good to
happen nice and early (but not _too_ early), builds resistance for later on.
Haven't been hacked since.

