

M64.pl – how I learned that the default settings are not production settings - artursapek
http://artur.co/articles/m64.pl.html

======
buro9
On the cheap Linode instances I setup for others, I always install this:
[http://www.configserver.com/cp/csf.html](http://www.configserver.com/cp/csf.html)

It includes the equivalent of fail2ban, manages iptables, watches log files,
can email on failed logins, processes gone mad, and a few other things through
a very simple config file.

It's that simplicity of the config file that I like, it means I can hand it to
someone and know that they're unlikely to break something due to
misconfiguration or failure to actually apply something.

This is on top of obvious things like running ssh on a non-default port,
turning off password auth and restricting ssh to named users only (never
root).

The install script is easy enough:
[http://www.configserver.com/free/csf/install.txt](http://www.configserver.com/free/csf/install.txt)
so it's really the case that I can do whatever work I'm doing for someone
(hello Wordpress) and this can be my simple security solution which at least
means I'm sure that I haven't left every window open.

~~~
deoxxa
Why do you run ssh on a non-default port?

~~~
rurounijones
Brute-forcers only hit the default port. If there is no response on the
default port then they just move on to greener servers.

Moving off the default port is a good way to avoid your logs getting filled up
with failed login attempt messages.

~~~
tacticus
Though something like fwknop would be a nicer way of resolving that.

------
kybernetyk
I hope you have killed and reinstalled the box because once a system has been
compromised you can't ever trust it again.

~~~
michh
In this case, if I read the article correctly, only a single account was
compromised. I'm assuming the www user doesn't have sudo privileges (though I
might be wrong as it shouldn't have a password either and it clearly did..).

Sure there might have also been a local privilege escalation vulnerability, so
rootkits are definitely something to check for. Depending on the situation,
re-installing might be less work, so then it'd make sense.

But especially if you're not the only one using the machine, having to
reinstall the OS every time a single user fucks up and has 'password' for a
password, it's going to get really tedious really quickly.

~~~
jon-wood
> But especially if you're not the only one using the machine, having to
> reinstall the OS every time a single user fucks up and has 'password' for a
> password, it's going to get really tedious really quickly.

This is what automated installs are for. Kill the box, fire up a new one, and
run the provisioning scripts. As a bonus you've got actual documentation
showing how the box should be set up.

~~~
michh
If an attacker can get elevated privileges through an user account, the users
themselves can do so as well.

And you shouldn't trust them either, so you need to assume they have if they
could have. That means definitely reinstall when you find out you've been
vulnerable to e.g. a local root exploit, regardless of user accounts have been
hacked.

------
scott_karana
I don't think the www user has a password in _any_ distro, by default.

Meaning one of his administrators must have set one (likely for temporary
troubleshooting), kicking off the whole issue...

edit: if SSH access was indeed the first cause. Running any upload-and-run
script as `www` would let them set the password themselves, I think.

~~~
euank
For anyone wondering how to properly troubleshoot in this manner without
breaking things:

Run 'sudo su www -s /bin/bash'. using '-s /bin/bash' will override the usual
nologin shell, and running 'su' as root will mean a passwordless account can
be su'd too.

This will allow you to try accessing files and directories as if you had the
user 'www's privileges without having to make the 'www' account regularly
usable.

You should never set a real shell or password for any accounts that a real
user will not be using.

------
eli
Disabling password logins is good, but there are other steps you could/should
take. These are all pretty simple: [https://library.linode.com/securing-your-
server](https://library.linode.com/securing-your-server)

Also, I would guess the attack is fully automated.

~~~
hackerboos
I'm scared noobs are going to blindly turn off passwords in their SSH configs
without correctly setting up keys.

~~~
eli
Linode has "console" access through the admin. But yes, good point. One should
follow the linked document steps in order.

------
nodesocket
Also, heads up. Don't ever run Elasticsearch open to the world in iptables.
This is actually very common because users want to use Kibana
([http://www.elasticsearch.org/overview/kibana/](http://www.elasticsearch.org/overview/kibana/)).
Dynamic scripting is allowed by default (< version 1.2.0) and be exploited
easily. It is especially nasty if you run Elasticsearch as root (also very
common).

~~~
x0x0
yeah, "special" people at my work did this. Along with installing /
automatically starting elasticsearch on every ami so that _every_ box could be
rooted, not just the search cluster. And putting the fucking admin pam on the
boxes instead of learning about agent forwarding. And not securing processes
in any way -- why would you use chroot or docker, that would throw up
obstacles in front of the nice chinese people who want to use your ec2 boxes.

it was a very long week.

------
stiff
I once had a box "bot-netted" by some automatic scanner after I set up
postgres on it, being in a hurry after another box crashed unexpectedly. It is
stunningly idiotic to me that SSH seems to be enabled by default for all users
(at least on Ubuntu server).

~~~
zaroth
All users with a password set. So 'setting the password' has perhaps an
unclear side effect.

I guess if you disable password logins, someone would need to get a pubkey
into a user's ~/.ssh directory.

~~~
tankenmate
Or more importantly, once in they insert a public key in there so if you
naively disable password access they'll still be able to get back in. Change
the password AND remove any keys in ~/.ssh. Also a debsums or rpm -qV is a
good idea. A complete re-install is my preferred method though.

------
xtrumanx
I've been considering moving an app off of Heroku and to Digital Ocean so I
could learn the ropes of administrating a Linux server but I feel there's so
much I don't know that I should know if I wanted to effectively manage my own
server.

If I got hacked with m64.pl, I'd get as far as running top and killing the
process. His immediate assumption was that someone got a file that file on to
his server and so he went on to check the logs and found all the failed login
attempts. I would have wondered what the file was, googled around and sat
around confused.

Luckily deploying a new instance on Digital Ocean sounds simple enough so I
would probably just do that but that approach would leave me with a sense that
I don't know what I'm doing at all. There was a time when my solutions to
problems on Windows machines was to restart and/or format but I don't do that
anymore but still meet people who do. I'd rather not start from square 1 when
I make the leap to Linux.

What are the things one should know to legitimately say they can manage their
own Linux server? I could try to just wing it and solve every problem as I
encounter them but for right now I'd just like to identify the gaps in my
knowledge.

------
cel1ne
Beware of this:
[http://unix.stackexchange.com/a/8886/11172](http://unix.stackexchange.com/a/8886/11172)
To fully lock down SSH you have to check PAM config as well.

My 2 cents about defending against scripters:

* SSH: non-standard port, no passwords, special user-group who is allowed to login.

* WWW: redirect everything to https. For some reason bot-attacks dropped from hundreds to none over night. :D

------
mistaken
The m64.pl executable seems to be a vanilla version of cpuminerd. By default
it connects to [http://127.0.0.1:9332](http://127.0.0.1:9332) and mines using
the scrypt algorithm. It could have been started with different parameters, or
the port was tunneled to a pool. Would be interesting to know the ip.

off: does anyone know which font he uses in the terminal? Looks awesome.

~~~
euank
Assuming you're referring to the font of the blog's text...

Firefox: Right click, inspect element, click the "fonts" tab on the right.

Chrome: Right click, inspect elmeent, look at "Computed" styles on the right,
find Rendered Fonts.

Ironically, the answer is he's not using a font. It uses Times New Roman on my
Chrome and DejaVu Serif on my firefox because those are my default system
fonts, not because he specified them. If you look at the source of the
webpage, you'll see he has zero stylesheet links, all his styles are inline in
<style> blocks... So searching font in that one source page is sufficient to
find he only styles code blocks, the rest is default.

If you're referring to some other font, I have no clue, sorry!

~~~
mistaken
Thanks, but I was referring to the screenshot :)

~~~
artursapek
It's
[http://www.fontsquirrel.com/fonts/M-1m](http://www.fontsquirrel.com/fonts/M-1m).
I don't think I'll ever switch from it. Very narrow and efficient font.

------
viraptor
While someone could set the password on the www user manually, I get the
impression that the author is the only admin and would remember that. Since
the only authenticated user is www, it's likely someone just exploited his
application / webserver and set the password on www for later ssh logins.

The author will soon know that if the situation repeats without the ssh
access.

------
dbalan
Using a tool like fail2ban is also favourable. Most of the bots will give up
and look for other targets even if you ban them for a hour.

~~~
danieldk
The problem with fail2ban is that you have another possible vulnerable program
that an external user can feed information to (although very restricted).

I have always been surprised that fail2ban is so popular, since iptables can
do rate limiting, etc. So, it's easy to block most attacks with the in-kernel
firewall:

[http://www.debian-administration.org/articles/187](http://www.debian-
administration.org/articles/187)

~~~
threeseed
Here is the site: [http://fail2ban.org](http://fail2ban.org)

Since clearly you are confused what it actually does.

~~~
danieldk
I know perfectly well what it does, and it does not change my point: extra
parsing is extra scope for vulnerabilities. You also don't address my main
point: you can use iptables rate limiting to block brute force SSH attempts. I
used this for years and it works perfectly fine. Connecting to often within a
certain timespan and you get DROP'ed.

But you probably thought that I was suggesting to block IPs by hand. I wasn't.

------
mqzaidi
You should also run logwatch - it will show you all the attempts that keep
happening on any public servers running ssh. I once had to look at a box
compromised, which was then used to brute force other servers, and it
contained a file with 100-200 passwords for other hosts it brute-forced.

------
Fuxy
I was hoping for a more detailed account of how he managed to log in as www.

That looked like a pretty amateur attempt yet he managed to log in as www.

Was there no password for www?

Why was it even available to SSH into?

This leaves a lot of questions unanswered.

------
zaroth
A couple funny things; the intruder changing the www password, and running a
mining script which is going to generate basically zero satoshi before getting
shut down.

I guess change the password so someone else can't break in after you?

I can't explain the mining script. Maybe they want to alert you the box is
owned, as a means to find only targets who don't wipe their system after being
infected? What's the point...?

~~~
Kiro
Even if the script only manages to mine very little it would still be
worthwhile considering this is probably 100% automated.

------
adders
The script appears to be be at least part minerd, a program for
bitcoin/litcoin mining.

Details of minerd at
[https://bitcointalk.org/index.php?topic=55038.0](https://bitcointalk.org/index.php?topic=55038.0)

------
andyhmltn
Seeing a bruteforce in the logs isn't at all uncommon. I see it every time I
have a new server. Turn off password authentication and disable root login via
SSH. That way, you're golden!

------
nathell
A similar if not same thing:
[http://ubuntuforums.org/showthread.php?t=2203663](http://ubuntuforums.org/showthread.php?t=2203663)

------
keeperofdakeys
And if you do want to use password authentication, simply whitelist the
allowed logins. This assumes you are using a strong password.

~~~
threeseed
Other options: Run it on a different port, use certificates or YubiKey for
authentication, firewall it to your home computer, use Fail2ban or Denyhosts.

~~~
keeperofdakeys
Running it on a different port is only protection against random attacks
(sweeping ip addresses looking for common passwords). This seems much more
like a targeted attack, someone took the time and effort to try to get in. A
small port scan isn't hard, and will get around changing the port. Of course,
the other things you've listed are solutions for this.

I was just posting a single option to complement the other comments.

------
edoceo
Duh. Always lock SSH down

