I've been running a similar experiment. And noticed similar things as the author.
However, I decided to run an additional experiment to contact Amazon since the IP was originating from an EC2 instance. Amazon contacted me after I filed an abuse report and said they were investigating.
A week goes by and I'm still getting hammered. So I email Amazon and asked when it will be resolved. No response. So I email again, again, and again. I finally get a response saying they have resolved it, but I'm still getting hammered from the same IP address.
So I email them once more asking - what is it exactly you have resolved?
No response.
I presume it is not in Amazon's best interest to resolve such issues as long as people are paying for their instances....
It upsets me to hear this. I always wondered how Amazon handled abuse on EC2 as it's an insanely large service. At some point it has to become an issue? Does anyone have any experience with other types of abuse on EC2?
That is always the most difficult part for me when trying to get these issues resolved - the legal side is complicated and I don't have the bankroll to get lawyers..
Or just disable password-based authentication. These "attacks" are little more than log spam and are meant to prey on systems that host novice users with weakpasswords. Require PKI and the log spam goes away.
And I hope nobody is using key based authentication for SSH without storing the private part of their SSH key on an external smart card only, which is unlocked by an external hardware pin pad.
I was being pedantic of course. Password based SSH authentication is fine for 99% of the cases, as long as they use a "secure" password and don't leak it by writing it down or reusing it elsewhere.
See, that's exactly why we don't allow password based authentication. While you can enforce strong passwords, you can't enforce that the user choose a password that is unique to your system, because you cannot know where else they may use this password.
The recent Dreamhost security disclosure is a great example. What are the chances that a contractor is using the same password for a Dreamhost shell account as they are my server? Pretty high, in my experience.
By disallowing password authentication, and requiring SSH-key auth, and requiring that contractors use a passphrase with their SSH-key, I have decreased the chances of a random break-in significantly.
I personally stand there with them (or over Skype) and ask them to show me how they manage their keys.
Most devs know how to use ssh-agent. Those who don't are really excited to learn that they can increase security without an increase in effort.
Password uniqueness, on the other hand, is something that requires continuous effort. There's an ongoing temptation to reuse the strong password that they've memorized for my server in other locations.
Like most things in security, it's a matter of layering and balance. You layer your security, and you stop at the point where the added effort doesn't justify the added security. I view the progression like this
OK: Strong passwords
Better: SSH-key authentication (much harder to brute force, not susceptible to disclosure through third-party security disclosure)
Best: SSH-key authentication with passphrase (prevents unauthorized use of private key if stolen)
Insane: SSH-key stored on external drive with imaginary super-encryption
> Best: SSH-key authentication with passphrase (prevents unauthorized use of private key if stolen)
It doesn't prevent unauthorized use, it just makes it more difficult to attack with brute force. Once someone gets your private key, you should always assume it has been compromised, and replace it immediately. It's only a matter of time and resources before a determined thief has cracked your passphrase.
Edit: not to say I disagree with your recommendation that it's best, just pointing out that compromised private keys should not be taken lightly.
"external drive" is not what a smart card is. You can not read the key from the smart card. The actual crypto operations involved in performing the authentication are handled on the card it's self.
If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised. This has real value and is not "insane"
In your setup, if their machine is compromised, it doesn't matter if they password protect the key, the attacker can key log the key password or replace the ssh executable with one which leaks the key, then ssh into your servers at will.
"If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised."
If the machine you're ssh'ing from has been compromised by a halfway competent attacker, there's no need for them to get the key from the smart card. They can log in to the remote machine by simply piggybacking on to your ssh session (with a compromised ssh client).
For the sake of those who may not realise: You can take a password protected ssh key and remove it's password protection (if you know the password). And do the opposite of course (adding a password to a key without one)
You can do this with policy (or a custom SSH client) but it's really a mistake to treat it as the same situation unless your users are in the habit of sharing their private SSH key files around.
In most situations, getting access to the key file requires a local client compromise - at which point the attacker could simply install a keylogger.
I'd still prefer ssh keys over a strong password. I don't know anyone who is perfect with regard to password sharing. Once you've memorized a wicked good password, there's a strong temptation to use it.
Say you've committed a really strong password to memory. One day, you find yourself setting up an account for some related service that you want to be super secure, but you're going to have to access it regularly. You decide to use your strong password. A year goes by and some hacker gains access to the said service's user database. Your super-strong password is now useless.
The great thing about ssh-keys is that the private key should be on your computer only. Generalized attacks don't go after single private keys, because you're only getting one set of credentials. If you're under focused attack, there are much easier attack vectors than trying to swipe a priv key from a dev's machine. If you can convince your dev to use passphrases on their ssh keys, you mitigate that attack vector as well.
Neither is perfectly enforceable, but using a passphrase with ssh-keys and ssh-agent is trivial. It's not perfect, but it's better than password auth.
Our new national ID cards (called 'citizen card') here in Portugal work like that.
They have a keypair and the private part is never transmitted out of if: the card itself can (when plugged in a reader) sign and encrypt data sent to it. So it's secure even when using a public (not trusted) machine; I mean, it can spy on your connection, but it can't clone the key.
The certificate can be used in browsers to login to websites that support it or any other applications that implement the right PKCS standard.
Yes it does. And it's pretty cheap too. I have an OpenPGP v2 smart card and an SPR-532 smart card reader. I will be writing up how I did it on my blog at https://grepular.com/blog/ probably next week. Need to find an hour or two to write the thing up.
I picked up an OpenPGP card and reader a few months ago and love the technology. My only beef is that the reliability of the software that talks to the smartcards is awful on the Mac.
Right now, it's definitely an approach to take if you are crypto nerd -- it's not ready for regular humans.
Not to my knowledge, but similar approaches have existed for a long time now e.g. RSA SecurID [0]. One day I'm going to set up my Yubikey to perform a similar purpose, but my motivation levels correlate with my paranoia :)
I have a yubikey myself. The smart card method is much better though. When using a smart card, it is impossible even for the owner (who knows the access pin) to read the actual key. The crypto operations involved in the authentication take place on the card it's self. The smart card is protected by a pin, and will wipe it's self if the pin is entered incorrectly too many times. You can even use a smart card reader with a hardware pin pad built in so neither soft nor hard keyloggers can intercept it.
The weak point is you still don't know what you're signing, who or what you're authenticating, etc. if you don't have a trusted display and do the entire transaction on trusted hardware. (your host may be compromised, and could show arbitrary data, proxy connections, etc.) Protecting the key does go a long way, but even with a secure passphrase input and smartcard key operations, there is risk.
The ideal, IMO, would be some secondary hardware with a limited, secure OS which lets you put specific limits on how the key is used, showing a hash or identifying details of each transaction on the secure display, prompting you to verify each. E.g. to send a payment, the amount and payee (and date/sequence number) are shown on the secure display to verify before authorizing.
password is the weakest form of authentication on open systems i.e. ones which you have to hand over to monkey grade users.
If you have a reasonable password policy yourself, I very much doubt it will cause you any problems.
I've been attacked by these for 10 years, probably totalling millions of attempts and I have a user name in their databases but due to a sensible password, nothing has got in.
I tend to use PKI and keyboard interactive logins though.
You could also set up your security group such that incoming 22 is denied by default, and open it up to individual IPs on-demand using a script like this:
Or better yet, don't run SSH at all. Leave it to the pros.
I had an experience about a year ago on one of my home servers that convinced me of this. It was a Debian machine that I kept religiously up to date (daily cronjob and manual checking). It only had an external SSH port, I used log2ban, and disabled all logins except for one non-root user.
I got the common ssh brute force spam. Then one day I happened to be checking logs, and noticed rootkit logs in the /root directory. My server had been hacked! Fortunately whoever had broken in had completely botched it and left traces all over the system. It looked like they had given up at some point, despite having root access.
I later discovered that the attacker had used a 0-day openssh exploit that had been released that very morning. For that time period of 24-48 hours between release and fix, my server had been hacked. I only wonder how many other servers were hacked by skilled people that didn't leave evidence behind.
So morale of the story - you can't secure anything that is publicly facing on the internet. And if you really need to do it, it requires constant vigilance, DMZs, and all that other stuff that is not worth it unless you're being paid.
I agree - I did this a long time ago; while it's fun to go through logs sometimes, I prefer to weed-out the general bombardment. My hits dropped from about 90 a day to 0 (after 6 months).
A long time ago I used to have a general purpose server on a DSL line with port 22 open. I thought everything was fairly secure, patched up, etc... I never thought brute forcers would be able to guess one of my accounts (which was a friend's account actually). The kicker? The password was the username. I found out about the rootkit a few days after they got in. I tell ya, kids these days just don't know how to hide a backdoor.
A really good port knocker is your best defense against rogue attackers of an external service (because there is no defense against a 0-day). For an iptables-supported system I recommend Knockknock by Moxie Marlinspike (http://www.thoughtcrime.org/software/knockknock/).
It doesn't specifically target distributed attacks, but I'm not really sure how something like that could know whether incoming requests are part of a botnet or ligitimate logins.
Or use only public-key authentication. I've run many internet-exposed servers for years without problems or kludgy obscurity measures like port knocking simply by disabling all forms of password authentication. This is faster, easier and has the advantages of not requiring non-standard client software, being observable by an attacker on the same network, and avoids the sort of rote password entry which causes mistakes and discourages long, complex passwords.
You only need the ability to write to a file in your home directory. In the case of a VPS you could do that before editing the sshd config (in large environments this is trivially automated as part of your install process, which is why e.g. Ubuntu EC2 images ship with public-key-only SSH by default).
In the case of a shared server you can't disable password auth, which is a powerful argument for using a unique password since that system is far more likely to be compromised.
I think that a more reasonable explanation could be that the attackers were careless about building the dictionary. If it's built by parsing files and you feed the wrong file or the parser does not work correctly you end up with a dictionary with lots of bogus entries. That seems a simpler explanation than:
"The best guess is that these passwords were collected from an unhashed password database, or from a trojaned SSH server or client."
or
"This might be due to the brute force tool not properly interpreting comments in the dictionary file, or the attacker not understanding the comment notation"
It could be more haste than incompetence. Anyway, if you think about massive brute force attacks like this, they are 'silly' easy no-brain attacks more likely to come from script kiddies who downloaded a tool someone else wrote than real attackers.
However, I decided to run an additional experiment to contact Amazon since the IP was originating from an EC2 instance. Amazon contacted me after I filed an abuse report and said they were investigating.
A week goes by and I'm still getting hammered. So I email Amazon and asked when it will be resolved. No response. So I email again, again, and again. I finally get a response saying they have resolved it, but I'm still getting hammered from the same IP address.
So I email them once more asking - what is it exactly you have resolved?
No response.
I presume it is not in Amazon's best interest to resolve such issues as long as people are paying for their instances....