
What SSH Hacking Attempts Look Like - iou
https://medium.com/@dmrickert/what-ssh-hacking-attempts-look-like-8f698e70a4f5
======
ejrv
One thing I've found amusing is you can cut down SSH hacking attempts to
almost zero (at the cost of compatibility with legacy systems) by using very
progressive encryption. If you force use of ChaCha20/Poly1305, ED25519 and so
forth in sshd_config, you'll see these attempts almost disappear. There's
probably an observation to be made about just how overwhelmingly many of these
attempts come from fairly old software setups.

~~~
maxk42
Good info! Got any good resources on configuring sshd algos?

~~~
nirv
I highly recommend Mozilla's OpenSSH configuration guideline[1].

[1]
[https://infosec.mozilla.org/guidelines/openssh#Modern](https://infosec.mozilla.org/guidelines/openssh#Modern)

------
tempfs
Some notes from an InfoSec person.

\- Have a stand alone SSH server that is something like a R-Pi[type B] running
an OS that gets patches regularly via unattended-upgrades and reboots itself
at least once a week.[could also be a minimal VM like AlpineOS if you need
Gbps+ line speed]

\- Have this R-Pi and your network gear plugged into a UPS that can withstand
at least a couple of hours of power outage.

\- Use non-standard ports on your perimeter FW for forwarding to the R-Pi.

\- Run fail2ban or something similar. Set a bantime of at least an hour[3600
sec] after no more than 5 attempts in 600 sec. This is mostly to discourage
anyone who stumbles across your perimeter listening port.[which enough will]
Password login will be disabled anyway, but a bot might not check for allowed
auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>]

\- No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no].
Use public key based authentication.[PubkeyAuthentication yes] Put the public
SSH key[id_rsa.pub] of any machine that you wish to authorize into
~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-
authorize remote machines[with particular users] at will.

\- add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config

\- No root logins allowed![PermitRootLogin no]

\- Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed
to access to SSH.[whitelisting]

\- Periodically review your logs of perimeter FW, R-Pi FW and
/var/log/auth.log to see what's going on. It should be pretty quiet, but look
at them anyway once a week.

\- Sleep well knowing that you have a simple, layered and robust defense
strategy that also has power-event survivability.

Others have mentioned port-knocking which is a cool trick but not something
that I typically use in an actual daily defense strategy because I'm not sure
how much value it really adds.

Don't use your perimeter device to host SSH services. It will not get patches
fast enough when vulns come up.

~~~
rsync
"Others have mentioned port-knocking which is a cool trick but not something
that I typically use in an actual daily defense strategy because I'm not sure
how much value it really adds."

I don't know how much it adds, but it is non-zero.

The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero
(other than your own use).

Port knocking has no place in security _by itself_ but I think it's a
wonderful addition to a layered defense - my favorite one, in fact.

~~~
bitbang
Try using fwknop. Similar concept to Port knocking, but much more secure.

------
ransom1538
# cat /etc/ssh/sshd_config | grep PasswordAuthentication

PasswordAuthentication no

^ it should return that. The end.

~~~
jgtrosh
Is it worth mentioning useless uses of cat in this day and age?

~~~
sametmax
Please educate me

~~~
pmalynin
</etc/ssh/sshd_config grep PasswordAuthentication

~~~
sametmax
I've been taught wrong and I've done it wrong for 15 years O_o

Thank you.

It reminds me of the idiom of putting "# - _\- coding: utf-8 -_ -" at the top
of Python 2.x files. Which is incredibly useful, but nobody remembers, and
googles everytime.

This comes from the fact the first famous tutorials on Python where written by
emacs users, and this editor recognizes this idiom.

But fun fact, the regex matching this line is "^[ \t\v] _#._?coding[:=][
\t]*([-_.a-zA-Z0-9]+)" (see
[https://www.python.org/dev/peps/pep-0263/](https://www.python.org/dev/peps/pep-0263/))
and accepts many variations (including a vim format).

The simplest variation is the dumb "# coding: utf8", which is not only
straightforward and easy to remember, but way less magical for anybody reading
the code.

Bottom line, if you are stuck in 2.x, use "# coding: utf8".

------
lol768
Fairly standard stuff, definitely interesting to see all the IoT credentials
attempred.

I'd recommend SSHGuard over fail2ban though, I seem to remember the version of
fail2ban in the Debian repos completely choking on IPv6 and failing open which
is obviously undesirable.

~~~
jazoom
I feel like fail2ban failing open is a good thing. If it fails closed you
can't SSH into the server. If it fails open it's just as though you aren't
using fail2ban.

Actually, my biggest fear with fail2ban is it failing and locking me out of a
remote server. If it indeed always fails open then there's no reason for me
not to use it.

Though, I don't know how it could be guaranteed to always fail open.

~~~
lol768
> I feel like fail2ban failing open is a good thing. If it fails closed you
> can't SSH into the server. If it fails open it's just as though you aren't
> using fail2ban.

But it fails open whilst you _think_ there's a layer of protection there.

> Actually, my biggest fear with fail2ban is it failing and locking me out of
> a remote server

Most of these tools will let you whitelist an address - and presumably there's
usually a way back in via a serial console/IPMI-KVM interface?

~~~
jazoom
>But it fails open whilst you think there's a layer of protection there.

That's pretty easy to test for.

>will let you whitelist an address

I don't have a static IP address.

>usually a way back in via a serial console/IPMI-KVM interface?

I haven't been able to do this at my provider when password login is disabled.
Maybe I just don't know the way.

------
ktpsns
I wonder why the author set up a dedicated SSH python server which mimics some
unix userspace as a honeypot. Why not a VM with a real standard widespread
linux distribution to see what attackers actually do? Obviously their codes
are not very sophisticated that they stop when "uname" fails with certain
arguments.

~~~
iforgotpassword
Because it's much easier to set up? It's not breaking after any session no
matter what the attacker does. Every session is isolated if several run
concurrently. Getting this with a VM involves quite some effort. Then you need
to make sure the attacker cannot use any commands to cause external damage but
still make them seem to work. But the latter is also a problem with the
simulated environment. It's quite easy to detect if you're not just blindly
sending automated keystrokes without checking the feedback.

~~~
ktpsns
Objectively there should be no difference between a honeypot VM and an actual
"productive" VM. I acknowledge that it is much harder to setup, maintain and
supervise a network of honeypots VMs instead of a dedicated isolated honeypot
server process which just simulates a system. However, as mentioned in other
comments here, there is no gain of knowledge if you don't simulate a whole
system but only parts of it.

------
gsich
Changing the SSH port proved to be most successful. Yes, of course you can
find the port with a port scan. But it keeps the logs clean. And I don't want
to waste CPU cycles on some brute-forcing idiots.

~~~
gregmac
(I'm primarly a developer, not sysadmin)

Is using non standard ports a reasonable approach? On one hand, it's kind of a
security-by-obscurity measure, and it's also a (very minor) inconvenience to
real users. However if it's not being used in place of other reasonable
security measures, I'm not really sure what's bad about it, but it does feel a
bit janky to me.

Same goes for other non-public services such as VPN.

What ports should be used? I think clearly you don't want to use other well
known services (eg, running ssh on port 25 is just going to fill your logs
with failed connection errors). Should you use ports >1024? >10000?

~~~
teddyh
> _I 'm not really sure what's bad about it_

You say it yourself:

> _also a (very minor) inconvenience to real users_

I would disagree with the “very minor” part. To paraphrase myself
([https://news.ycombinator.com/item?id=6617312](https://news.ycombinator.com/item?id=6617312)):

As I understand the argument, it’s “Changing port number add security,
therefore it’s a good idea.” I think nobody argues that it _adds security_.
The problem is that:

1\. It adds _very little_ security: 16 bits is not much, and the result is not
256 bits (say) of SSH key plus 16 bits equals 272 bits, but instead
effectively still 256 bits, or 256+8×10⁻⁷³ bits.

2\. The security it adds is itself bad (sent in cleartext, easily brute-
forced)

3\. These problems stand against the many drawbacks of this previously
discussed (complexity, confusion, etc.).

And the final argument: If increased security is what you want, _simply
increase your key lengths and /or password lengths_, and you will get much
more than 8×10⁻⁷³ bits of security, without any of the above problems.

~~~
blattimwind
You're looking at this the wrong way, because you seem to assume that non-
standard ports adds to security the same way the key length does, which is not
the case.

~~~
teddyh
I can’t think of any way it could be otherwise, but please, feel free to
enlighten us all how a non-standard port number adds any more security than
would an additional 16-bit key.

~~~
blattimwind
Non-standard ports, port knocking and similar things work before anything
reaches sshd; so they can mitigate vulnerabilities in sshd. Obviously a non-
standard port is the weakest fellow in the bunch, but that does not mean the
effect is zero. An extra 16 bit key length would obviously never mitigate a
sshd vulnerability.

An entirely different effect has been brought up by others already; a non-
standard port is enough to evade ~99 % of all automated attacks, therefore you
can run at higher log levels and might notice actual attackers earlier.

~~~
teddyh
> _Obviously a non-standard port is the weakest fellow in the bunch, but that
> does not mean the effect is zero._

Like I said, nobody argues that it adds _zero_ security, but that, for all the
hassle it introduces, it adds _way too little_ security to be worth it. 16
bits is _nothing_ , and can be brute forced quickly.

Regarding port knocking: I’m assuming that most people don’t write their own
port knocking software, but use something standard. The configuration of any
port knocking scheme is equivalent to an additional separate key of length
log2(number_of_possible_port_knocking_configurations). What is the effective
“key length” of port knocking schemes?

Also remember that the “key” of port knocking, just like a non-standard port
number, is transmitted in the clear, so anybody listening to the traffic can
see it. (And if you assume that the attacker _can’t_ listen to the traffic,
why are you even using SSH instead of something simpler like telnet?)

------
vbezhenar
I never understood why people would use fail2ban or similar approaches. Just
set proper password and move on. SSH traffic is tiny, it's not a problem.
You'll never be hacked with a proper password. Let them waste their time.

~~~
vxl
The failed attempts make it harder to monitor for other attacks because of the
noise in log files, network traffic, etc. and if an attacker IP is blocked
early they can't try more effective attacks.

~~~
Krasnol
I've set up a second obstacle by allowing access to the device from a certain
range of IPs on the router. Cleaned up the log pretty good.

~~~
seldomcomment
logins can be delayed [0] and TOTP can be used in conjunction with SSH
(without google authenticator)

[0] [https://unix.stackexchange.com/questions/105553/how-to-
provi...](https://unix.stackexchange.com/questions/105553/how-to-provide-
login-delay-in-ssh#105559)

edit:

quick and dirty notes of my setup [1]

[1] [https://pastebin.com/yYzSrM61](https://pastebin.com/yYzSrM61)

------
daveguy
I wrote a little utility to keep ssh ports off unless a totp packet is sent to
a UDP port. I haven't worked on it much since the initial proof of concept.
There are several things that could be done to improve it (open only for the
address that sent the packet, symmetric key for safer secret distribution,
etc). Shared key would involve encrypting time and only open if the decrypted
time matches current.

The totp packet is a full hash of time + shared code so brute force is
completely infeasible.

It's currently in python, it would be much better in something like go with
daemon monitoring. It is very simple, and it practically eliminates attempts.
If it became widely known there would probably be attempts on the udp port. An
additional improvement would be randomize the listening port based on the
time+hash:

[https://dhj.io/posts/2017/05/16/totp-
knock/](https://dhj.io/posts/2017/05/16/totp-knock/)

~~~
finnthehuman
In case you were unaware and didn't just want to write your own, fwknop is an
off the shelf tool to do the same.

[http://www.cipherdyne.org/fwknop/](http://www.cipherdyne.org/fwknop/)

~~~
daveguy
I appreciate the reference.

I looked at that before starting and reference it in the blog post (but should
probably include a direct link to the software rather than the original
concept post and article that include links).

The similarities are they both use a single packet and they are both an
improvement on port knocking.

This is a simpler solution (pairing UDP with TOTP). No TCP connection is
required. Making it ip address specific would prevent replay attacks (although
even without address specific the replay would only be valid for ~30s).

Another benefit is that a port scan will reveal no ports open (because it is a
one-way UDP based protocol).

fwknop is definitely a polished solution where this is just a proof of concept
for the TOTP/UDP as a shared key knock.

Thank you for the feedback!

EDIT: I updated the post to include a direct link to fwknop rather than just
links to the posts/articles about it.

------
arca_vorago
My current take:

nftables instead of iptables

port-knocking

non-standard port

key+pass access/auth

ip whitelist

good logging

ED25519 wherever possible!!!

~~~
Artemis2
I love the idea of Single-Packet Authorization with fwknop instead of port
knocking:
[http://www.cipherdyne.org/fwknop/](http://www.cipherdyne.org/fwknop/)

~~~
AlexCoventry
How does fwknop compare to knockknock?
[https://moxie.org/software/knockknock/](https://moxie.org/software/knockknock/)

~~~
michaelrash
The design decisions that govern fwknop provide guidance on how fwknop is
different from knockknock: [http://www.cipherdyne.org/fwknop/docs/fwknop-
tutorial.html#d...](http://www.cipherdyne.org/fwknop/docs/fwknop-
tutorial.html#design)

~~~
AlexCoventry
Thanks.

------
kuschku
This all is easy if you only get a handful of login attempts. But when you get
more, and more, it becomes an issue. On one of my servers it got so bad that
SSH was at the end constantly using an entire CPU core.

I don’t have logs from that server, but here are logs – just the failed
attempts – of 2 months from a server that was less severely affected:
[https://s3.kuschku.de/public/failed_ssh](https://s3.kuschku.de/public/failed_ssh)
[327M]

And that was despite already blocking massive areas of IP space already.

~~~
larkeith
Fail2Ban and psad will do wonders for that.

~~~
kuschku
Yeah, I now simply removed 0.0.0.0/0 dport 22 from the whitelist and only
added a select few IP ranges. That has calmed the storm significantly, and if
an IP from this range (my ISP) tries it anyway I can contact my ISP (a small
local ISP) to take that down.

~~~
larkeith
That's even better, if you don't need to access from arbitrary IPs.

------
deftnerd
My servers generally have an IPv4 address and a whole IPv6 subnet. I like to
disable SSH except on a specific IPv6 address.

Many botnets just focus on IPv4 and there is a lot more territory to scan on
IPv6. I usually couple that with moving to a non-standard port, enforcing
modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc

------
m0nty

        $ grep -c sshd /etc/hosts.deny
        1192
        $ uptime
        11:58:51 up 327 days, 21:33,  1 user,  load average: 0.13, 0.11, 0.09
    

I'm using DenyHosts for this, there are alternatives but this works for me.

[http://denyhosts.sourceforge.net/](http://denyhosts.sourceforge.net/)

~~~
plange
you should probably upgrade that kernel

~~~
m0nty
If only there were some way to upgrade the kernel without rebooting. What a
god-send that would be.

~~~
thenickdude
The Spectre/Meltdown patches required a reboot.

~~~
m0nty
Not all architectures are affected by it.

------
gist
I think that by using obviously easy passwords anyone who really is looking
for something is going to go to the next target. After all impossible to
believe anything of value is behind 1234567 or changeme as a password. Even
'baseball' makes slightly more sense. Besides someone running an ssh server is
not exactly mr computer newbie. Maybe a desktop you pick out of the trash
might contain something interesting if that was the password.

In a classic CIA honeypot I would imagine the women are carefully matched to
the target. If you are "James Bond" you get a better looking woman to try and
trap you. If you are "Wallace Shawn" (in looks) you would probably figure out
really quickly it's a trap if the woman looked like Angelina Jolie.

------
kamaraju
I found fail2ban is necessary but not sufficient. It requires some initial
number of failed attempts before the attacker is blocked.

To prevent even those initial attempts, I use multiple layers of defenses such
as country based blocking or blacklist based blocking. If an attacker slips
through despite all this, I blacklist them manually in /etc/hosts.deny.

If there is any interest, my scripts are in
[https://github.com/KamarajuKusumanchi/hosts.deny/](https://github.com/KamarajuKusumanchi/hosts.deny/)

------
codetrotter
> It absolutely shocked me that it is worth the time of these drive-by attacks
> to try combinations like root/password, root/root, or root/admin. Apparently
> they have enough successes using those incredibly insecure combinations that
> it is worth their time.

It is fairly evident to anyone that has looked at the traffic that a honeypot
gets that most of the activity is automated.

Most of it is purely botnets, some of it is automated password guessing
followed by manual login. I think the amount of people manually trying
passwords is insignificantly small.

~~~
highesttide
I think they mean "time" for the botnet in this case

------
znpy
I am slowly closing all non-vital ports on my router and just letting services
accessible from within the VPN.

No open port = no hacking attempts.

Investing a week of on-and-off studying and tinkering with openvpn is really
paying off.

~~~
zAy0LfpBZLC8mAC
> No open port = no hacking attempts.

That's just obviously nonsense? Closing the port does not change anything
about the attempts. Nor about the success rate of the attempts, if you aren't
being an idiot with insecure passwords.

~~~
fencepost
_Closing the port does not change anything about the attempts. Nor about the
success rate of the attempts, if you aren 't being an idiot with insecure
passwords._

How about "No open port = no concern about possibly vulnerable services
running on open ports"? I actually worry less about passwords and more about
overflows, protocol problems and parse errors these days.

~~~
zAy0LfpBZLC8mAC
Yeah, sure, but (a) then those problems potentially also affect the VPN
endpoint, so it's still a trade-off (b) much of the complex protocol machinery
often is behind the authentication barrier, so the risk of just exposing the
port to the public isn't necessarily that big, and (c) a firewall doesn't
necessarily protect your vulnerable service, often there are browsers on the
inside that an attacker could use to access those "protected" services.

My point isn't that blocking off ports at the firewall is always pointless,
but that it's usually a trade-off, and keeping a vulnerable service running
behind the firewall can still be a risk, and many "hacking attempts" are just
irrelevant if you follow general best security practices, so it's pointless to
do anything specifically to prevent them.

------
jlgaddis
Someday I'm going to put all the different pieces together and have things
like SMTP & SSH auth failures, hosts hitting my spamtraps, bots scanning for
open 23/TCP, and so on, to where I end up with one central host collecting all
those events and injecting /32 routes for those IPs towards null0 in my core.

~~~
jacquesm
A friend of mine built this complete with a whole pile of honeypots and lots
of companies use it.

[https://www.blockedservers.com/](https://www.blockedservers.com/)

Super useful.

------
fareesh
How configurable is fail2ban? Is it service based? Am I in danger of locking
out a legitimate user like someone who makes 25 ajax requests within the span
of 3 seconds? Can I make it work for authenticated HTTP endpoints as well?

------
WillieStevenson
Long ago I made the same thing.

[https://livesshattack.net](https://livesshattack.net)

------
jwilk
The numbers on [https://cdn-
images-1.medium.com/max/1600/1*47UCBwMdQGFNtk2ao...](https://cdn-
images-1.medium.com/max/1600/1*47UCBwMdQGFNtk2aoXB5gA.png) look very
suspicious.

~~~
raimue
Looks like the graph was generated from the Top 10 only. Obviously every item
appears exactly once, so they all have the same 10%. The presentation as a
graph is useless.

~~~
larkeith
More notably, two slices are listed as 11%, for a total of 102% - in a pie
chart!

