
Endlessh: An SSH Tarpit - stargrave
https://nullprogram.com/blog/2019/03/22/
======
xoa
Neat little project to needle some of the botters a bit (though I assume
they'll all evolve to recognize this fairly quickly). I'd echo tyingq's
comment below though:

 _Any tarpit has the potential to piss someone off. I 'd run it on a
sacrificial server with no obvious way to tie back to who is running it._

Yeah, just speculating but at least from what I've seen in the past if you
successfully tarpit some script kiddie and they notice the IP their scan got
stuck on there is some potential to move from "one of a billion random lowest
common denominator bulk scan targets" to "paid attention to specifically some
minimal amount", which is a genuinely different scenario. Even if all that
amounts to is a relatively low volume revenge DDOS pointed at you for a bit
it's still more of a disruption then if the auto scan had just moved on
without seeing anything of note in the first place. This looks like fun on
some systems, but on anything real I'm inclined to just stick to cutting down
on log spam via single packet auth or a port knocker or the like. The old
outrun-the-hiker-not-the-bear aphorism fits a lot of cases, just something to
keep in mind before implementing something like this if you aren't directly
experimenting with more active reactions.

Conversely as a research project I'm now actually curious what sort of extra
attention even something like this could attract. Maybe these days everyone
would just adapt and move on instead and the above is all obsolete?

~~~
imglorp
Good advice, being ready for a revenge DDOS, but I suspect most of these SSH
hunters are busy business people who have a quota of zombie hosts to fill for
whatever mining or spamming they're up to. DDOSing doesn't pay like those
things do.

~~~
btown
Even sophisticated/organized hackers have free time, and if someone "messed
with them" and caused them to need to push a hotfix at 6pm on a Friday because
their cluster got stuck on your tarpit, you can bet they'll give the tarpit
operator some of their attention. Woe to you if they happen to find something
that's not patched.

~~~
ianai
Turn it around. How would hackers adapt if nearly every host responded on ssh
the same way? Maybe it’d be behavior worthy of being in openssh or elsewhere.
Don’t make it cheap to cheat. This might not be the holy grail, but it’s a
good avenue.

~~~
gh02t
Seems to me like they'd just implement a timeout on waiting for the
identification string. I don't know the SSH protocol that well, but from the
article it sounds like it's easy to bypass this tarpit if you don't strictly
follow the SSH standard. I doubt a timeout would have much impact when
connecting to legitimate hosts since they probably respond quickly with their
identification, so the fix is easy.

------
LinuxBender
I like this! It's tiny, simple, yet effective.

    
    
        2019-03-22T19:54:06.303Z ACCEPT host=::ffff:196.52.43.xx port=50327 fd=4 n=1/4096                                        
        2019-03-22T19:54:38.838Z CLOSE host=::ffff:196.52.43.xx port=50327 fd=4 time=32.535 bytes=199
    
        2019-03-22T19:57:12.008Z ACCEPT host=::ffff:141.98.81.xx port=53646 fd=4 n=1/4096                                        
        2019-03-22T19:57:21.118Z CLOSE host=::ffff:141.98.81.xx port=53646 fd=4 time=9.110 bytes=30
    

The 141 bot never stays longer than 5 seconds. I'm guessing they are using
'expect' and waiting 5 seconds for a particular string.

The only thing missing is setting some glibc hardening options

    
    
        RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH FORTIFY Fortified Fortifiable  FILE
        Partial RELRO   No canary found   NX enabled    No PIE          No RPATH   No RUNPATH   No 0  8 ./endlessh
    

I modified one line in the Makefile

    
    
        CFLAGS  = -std=c99 -Wall -Wextra -Wno-missing-field-initializers -D_FORTIFY_SOURCE=2 -O2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -ftrapv -s -g -Wl,-z,relro,-z,now -Wl,-z,noexecstack -pipe -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection --param ssp-buffer-size=4 -fPIE -pie -m64 -mtune=generic
    

It's better now

    
    
        RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH FORTIFY Fortified Fortifiable  FILE
        Full RELRO      Canary found      NX enabled    PIE enabled     No RPATH   No RUNPATH   Yes 3  4 ./endlessh

~~~
dfox
Do you realize that the thing does not process any attacker controlled input
and thus all the hardening is completely pointless exercise?

~~~
joh6nn
The number of times in my life that I have seen "this should never happen"
scroll by on STDOUT suggests that it's time well spent.

~~~
dfox
If you cannot trust that 725 line program without any mention of recv() nor
read() will never call recv() or read() on network socket then you have much
larger reliability (not security, because whether something that is broken to
the point of being unusable is secure or not is completely irrelevant) issue
in your system that is not going to go away by applying random hardening
options to said program.

------
userbinator
This is the Internet equivalent of keeping phone scammers on the line by not
just refusing to hang up, but actively interacting with them until they hang
up (search YouTube for "tech support scammer trolling" for some
entertainment...)

That said, I'm a little confused even after reading the linked RFC paragraph;
it claims "The primary use of this feature is to allow TCP-wrappers to display
an error message before disconnecting." but I'm not familiar with that term
and there are no other occurrences of the word "wrapper" in that document.

 _Firefox and Chrome will spin on that server for hours before giving up._

For a web browser, this behaviour makes a lot of sense: timeouts are usually
based on "I've waited for X and haven't received anything", not "this
connection is like collecting water from a dripping tap". As long as there is
a slow but steady stream of content coming in, you wouldn't want to abort
until the user gives up or an internal limit on the data collected is hit
(which is probably what happened with the author's tests --- a quick search
doesn't yield the maximum size of the response headers browsers will accept,
but servers usually have a configurable limit of a few KB at most for the
request headers alone.)

~~~
schoen
TCP Wrappers was a popular access control mechanism in earlier decades:
[https://en.wikipedia.org/wiki/TCP_Wrappers](https://en.wikipedia.org/wiki/TCP_Wrappers)

It does a number of things in userspace (with a simple minimalist code base)
that people might nowadays more often do in a firewall.

------
EvanAnderson
I haven't used it in years, but Tom Liston's "Labrea" tarpit is an interesting
take on this concept:
[https://github.com/Hirato/LaBrea](https://github.com/Hirato/LaBrea)

It listens in your unused IP space and both tar-pits scanners and creates
actionable intelligence about scans against your hosts.

~~~
lucb1e
I love the fake arp responses, that's a brilliant way so you can run this on
any machine (doesn't have to be a router) and not worry about managing new
hosts (no black/whitelist to manage of which IPs are unused).

~~~
penagwin
I'm definitely not going to try this because it sounds like an easy way to get
in trouble, but would this work on a host like DO/linode/vultr, etc? Again,
this seems really easy to detect

~~~
LinuxBender
Yes be super careful with Labrea on VPS providers and dedicated server
providers. They tend to get upset if you are trying to pick up IP's. In most
cases it won't even work on KVM hosts.

I've used Labrea on unused address space to make ARIN/RIPE happy back in the
day and that was harmless, as entire /16's and /17's were unused. _They used
to nag about nothing in the /16's being pingable, so Labrea made it all 100%
pingable_

~~~
DKnoll
Why hold onto addresses you're not using? This is against the allocation
requirements and lead to the IPv4 exhaustion we have now.

~~~
zylent
If you have an ARIN allocation, there is no reason you would ever want to
decrease the amount of addresses you own. You own them.

~~~
def8cefe
Whether you own them or not is subject to a lot of debate and not clear cut.
If you owned them, why would you need to fool ARIN into believing they are
being utilised?

The agreement you make to get an allocation requires a certain level of
utilization depending on the size of the block to prevent squatting and
exhaustion of resources. This is a good thing, and if it was actually
respected we wouldn't be in a mad rush to migrate to IPv6.

~~~
rwbhn
"Mad rush" \- heh. I'd hate to see how long something rolled out at a
leisurely pace would take!

~~~
y4mi
> _IPv6 is intended to replace IPv4. In December 1998, IPv6 became a Draft
> Standard for the IETF, who subsequently ratified it as an Internet Standard
> on 14 July 2017_

it only took 20 years to standardize after all. we're seriously rushing here!

------
HocusLocus
I opened up port 22 once so that it connected (without crypto) and issued the
phrase,

YOU ARE STANDING AT THE END OF A ROAD BEFORE A SMALL BRICK BUILDING. AROUND
YOU IS A FOREST. A SMALL STREAM FLOWS OUT OF THE BUILDING AND DOWN A GULLY.

>

and if they typed in something it would continue to play the game. After
hundreds of thousands of automated SSH worm probes over several months, an
actual human did connect and play the game for a few minutes! But even this
triumph was short lived, they did not even play it to the end to see if
something interesting might be unlocked (it would have been an invitation to
contact my email address).

------
viraptor
If you don't want to waste resources and run a separate service, you can do a
very similar thing with just iptables. Look for the TARPIT target.

> Captures and holds incoming TCP connections using no local per-connection
> resources. Connections are accepted, but immediately switched to the persist
> state (0 byte window), in which the remote side stops sending data and asks
> to continue every 60-240 seconds. Attempts to close the connection are
> ignored, forcing the remote side to time out the connection in 12-24
> minutes.

~~~
craftyguy
I'm guessing this may not work, since OP's app sends every 10 seconds and
seems to imply that there is some timeout associated with receiving data from
the server, but not necessarily a timeout associated with receiving the
correct 'SSH-' string needed to move on.

------
JohnFen
This is awesome. Like the author, I run my real SSH server on a nonstandard
port. Unlike the author, I run nothing on the standard SSH port.

I think that I'll start running this there.

------
devy
On a different note, I love this other tool he wrote: Enchive

[https://nullprogram.com/blog/2017/03/12/](https://nullprogram.com/blog/2017/03/12/)

~~~
naveen99
Nice ! Reminds me of this:
[https://bitcointalk.org/index.php?topic=196378.0](https://bitcointalk.org/index.php?topic=196378.0)
using bitcoin keys for encryption in addition to handling money.
[https://github.com/jackjack-jj/jeeq](https://github.com/jackjack-jj/jeeq)

------
0db532a0
Wouldn’t cycling over the lines of an ASCII middle finger be more efficient
and equally as useful as generating random strings? The middle finger would
fit in a few cache lines at most.

------
cryptonector
Massively async ssh scanners won't care. They'll time out and move on without
being held up at all.

~~~
cosmin800
I can confirm that most ssh scanning attempts/scanners I've seen implement
timeouts and will usually move on after 30 seconds or so.

------
HashBasher
I wonder if you could make the attacker crunch out some shah-256 and you can
profit by mining Bitcoin.

~~~
Jach
Not bitcoin but forcing a proof-of-work scheme has been proposed as an anti-
DoS measure before:
[http://www.csc.kth.se/utbildning/kth/kurser/DD143X/dkand12/G...](http://www.csc.kth.se/utbildning/kth/kurser/DD143X/dkand12/Group5Mikael/final/Jonatan_Landsberg_and_Anton_Lundqvist.pdf)

~~~
simias
The original hashcash algorithm was meant to quench email spam which is not
completely unrelated.

But in this case I think the parent was proposing tricking the attacker into
computing a PoW without realizing it, which sounds pretty tricky.

~~~
koolba
Just need a protocol that allows for server initiated key renegotiation with
PBKDF. Then you trick the client into repeatedly calculating some new key.

------
iknowordidthat
Is a poll necessary?

Simple timer to count to 10. Go over all open sockets, drain incoming, pump
junk. Handle accept. Repeat.

------
ape4
Would be nice if it logged something fail2ban could act on. So it could ban
the IP-address - after bugging the bot for a couple hours.

------
oedmarap
Neat little tool! Since switching to Teleport I haven't had OpenSSH listening
publicly for a while, just reverse proxies. Still, I'll prob spin up a small
server for this to observe the patterns and IP locations.

Also, don't think that posting this blog article on March 22 didn't go
unnoticed :)

~~~
mises
What is significant about March 22?

~~~
berkut
SSH's default port is 22.

------
pmc
SSH brute-force attacks still pose a significant risk. See our related paper
on SSH auditing driven by attacker's attempts here
[https://pmcao.github.io/caudit/](https://pmcao.github.io/caudit/)

------
earthboundkid
I wrote one of these for HTTP:
[https://github.com/carlmjohnson/heffalump](https://github.com/carlmjohnson/heffalump)

------
nullc
I found that disabling password authentication makes ssh probing mostly stop.

Attackers that are smart enough to detect and avoid pubkey only hosts will
probably easily adapt to this program.

------
AlexCoventry
Perhaps this could be useful if you ran it on a vast number of ports. Just
running it on the one doesn't seem to make an attacker's life much more
difficult.

~~~
LinuxBender
That would be easy to do with an iptables DNAT rule. You can take a large set
of ports and forward them to this daemon.

    
    
        iptables -t nat -I PREROUTING -m tcp -p tcp --dport 1:79 -j DNAT --to-destination x.x.x.x:2222
    
        iptables -t nat -I PREROUTING -m tcp -p tcp --dport 81:442 -j DNAT --to-destination x.x.x.x:2222
    

...etc

Just remember to open those same ports in the INPUT rules. If you get DDoS,
then also create NOTRACK target rules in the raw table to avoid hitting the
conntrack table.

~~~
fxfan
You truly sound like a bender. Is there a quick start to iptables somewhere?
Is iptables the defacto network tool?

And can one prevent ddos using iptables?

~~~
LinuxBender
> And can one prevent ddos using iptables?

No, but you can minimize the impact or effectiveness of the attack. This is a
long topic however. Probably better suited to a blog post or a youtube video.

------
nullhead
Set this up on a server last night. In 20 hours I've got 75 connections. Took
a capture of trying to connect to it from my laptop:

[https://www.cloudshark.org/captures/b64150b66908](https://www.cloudshark.org/captures/b64150b66908)

IP addresses have been change to protect the innocent. This is from macOS but
definitely seeing some different behavior from other clients.

------
mehrdadn
This is great but what happens when clients start watching out for strings
they don't expect? Seems like only a matter of time.

------
damir
Awesome! Love reading hacks like these.

------
blastbeat
Assuming you run a legit SSH server on port A and an SSH tarpit on port B.
What hinders an attacker to connect to port A an B at the same time? What is
the advantage having an attacker connected for weeks on port B?

~~~
sm4rk0
They will usually try port 22 for SSH, only. So, if tarpit is on port 22 and
real SSH is on another - mission accomplished: you're making them (at least
one of their threads) wait.

------
z3t4
I can recommend setting up a server and do extensive logging. Then watch
hackers as they attempt to break in. I had a server hacked last week and today
i read the logs to figure out how they did. From first discovery, trying to
figure out what system they're on, to gaining a shell, and figure out how to
execute stuff, gaining network access. It was better then watching an actual
movie. And a humbling experience. But also entertaining, watching the struggle
against my hardening and even stumbled on a bug, feeling his/her frustration.
And the wow feeling when they finally got a shell.

~~~
acct1771
Postmortem on your blog?

------
tejtm
What a wonderful opportunity to rickroll ... to prolong detection time rot13
it or walk through all the ciphers you can get away with.

cause we are never going to give you up ;)

------
maga_2020
very interesting, and useful (including newbies like me who do not appreciate
the complexities and dangers of having internet-accessible IP address).

I wonder if security-oriented OSs like openBSD could offer something like this
out of the box. So that an admin can just say enable_tarpits='yes', and it
would automatically enable tarpits like this for all the currently installed
packages (with some default ports).

~~~
wahern
OpenBSD ships spamd, a greylister which also tarpits.

PF has the max-src-conn-rate state tracking option which is commonly use to
blackhole scanners:

    
    
      pass in on egress proto tcp to port ssh keep state \
        (max-src-conn-rate 10/60, overload <scanners>)
      block quick from <scanners>
    

Rather than blocking you could instead redirect new connections to a tar pit
by replacing the block rule with something like

    
    
      pass in on egress proto tcp from <scanners> to port ssh \
        divert-to 127.0.0.1 port 2222
    

where your SSH tar pit service listens on 127.0.0.1:2222. This is basically
the reverse of how spamd works, which diverts all inbound SMTP connections not
in the <nospamd> table to spamd. When spamd's greylisting rules are satisfied
it adds the sender's IP to <nospamd>.

IPv6 poses a problem for selective tar pitting, though. Realistically you need
to tar pit at least /64 subnets, but even then it's not difficult to get your
hands on a /48\. But if you lumped /48s together you'd have a huge false
positive problem. Also, AFAIU PF's state tracking capability can't track
subnets, anyhow.

------
GNOMES
I must not be understanding something...

Does this still allow whitelisted machines to connect, or is this just a troll
thing to do?

~~~
LinuxBender
I could see a few ways to use this. For me, what makes sense for my public
sftp servers is to put this on a higher port, then use ipset lists to
dynamically add bots to a DNAT rule and send them to this instead of the
proper sshd.

i.e. you get 2 chances to authenticate correctly, then I put you in this
hamster wheel for a day. Hamster wheels and intermittent fasting are all the
rage these days.

~~~
JohnFen
I was thinking of something similar, but in reverse -- turn this into part
tarpit, part honeypot.

I was thinking I could take the IP address of anyone who hangs out in the
tarpit for longer than a minute or so and automatically add it to my
firewall's blacklist.

~~~
1996
I used to run an honeypot.

For fun and giggles, I also kept the user and password they tried to see if
any of my systems was at risk

------
stcredzero
Could the return strings be generated by a Deep Learning powered fuzzer?

~~~
viraptor
What are you trying to achieve?

~~~
mises
He's trying to crash a script-kiddy's box with fuzzing. It's a fun thought,
though I'm not sure it's practical as you'd only get a few tries. Fuzzing is
typically an approach of throwing it the wall and seeing what sticks; if your
deep learning found something that crashed most ssh clients that would just be
a zero-day.

~~~
stcredzero
Maybe deep learning technobabble that might convince a script kiddie into
"researching" what was coming back?

------
egorfine
Beautiful!

------
allanrbo
Ewww, sticky service! Hilarious! :-D

~~~
ahazred8ta
Consistency is a virtue.

'Tacky' is a consistency. O:-)

------
ams6110
I'm not sure I like this. How secure is it? Anything you put on port 22 is
going to be attacked relentlessly. I think I'd rather have OpenSSH handling
those attacks than some tarpit program that may or may not have been subjected
to the audits and the trial-by-fire that OpenSSH has.

~~~
tyingq
Any tarpit has the potential to piss someone off. I'd run it on a sacrificial
server with no obvious way to tie back to who is running it.

~~~
sverige
Why worry about pissing off the kind of person who scans for open SSH ports
for no apparent legitimate reason? Are you concerned that the Chinese are
going to try even harder to hack your machine, like they're not already doing
everything they can to break in?

~~~
tyingq
They may be the same type of person that would launch a ddos, for example.

~~~
JohnFen
I'm not concerned about this, personally.

On my home server, there are only two ports open to the internet -- one for my
SSH server, and one for my VPN server. Both of those are running on
nonstandard ports.

In order to DDOS me, an attacker has to find those ports, which would require
a comprehensive scan (increasing the chances that they'll get blocked by my
ISP or detected by my antiscan monitor) or a huge amount of luck. And even
then, they'll just be able to DDOS my SSH server and VPN. That wouldn't be
awesome, but wouldn't exactly be disastrous.

~~~
VvR-Ox
You can change your ports but it won't prevent "Eve" from scanning for those
as you mention yourself. I think this belongs to the realm of "security
through obscurity" tactics and won't be safe in the long run.

I can prevent being blocked by your ISP by adjusting the timings etc. of my
scripts.

I think it's far more important to take a look at the supported key exchange
algorithms etc.

~~~
JohnFen
You are correct. This falls into the cost/benefit calculation of security. A
DDOS attack on my home network would be annoying, but not devastating, so I'm
willing to accept a little vulnerability to them in exchange for other
benefits.

> I can prevent being blocked by your ISP by adjusting the timings etc. of my
> scripts.

But that won't let you escape detection by my system's own defenses. Well, you
could if you slowed down the scan enough -- but "enough" is pretty darned
slow.

