

Who's your SSH buddy? - jgrahamc
http://blog.jgc.org/2011/07/whos-your-ssh-buddy.html

======
nodata
This is a perfect example of what not to do: the guy has ignored the lessons
of denyhosts (etc.) and came up with his own regular expression that doesn't
do the right thing.

Code re-use please!

~~~
ars
Or:

apt-get install fail2ban

Works out of the box to prevent brute force attacks against ssh, and can also
be configured for other services, like web authentication, POP, IMAP, etc.

~~~
xolox
Just a friendly tip: Try to verify whether Fail2ban actually works and keep
monitoring your log files (Logwatch). I once found some dumb ass trying to
brute force his way into my Postfix server, unfortunately Fail2ban didn't
catch it and I only noticed when Logwatch showed me the authentication
failures (several hours and thousands of attempts later).

~~~
nodata
Ouch. Did you tell upstream so that they could fix this for everyone?

------
rb2k_
> someone I could call and give credentials to so they could log in and
> shutdown the machine.

I'm not going to be able to read out my private key over the phone. I guess
this is only for password based authentication.

~~~
buro9
Why not just pastebin your private key, shortURL it and when the time comes
you only have to give the shortURL.

A key alone, with no idea what it will unlock, is going to be a useless thing
for anyone else.

Obviously I'm aware how crazy a public store of private keys sounds, but keys
are really only useful if you know what lock they fit.

~~~
Auguste
Just for the record, Pastebin is not completely secure. Even private pastes
can still be accessed if you know the URL - they aren't password protected
like your account may be.

But like you said, if you posted the key alone anonymously, it will be hard to
guess what lock it fits.

~~~
vog
_> Just for the record, Pastebin is not completely secure._

Note that it already happens that people put their private keys into pastebin,
by accident or probably by sheer incompetence.

 _> if you posted the key alone anonymously, it will be hard to guess what
lock it fits._

As an attacker, I would simply collect as many of those "published" private
keys as I can get. Then, when attacking a bunch of systems, I'd simply try one
key after another on each system.

The important difference to the "real world" is that an attacker can try lots
of "locks" at once (i.e. can connect to multiple target systems at once).

~~~
jrockway
_When attacking a bunch of systems, I'd simply try one key after another on
each system._

This was a good idea in 1990. But now you get IP-banned after three bad login
attempts, so you have to be smarter.

~~~
vog
Good point. However, that's what botnets are for. Why "wasting" those for
distributed denial of service if you could break in instead?

------
espo
My SSH buddy is iSSH from the App Store. Always available when I need him.

~~~
ceejayoz
Assuming you've got a data connection, that is.

------
forgotusername
This is a pretty pointless idea. If someone has root on your machine for even
500ms that's already too late to trust any of its state without a complete
reinstall (and even this is not enough for an advanced attacker).

Inevitably, if you have such a buddy, all they can really say is "yup, I'm not
the only user here", and you're back at square one.

A better investment of the time would be systematizing your configurations, so
in this situation you can just rebuild a new machine and kill the old one
(ideally, after figuring out how the password was compromised in the first
place).

~~~
ceejayoz
> This is a pretty pointless idea. If someone has root on your machine for
> even 500ms that's already too late to trust any of its state without a
> complete reinstall (and even this is not enough for an advanced attacker).

As the article says, he just wanted it shut down. Fixing it can come later,
but stopping it from sitting on your network sending out millions of spam
e-mails is still useful.

------
ErrantX
My custom monitoring program (probably not as advanced, it only does some
light monitoring) accepts a text message back to shutdown the machine.

Not perfect but can work well as a line of defence.

~~~
markbao
That's cool. How does it work? Is the text message just via email, and the
server can accept an email back with a specific command?

~~~
ErrantX
Servers are with Gandi - which has a hosting API. So... SMS gateway via some
shared hosting I have had for ages. This calls a script which figures out
which server I mean and uses the API to shut it down.

I did used to use Twilio until they dropped international text messaging etc.

~~~
wrl
What are you using instead of Twilio now?

~~~
ErrantX
A shared hosting account which has a free (5 credits a month) SMS gateway
included - I think it is reselling txtlocal.co.uk (who look quite good, but I
haven't gotten round to trying them out directly yet).

It was kinda thrown together in a few minutes after Twilio flaked :)

------
yock
> But it made me realize that I need to tighten up my SSH buddy plans for the
> next time.

I'm going to ask a really audacious question: Why?

His systems all worked as expected. Injection attacks were thwarted and login
attempts failed. He received (as it turns out, erroneous) notification about
the login attempts and knows he needs to do _something_ about it for the
future.

Why is that _something_ the sharing of credentials? Why is it that people
still allow for remote root login? Why do people still allow user SSH access
via password?

There's a better way, and it leaves you a damn sight better prepared for
intrusion attempts than receiving SMS messages that, as he so perfectly
demonstrated, were not actionable. Limit logons to PKI-only. Live happier.
Sleep easier.

------
davidandgoliath
Something to sincerely rethink is the idea of shutting the system down in the
first place. Call the network provider & have them unplug the ethernet.

Turning the system off could potentially reduce your ability to audit where
they originally got in. Anywhom.

SSH keys :)

------
hardy263
Though the main problem if someone breaks into your system, the first person
you'd suspect is your SSH buddy, because there is a possibility of them using
a public machine to log in. Who would want to carry the burden of being your
first suspect?

------
sixtofour
Many messages here like "why not do X?" or "why in the world are you doing Y?"
All good suggestions, but sometimes X or Y may not be good enough, or you may
have implemented them incorrectly (like the author's errant regex), or the
completely unexpected Z might happen; no one expects Z.

I don't see anything wrong with having a human backup. Heck, what if you die
and things need to be wound down?

~~~
peterwwillis
Dying is outside the scope of application. Please contact nearest heavenly
body for ticket submission process.

------
skrebbel
i don't get this. is it about a personal site? if so, how paranoid can you be?
or is it about a professional site? if so, who not try calling a colleague?

------
jerf
At the risk of talking about the actual topic at hand instead of diving off
into minutia about putting SSH on your phone, I'm not sure what this gets you,
from a paranoia point of view. Assuming the idea is that you want 100%
coverage of your server, this might marginally increase your odds, but not
really all that much. Times when you aren't available to administer your
server are likely to correlate highly with the times when your buddy isn't
available. And what if you'd never gotten the SMS in the first place because
you were entirely out of service?

One person simply can't cover a server 100%, adding a "buddy" doesn't help
much. Either you need a full-on netops operation, or you need to be able to
deal with not having 100%.

------
kondro
Umm... my iPhone, iPad and MacBook are my SSH buddies?

------
jarofgreen
Isn't the real problem that he only has 1 sys-admin for his servers (as far as
I can tell)? I mean he can have as many SSH Buddies as he wants but if the
alert comes in when he is asleep or drunk or whatever it's pointless.

Maybe the alert goes to multiple buddies who have the tech skills to handle it
... maybe he could be their SSH Buddy in return? In other words, rather than a
SSH-Buddy, have a pool of server sysadmin friends for emergencies.

ps. If the original poster is reading this, I would have posted this on your
blog but I didn't have other account passwords to hand ... if your moderating
all comments anyway, why not allow anonymous comments?

------
dmoney
Why not have the IDS shut down the machine itself?

------
antics
I can't believe no one has suggested this yet: this is the _PERFECT_ argument
for two-factor authentication. He's even already sending texts to himself.

For every new connector, send some sort of code to your phone, and require it
to be inputted back into the computer before any access is given whatsoever.
Summarily disregard every single connection and command given without this
code re-input.

You're even most of the way there: if you're sending messages to yourself, you
might as well implement this behavior with it.

------
tingletech
The unix on call operator in my data center is my ssh buddy

------
nhooey
I've had an "ssh buddy" for years, but for a variety of things such as asking
for a "download on how to hotwire a motorcycle".

I have an agreement with many friends to be an operator so they can look
something up on the web, or anything else requiring some efficient technical
prowess.

We just call up each other and say "operator", and if you're near a computer,
you help out with whatever it is.

------
koushikn
No ssh client still on iphones?

~~~
jgrahamc
I should have had one installed. Now I do. But I still need an SSH buddy
because I might not have data access (for example, if I am abroad).

~~~
kondro
It's expensive, but have data-roaming enabled on your phone. You'll only be
using it for emergency server-maintenance after all - and surely you can bill
it to your employer in those emergencies anyway.

~~~
jrockway
If this is for an employer, there should probably be someone monitoring the
servers when jgc is on vacation.

~~~
kondro
There are plenty of employers that only have a single sys-admin on staff.

------
naner
_Ultimately this turned out to be a false alarm. Although the machine was
under attack (on many levels: there was activity hitting the packet filter,
trying all sorts of injection at the Apache level and having a go at SSH) the
actual alert (based on looking in auth.log) was a false alarm based on a bad
regexp._

Any particular reason you didn't want to use Snort or Bro?

------
peterwwillis
Alternative Solution, with iptables and perl below.

You send the magic string to any tcp port and it'll instantly kill all SSH
logins and disable the root shell. Send the string again and the process is
reversed. Caveat: if your syslog contains trigger entries more than a year old
this will blow up.

I also recommend you disable root logins and password authentication, but if
you insist on enabling them, this may work for you. Modify as necessary.

    
    
      iptables -t raw -A PREROUTING -p tcp -m string --algo bm --string "_-()ThisIsAReallyLongAndComplicatedRandomStringToMatchOn()-_" --from 0 --to lengthofthelongstring -j LOG --log-prefix "29CharacterMaxTriggerString "
    
      #!/usr/bin/perl
      # sentinel - disable root based on a syslog trigger
      # Copyright (C) 2011 Peter Willis <peterwwillis@yahoo.com>
      use strict;
      use POSIX qw(mktime);
      my $LOG = "/var/log/syslog";
      my $LOCKED = 0;
      my $TRIGGER = "29CharacterMaxTriggerString";
      my %M = ( "jan"=>0, "feb"=>1, "mar"=>2, "apr"=>3, "may"=>4, "jun"=>5, "jul"=>6, "aug"=>7, "sep"=>8, "oct"=>9, "nov"=>10, "dec"=>11 );
      
      for ( ;; ) {
          sleep(1);
          open(F, "<$LOG") || die "Error: $!";
          for ( ;; ) {
              sleep(1);
              while ( <F> ) {
                  select(undef, undef, undef, 0.001);
                  #print STDERR "Reading \"$_\"\n";
                  if ( /^(\w+) (\d+) (\d+):(\d+):(\d+) \w+ kernel: $TRIGGER / ) {
                      my $time = time();
                      my $stamp = mktime($5, $4, $3, $2, $M{lc $1}, (localtime($time))[5]);
                      if ( $stamp > $^T ) {
                          #print STDERR "Found Trigger\n";
                          trigger();
                      } else {
                          #print STDERR "Error: found trigger but timestamp $stamp is before script begin time $^T\n";
                      }
                  }
              }
          }
          close(F);
      }
      
      sub trigger {
          if ( $LOCKED ) {
              system("/usr/bin/chsh -s /bin/bash root");
              $LOCKED=0;
          } else {
              my @procs = map { @_=split(/\s+/,$_); $_[1] } grep(/^root\s+.*sshd:/, `ps -aux 2>/dev/null`);
              #print STDERR "Killing processes: @procs\n";
              kill(15, @procs);
              kill(9, @procs);
              system("/usr/bin/chsh -s /bin/false root");
              $LOCKED=1;
          }
      }

------
modokode
Wouldn't a OTP scheme like OPIE be able to do this decently? My own computers
do not permit root login via ssh, but still it'd be more or less a trivial
matter to set up a local account for your "SSH buddy", and then over the phone
give him the next OTP in sequence for the root account or so for a successful
su - -c halt.

------
thehodge
Isn't there a service where you can SMS something like shutdown, restart or
other commands for the box to respond to, that would help in that instance
where net access is poor

------
tete
Also why would you allow either password based and root login when you are
"pretty careful with machine security"?

I had to smirk when I clicked on complex password scheme.

~~~
kahawe
I found that password scheme pretty OCD and not very secure considering once
you have that paper, you have a pretty good chance of very quickly brute-
forcing your way in.

~~~
scott_s
If someone breaks into your house or office to get that piece of paper, your
problems are bigger than a compromised server.

~~~
kahawe
But a piece of paper is easier to snatch than having your passwords written
down nowhere - without stealing a whole wallet or breaking into houses.

~~~
modokode
If having your password on a note is good enough for Schneier (
[http://www.schneier.com/blog/archives/2005/06/write_down_you...](http://www.schneier.com/blog/archives/2005/06/write_down_your.html)
), it's good enough for me. For pretty much the same reason. If you're going
to even use a password, it should be a good one. And if it's a good one, you
aren't very likely to be able to memorize it.

~~~
kahawe
With upper and lower case and "1337" speak thrown in between and a few other
characters replaced, it should not be that hard to come up with a decent
enough password that you can even remember.

And why not a password manager instead of paper? Would be even better if it
was one that reads your fingerprint or so.

------
moe
I think that guy needs to work on his paranoia.

~~~
kahawe
...or at least work on the security in a more well-thought way!

------
bahman2000
can you not ssh from your phone?

------
keyle
Any idea of what he uses to send SMS from his box?

~~~
troels
If you don't want/can afford an sms-gateway, you might want to look at notifo
[1]

[1] <http://notifo.com/>

------
urbanjunkie
Before you get into half-arsed solutions like having an ssh buddy, disable
root login and password authentication.

~~~
jgrahamc
I don't understand what's 'half-arsed' about deciding up front who I can trust
to have login credentials for my personal machines.

~~~
Zakuzaa
Prevention is better than cure.

~~~
jimktrains2
Preparation is better than being unprepared.

(Not that I'm arguing your point. Just taking all steps that you can think of
to prevent something, doesn't mean you'll actually prevent it. Being prepared
for a worst case is always a good idea.)

------
Kwpolska
I have a `poweroff' account on my machine with sudo poweroff as a shell with
possibility to use the command without a password.

------
gcb
now that your friend logged as root, you did have a breach.

------
AccordionGuy
SSH buddy? I don't even have a porn buddy!
<http://www.youtube.com/watch?v=mgnDHbeVGG4>

