
Bastion hosts - slem
https://www.pandastrike.com/posts/20141113-bastion-hosts
======
tptacek
Jump boxes are a good idea. A better idea, if you're in AWS, is to use a VPC
and just VPN into your environment; the VPN terminator in the VPC serves the
same purpose as a jump box.

If OpenVPN scares you, you can still do layer-4 filtering to minimize the
impact of losing the OpenVPN server (ie, to get it to "no worse than if we
hadn't used VPCs in the first place).

"Single-packet authentication" is, I think, silly, and I've never recommended
it to anyone. It's hard to think of a case where hand-rolled authentication in
front of SSH has saved anyone with a proper SSH configuration (keys-only-no-
passwords). You can skip that step.

~~~
phil21
> A better idea, if you're in AWS, is to use a VPC and just VPN into your
> environment;

Is it though? Is having your (and your co-workers) desktops/laptops/and
possibly more directly able to access production servers at a packet level a
good idea?

I would posit that it is not. A bastion host can be used to lock down more
than firewalls, and reduces the attack surface living on your network
considerably.

> VPN terminator in the VPC serves the same purpose as a jump box.

It certainly serves the same network separation purposes a jump box does, but
is not able to dictate the access environment nearly as much. If only your
jump box has access to servers (and production servers properly filtered),
every bit of remote access must go through that single ingress/egress point.
And that point can have security policy built-in as well for multi-person
teams. For example: I can enforce certain key types being utilized on remote
hosts, logging of ssh sessions, etc.

I also agree the "single packet authentication" is silly though. Most bastion
hosts these days will be behind VPN already. Just another riff on port
knocking really.

~~~
jpgvm
Bastion hosts are generally a bad idea. There is not a single implementation
of a bastion host that doesn't have significant security holes. This is doubly
true if you want to do proper group based access control.

Most notable of these issues is the nature of either needing SSH agent
forwarding enabled or storing a copy of the private keys on the bastion host
(please don't do that). There are ofcourse ways of dealing with this but they
aren't worth it.

It's fundamentally much better to just harden every system and implement
proper federated authentication and authorization with LDAP. It also makes it
much nicer to implement things like MFA and device level authentication.

Remember, it's very rarely that an attacker is going to be able to
successfully walk in or break down the front door. Your security holes are
very unlikely to be in your administrative channels like SSH/RDP but rather
crappy applications or services you may be running on your systems.

Those crappy bits of software + insider threat are the 2 main things you
generally need to deal with. Ensuring systems are sufficiently isolated from
one another helps with both of these and bastion hosts just add another layer
of shared stuff between workloads that should stay entirely separate.

~~~
tptacek
To a one, every medium- and large- sized application deployment I've tested
that failed to set up VPC (or something like it) has fallen to some stupid
oversight on some stupid backend, support, or dev machine. Don't rely on
individual host hardening for security; you'll fail.

~~~
jpgvm
I didn't say forgo the VPN. That would be unnecessarily increasing your attack
service. In fact I specifically mentioned doing whatever you can to increase
isolation between workloads, this includes ensuring access between dev,
support and production machines is either not possible or is via careful
controlled and audited means.

I only said bastion hosts are bad and generally decrease overall security
because it's almost impossible to craft one that doesn't create more security
holes than it closes.

------
lmm
Bastion hosts are the wrong approach, as are VPNs. They might have had some
merit in the days of handcrafted servers, where "hardening" was something you
could do to a single machine. But nowadays, especially on AWS, there's no
point in a configuration that isn't automatically reproducible - in which
case, why not harden all your machines?

Then you avoid having a single point of failure. If and when a single SSH key
is compromised, there are a limited number of hosts that key had access to -
and unless that's the log server, you have secured logs so you can figure out
what happened. Different keys can have different capabilities, rather than a
one-size-fits-all policy on the bastion. It makes it harder for an attacker to
know where to get the boss's keys. And you can expand your system
geographically without the performance compromises of a VPN; your servers
already make encrypted, secured calls to each other (over SSH, SSL or a very
small number of vetted protocols), expose only those ports that are actively
used. Your attacker could steal your entire system documentation and it
wouldn't make any difference, because the system is truly secured, not relying
on obscurity.

~~~
rudolf0
You're describing an overly optimistic scenario. "Why not just harden all your
machines?" Because it's more complicated than that. Yes, you can automate
basic hardening (configuration of access controls, limited use of SSH keys),
but how do you automate hardening of RPC between one server and another, or
hardening of those 3 internal administration web apps which may or may not
have some local file disclosure and crypto flaws?

Defense-in-depth is still the only sensible way to handle security. You do
what you can on the host side, but bastion hosts, jump hosts, VPNs, etc.
provide an additional layer to make lateral movement much more difficult for
an attacker.

------
neohaven
Direct quote from the page at the other end of this link :

"You appear to be using a browser that may not be able to display our site
correctly. Generally, this is beause it's an older browser that doesn't
support a lot of the newer, awesome features in open Web standards.

Rather than provide an untested and possibly subpar experience of our site,
may we suggest downloading one of these fine browsers? (We think you'll find
your overall experience of the Web will improve if you do.)"

This is followed with links for Firefox and Chrome. I am running Firefox 24.4
from CentOS repos. This is unskippable, and there's no simple way to bypass it
on the page. On a browser from March of this year.

Urgh.

~~~
cdubz
TBH, > 6 months old is hopelessly outdated. Welcome to the future, grampa.

~~~
vacri
The point of those LTS browsers is so that an employer doesn't have to watch
for UI changes in their browser every 6 weeks, and then retrain all their
staff when the browser moves things around or makes things look different.

------
peterwwillis
In the Enterprise world, jump boxes are more typically just a hack implemented
by IT workers who don't want to go about properly setting up a firewall and
802.1x/vlan/vpn/ipsec/etc to allow strict control to a protected network
resource. (Or because they don't have the hardware capability to do it, or
their network admin decided against it).

The only two proper excuses for ever using a bastion host are "I don't have
the money to do it right" and "I don't have the time to do it right."

------
corford
I'm a fan of the VPN terminator as jump point approach.

OpenVPN with tls-auth switched on is a pretty robust front door. If you
combine that with enforced key only auth for ssh, ensure all internal machines
run their own firewall and logically split different bits of your internal
network off in to their own subnets you've got a fairly secure foundation to
build upon. Another easy win is switching internal services over to TLS/SSL
only (where it makes sense/is possible to do so).

------
nullc
You appear to be using a browser that may not be able to display our site
correctly. Generally, this is beause it's an older browser that doesn't
support a lot of the newer, awesome features in open Web standards.

Rather than provide an untested and possibly subpar experience of our site,
may we suggest downloading one of these fine browsers? (We think you'll find
your overall experience of the Web will improve if you do.)

------
cthalupa
Jumphosts have their uses, but in this case, I really don't see it. As others
have pointed out, better yet is VPN only access.

As for the port knocking silliness, well, I don't see any reason for it.

[http://bsdly.blogspot.com/2012/04/why-not-use-port-
knocking....](http://bsdly.blogspot.com/2012/04/why-not-use-port-
knocking.html)

~~~
rsync
I've been doing this sysadmin thing for a long, long time, and have built some
large, high profile platforms,[1][2] and I can tell you that few things have
made me happier than port knocking.

Yes, it is a password with a tiny number of "bits". But nobody suggested using
_only_ port knocking, so that's not significant.

Yes, if my attacker is sharing the exact same NAT'd IP as me then they have a
window to attack ... my fully hardened server, which was _no worse_ than what
was presented to them before.

Finally (and this is what makes me happiest) my sshd is not exposed to
whatever new exploit might be discovered tomorrow that doesn't require user
credentials. It's not exposed to being DOS'd by a buffer overflow vuln. It's
not filling my logs with brute forcers all day and night.

I love port knocking.

[1] JohnCompanies (first VPS provider)

[2] rsync.net

~~~
peterwwillis
Mr. Sysadmin, I have four questions:

1\. When was the last time your sshd was hit with a 0day?

2\. Why aren't you running grsec to prevent buffer overflows (amongst other
things) ?

3\. Wouldn't it be easier to DOS your host with excess traffic (like all
modern DDOS attacks), rather than find a remote-code-execution buffer overflow
0day and use it solely to starve resources on a bastion host?

4\. Why don't you use a VPN? If you used a udp-based VPN nobody would even
know the port was open (connectionless) and you could even require a preshared
key, which is the more secure equivalent of port knocking.

~~~
rsync
The answer to #1 is "never". That's no reason not to be aware of the threat
and to guard against it.

What is the state of grsec on FreeBSD ? I'm not sure...

Answer to #3 is yes.

I don't use a full-blown VPN in a lot of cases because it's a lot of
complexity that isn't needed. knockd is extremely simple (look at the source)
and simple to use.

\---

This is the way all discussions about port knocking go:

"I like port knocking"

"But port knocking is not the christ child that will deliver us from all of
the evils of the world!"

No, it's not. It simply adds a small amount of incremental security at almost
zero cost. All else being equal, my sshd was visible before, and now it's not.

~~~
peterwwillis
Why do you encrypt your traffic?

If the port knocking feature exists to "hide" your network service, you're
saying the security of your service is based on the idea that nobody can see
your traffic.

But if someone _could_ see your traffic, and therefore your connection, and
thus the port your service runs on, they could also clearly see the contents
of your traffic. So port knocking would be useless, and everyone could see
your traffic, and probably be able to do anything they wanted with it.

But since you use port knocking, you clearly believe nobody can see your
traffic. So it then follows that you don't need encryption.

So clearly you should be using telnet.

~~~
e40
_So clearly you should be using telnet._

This is just an absurd statement. You have conflated to completely separate
issues.

 _you 're saying the security of your service is based on the idea that nobody
can see your traffic._

No, it's because no one can find the service, not see the traffic.

~~~
peterwwillis
Okay, let me see if this explains it better:

1\. Anyone who can see your traffic can see where the service is.

2\. We assume anyone can see your traffic because you're using an encrypted
connection. (The conversation should have stopped here, but i'll continue
anyway...)

3\. The only thing port knocking does is obscure your service from port
scanning.

4\. The equivalent method to preventing someone finding your service via port
scanning is changing the port number.

5\. Changing the port number prevents all non-direct-attacks from finding your
service. (Nobody scans every TCP+UDP port of every IP on the internet for one
service, because 563.6 trillion network packet send+recvs takes a long time,
which doesn't even factor multiple tries or error retransmissions).

6\. A direct attack by a committed attacker will eventually find your network
service.

7\. Port knocking does not add to security. It _partially_ obscures
information, but it does not prevent access, and it operates on faulty
assumptions about network security; the information is still there, you just
have to look for it differently. It is less reliable than even IP
whitelisting.

8\. In the real world (and not the "gee I think this sounds more secure"
theoretical world) every single other network service you have will be
exploited before OpenSSH. So trying to hide OpenSSH while not hiding your
other services becomes farcical.

~~~
e40
_1\. Anyone who can see your traffic can see where the service is._

Dude, right there you lost me. 99.99999999% of the internet can't see your
traffic. The NSA can. Your ISP can. Random threats on the 'net at large
__cannot see your traffic __.

~~~
peterwwillis
You are radically misinformed. As a hacker, I have about a dozen ways to see
your traffic. We don't encrypt credit card transactions because we're afraid
the NSA or your ISP's network admins might pick up your card and buy some
sneakers on Amazon.

Ways I can see your traffic:

    
    
      * fucking with bgp
      * fucking with routers in general
      * fucking with switch port tables
      * fucking with proxies
      * spoofing
      * sniffing wireless networks
      * sniffing wired networks
      * tapping a cage backbone
      * rooting an isp
      * rooting a dslam
      * rooting pops
      * rooting firewalls
      * rooting endpoints on management ports
      * abusing hosted networks
    

The single easiest way to view someone's traffic is to use the completely
fucked-up retarded configuration of an intermediary router. Do you know how
many routers sit between you and your destination? Do you know how many of
those have 10-year-old configurations enabled via telnet on publically-
routable interfaces with default snmp community strings and 6-character
passwords?

Don't assume that just because you aren't aware of a particular attack that it
doesn't exist or doesn't apply to you. Hackers are only limited by their
creativity.

Please don't trust the network.

------
kev009
This article goes from mediocre to bad when it starts advocating security by
obscurity.

~~~
the_french
Security by obscurity is perfectly fine when it's part of a layered approach.
It will help keep attackers from learning the details of your architecture and
will also stop skiddies in their tracks.

~~~
arde
This. Obscurity has too much bad press. It's not to be depended on, but it can
be a useful addition sometimes.

------
slem
Bastion host(s) are a useful and important component of a system management
infrastructure. A bastion-host, in this context, is actually more properly,
but more obscurely, called a jump server. In this post I will simply use the
term bastion host. It is the most commonly used term for the system's
function: a server, which has undergone security hardening steps, that is the
operational and administrative control point for systems and hosts in a
datacenter (or AWS Region).

