Teleport is kind of big and sprawling. But they've repeatedly contracted Doyensec to do assessment work for it, and Doyensec is a fantastic firm. I think parked behind Tailscale, so none of your SSH infra is exposed to the Internet to begin with, it's a pretty great solution, and I'd do that again before I ever hand-tooled an SSH bastion host again.
so far I am mostly using tailscale + firewall. Using a firewall directly on the host as you mentioned seemed a bit dangerous - although we are trying it on a few servers. For now cloud provider firewall + tailscale.
Not quirky at all. Probably the aim is to inform the reader of the myriad things to do to keep bastion server secure, and then suggest there is an easier alternative. :)
Do not trust the firewall on the bastion host, if an attack can get into the bastion host, they can disable the firewall, so it cannot be used to limit egress. It's better than nothing, but consider using a firewall that's managed on a via a separate management network. I do agree that you should only allow SSH from a few known IPs.
Limiting the number of users is weird, and not recommended. Create all the accounts you need to provide individual accounts for the staff that need to access the bastion host, you will need that as things like HIPAA require named accounts for auditing. None of the accounts need any privileges other than the most basic. Users do not need sudo/root privileges on a jump host.
Other than those two complains, it's good recommendations.
A final recommendation: If you use AWS though, consider using Session Manager instead of SSH and drop the bastion host. You can still connect using the SSH command, using proxy command in OpenSSH, but no public IP or bastion host is required.
Yes, this. Also check out https://github.com/rewindio/aws-connect for a convenient wrapper around SSM to make it easier to use (I'm not the author).
This assumes that the attacker can get unconstrained root access to the system. It's fine to assume that attackers will but it's not as if you can't make that difficult.
Agree with the rest of what you said though.
- Shell compiled without built-ins
- No coreutils
- No sudo
- Root account disabled
- Read-only root filesystem
- No user home directories
- Destroyed and rebuilt from template every X hours on some maintenance schedule
Effectively, all you can do is ssh in, ssh out, and forward ports. It might be theoretically possible, but as far as I know, no one has ever compromised one, especially since you can already only get to the bastion from a government VPN anyway, and authentication to that requires a smart card, so there are an awful lot of things you need to compromise to get to that point.
This also answers the suggestion down the page of "why don't you just apply the same controls to every host and not have a bastion." Because the bastion is unusable and you want to actually use your other hosts.
Now, if you're just using a bastion as a jump host, you don't need to offer shells on it. Just allow people to proxy a port to behind the bastion and be done with it.
ForceCommand echo 'Nope'
ssh -J finaluser@finalhost user@bastion
There's also a lot you can do to harden that boundary. You can harden your kernel, you can execute user's shells in constrained environments like docker containers or restricted shells, leverage sandboxing technologies like apparmor or selinux, etc.
The user/root boundary can be a lot thinner than people expect, so I get why you'd want to point out that reliance on the attacker not escalating should be met with an evaluation of that boundary, but I think it may be understating the boundary to unconditionally not trust a host based firewall, or to say that getting onto the bastion itself is enough to disable the firewall when it does indeed require escalation.
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession"
IMHE, Bastion Hosts suck.
If you are forced to use one, send logs to a safer one-way storage encrypted and put tampering triggers everywhere you can in the Bastion Host. Also make sure you log outgoing connections. And make sure you can easily match incoming to outgoing.
If you absolutely have to use sudo on the Bastion Host force it to OTP only. Or if absolutely not possible, use 2FA, but this is a risk as something somewhere might not be properly protected and the password will leak. But the better way would be to have the bastion host run on some read-only image and not letting it upgrade or do any admin task at all. Maybe even remove admin users, SSH, the whole lot.
And related, do not have a single account with god-like access to everything. Isolate permissions. This is probably the hardest to get OK'd but it's the classic SPOF where they got you by the balls.
IMHE, Bastion Hosts suck.
The question isn't to replace, but to remove. If you apply the same security to the actual hosts (which you probably should anyway) then why have an intermediary?
Another really useful tool is logwatch.
I actually caught an intruder this way hijacking my system several years ago. They removed rkhunter, chkrootkit and a variety of log files. And, modified lines in the last logged in users log. But, a combination of logwatch and tripwire caught it.
Every time you build some infrastructure, you expend scarce resources like engineering effort (=opportunity cost), time, money, and complexity by adding moving parts to your christmas tree of technology. You should always critically evaluate what's the most low hanging fruit you can invest in for a given end goal (eg improving security) considering the complexity costs. SSH bastions can be worth implementing in some situations, but not top of the list in many cases.
The next sentence starts talking about "security compliance standards" - you sometimes have to submit to doing stuff for reasons of ticking boxes, but it's important to remember when you're doing what's best for security and when you're going through motions mainly to tick boxes for someone else.
# Configure idle time logout
ClientAliveInterval <value in seconds>
grep ^read /etc/profile.d/timeout.sh
It is not an idle timeout logout at all. Instead, it causes sshd to periodically send probes to the client. This has a couple of effects, most notably keeping tcp sessions "active" and frequently exchanging packets (this can be useful to keep connections through statefull firewalls alive if you are genuinely idle), and to rapidly detect and disconnect a client that has actually gone away.
I think the origin of this incorrect description is the CIS documents. They have the exact same gross mistake in them.
I think the ClientAlive probes are useful and should be on, but it's definitely not an "idle logout" as claimed.
Another risk with a bastion model is port forwarding. As an organization you have to decide what is appropriate for that bastion. Unrestricted forwarding? Restricted? Denied?
If your bastion is also used for automation to drop files into a staging area, you can limit that automation to file transfers and even limit what it may do with files. This prevents the automation from having a shell or performing port forwarding.
The keys should be outside of the home directories to prevent malicious tools from appending additional authorized_keys into the account. Make use of automation to manage key trusts and add a comment to keys to map them to an internal tracking system like Jira. This assumes your MFA/2FA is excluding specific accounts or groups via PAM and permitting the use of ssh keys with specific groups or accounts.
Match Group sftpusers
ForceCommand internal-sftp -l DEBUG1 -f AUTHPRIV -P symlink,hardlink,fsync,rmdir,remove,rename,posix-rename
Another thing mentioned in the article is iptables. In a PCI environment one may want to also have explicit outbound rules using the owner module to limit what users or groups are permitted to ssh out. So if your organization have a group of people allowed to use this host as a bastions, then one could write a rule like
iptables -I OUTPUT -m owner --gid-owner devops -p tcp --dport 22 -d 192.168.0.0/16 -j ACCEPT
Lastly I would add that bastions should have as minimal an OS install possible and have SELinux enforcing. Actions denied by SELinux should go to a security operations center after you spend some time tuning out the noise and false positives.
 - https://man7.org/linux/man-pages/man8/sftp-server.8.html
It would be interesting to hear what you think of Keykloak.
There's also one very important difference between those two:
- others can see your favourites.
- you can see both your upvotes and your favourites
so only use favourites for things you don't worry about others seing.
I don't know if this is important for you but for a lot of people here it probably can be.
CIS supported technologies: https://www.cisecurity.org/cis-benchmarks
CIS Audit AWS infra: https://github.com/toniblyx/prowler
Better to be proactive than reactive :^)
There are a few different players in this space, but the one to watch is Boundary by Hashicorp.
Basically managed authenticated proxy connections to any resource you could possibly need. Still young, so it's missing auditing and some of the convenience features, but give it a year and it will be a compelling open source competitor.
Teleport is great, but their centralized model is not suitable for all situations.. and the pricing (at least for kubernetes) leaves a lot to be desired.
There is also StrongDM, which is very similar with a better pricing model.
I am wondering why we need to configure this many steps as outlined in the article, and in general. What is the point of Teleport in the first place? Why is there no managed service that takes care of all of that, with me focusing on just deploying an app and running it in the VPC.
Can’t 99% of the use cases be put in a template and managed by a service provider, including the security?
these are entirely valid concerns. defense in depth, principal of least privilege. humans make errors.
- would you still require users connected to the vpn to go through a bastion host?
- would you ever run bastion/vpn through the same box?
- are there preferred access use cases for each?
You should generally do both things.
Wait, I should word that better. You should generally have both sets of controls: network access control with a VPN, and fine-grained, auditable SSH-level access control. I don't love the "Linux shell server" approach to providing those SSH controls.
I recently built an nspawn container with tinysshd server, with a .profile that execs telnet to the relevant system on login.
We had previously used an old version of Microfocus Reflections (terminal emulation) with stunnels deployed on all the clients and bastions. That was not containerized, but the server stunnels were set to chroot() on startup.
I recently was forced to support the latest version of Reflections, and since it doesn't support chacha-poly, I also built dropbear SSH server just for them. Reflections is very expensive (~$500/seat), and the best that it supports is aes256-ctr, using Tatu Ylonen's commercial ssh.com (which appears to be abandonware). I really hope we can get rid of that.
disclosure: we build SaaS on top of OpenZiti (the open source) so are opinionated in this domain. and, to be clear, the above is just one layer...other layers of security still apply.
still, having sshd listen on localhost and not a public ip is pretty cool imo. Ken and I did exactly that on a stream one day https://youtu.be/oSlwZcwZcsU if anyone is interested. The one extra step one could do is to convert sshd to only allow connections from localhost by editing /etc/ssh/sshd_config and set the ListenAddress to only 127.0.0.1
Make those bastions dark!
Sorry, one of those crappy it depends answers. The teleport node agents, the agent running on the server you want a session on, can be configured to listen to inbound connections from the proxy (but doesn't use port 22 by default), or can be configured in a reverse tunneling mode where it does outbound dialing towards the Teleport proxy service. When using the reverse tunneling mode, you don't need inbound access to the end nodes, but still need the nodes to be able to make an outward connection to the Teleport infrastructure.
This is how the cloud hosted Teleport works as well, we can't be expected to have outbound network access to peoples machines, so all the agents will dial the cloud hosted proxies, and setup reverse tunnels that are then used for the inbound connection requests.
In most setups though, the Teleport Proxies would then still have inbound connectivity and are meant to be internet facing, so a client can request an SSH or other session, but that single way into the environment can be hardened, layered with additional security, as the environment may require.
Note: I'm affiliated with Teleport, my comments are my own.
I'm trying to grok why they're better than SOCKS5 proxies... Is it because they provide shell access and a larger attack surface? ;-)
Many certifications or legal requirements demand that you log all changes to your systems, including administrative changes.
Bastion host are a well-understood (both by operators and auditors) way to implement that, so it's still a go-to solution.