I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits.
Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least).
Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
anyone who compromises [the VPN server] is in a pretty good position.
Sure. But without a VPN, anyone who compromises even one of your other hosts is in the same position. It's a lot easier to audit a single-purpose VPN server for possible security issues than it is to audit all the application code running on the rest of your production systems.
And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
I wouldn't. And even if you're right, getting a VPN login still doesn't get you anywhere. You still have to be able to ssh to the rest of the hosts. That's why we do security in layers.
Routing transparency. Sure, you can script a bunch of tunnels, but it's nice to handle routing at a lower layer. Having worked with both setups, I vastly prefer the VPN solution for ease of setup, use, and maintenance.
Also, different attack surfaces. Two layers of the same security measure (ssh) is, all else equal, not quite as good as two layers involving two different measures (VPN, ssh).
I have experience with many open source VPN servers. The purpose is a bit different -- we provide a VPN service to home users to encrypt their internet traffic. But the same problems should apply.
OpenVPN is the most compatible with a variety of clients. OpenVPN runs in userspace, so the clients for each OS and mobile platform interoperate well. The downside is, it does require a client program to be installed and configured. It's considered very secure, using SSL. Since it's userspace, moving large amounts of traffic means more context switching and higher cpu usage. Despite that, I've found it to be faster and more stable than the alternatives.
L2TP/IPSec is built in to most clients -- Windows, OS X, mobile. But every implementation is different and it's hard to configure a server to work with all of them. There are also more moving parts -- an IPSec server (openswan, strongswan, or racoon), and L2TP server (openl2tpd, xl2tpd) and a PPP server (pppd). IPSec seems to be a secure protocol but it's very complicated. I tend to distrust complicated security.
Pure IPSec has many of the problems of L2TP/IPSec with the added problem of difficult to configure in Windows and OS X.
PPTP is not performant or very secure. Other than the fact that almost every client supports it, I see no reason to use it for a new VPN.
Yes, I agree, I was just giving an example of how an additional layer can help protect against automated attacks, even for highly-secure services like SSH.
I also agree that SSH is less likely to have flaws than most VPN software. But on a properly configured bastion host, by-passing the VPN would just put you in a position where you can attack SSH. You would still need to by-pass SSH to access production servers.
Port scanners do generally scan non-standard ports too, you know...
I don't think that exposing SSH to the internet is that bad, but your argument is not sound - requiring a VPN does add security, because if there happened to be a vulnerability in it that allowed access, all it would do is expose SSH on the machines (I'm assuming you have proper firewalls set up), which you are advocating making public in the first place.
Saying it adds no security is false, because you'd require an unpatched vulnerability both in your VPN server and in the SSH server simultaneously. A zero-day in one is possible, but in both at the same time is far, far less likely.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited
So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.
When someone advocates using a VPN, that doesn't mean not using SSH too. VPN + firewall just restricts who has the potential to try to SSH to you, and provides additional protection and central access control/management.