
Show HN: A proactive way of securing your Linux server - prashantgupta24
https://github.com/prashantgupta24/firewalld-rest
======
kelnos
This just feels like moving ssh auth to another port, using an obscure
"authentication format" that nearly no one uses.

If you're using pubkey or certificate auth, have disabled password auth, and
restrict what users are allowed to ssh in, this just feels like added
complexity and points of failure (not to mention a possible source of crypto
vulnerabilities) with not much added benefit.

Having said that, it's a cool project in general, and I feel like it could be
useful to dynamically manage access to large groups of servers (perhaps not
just ssh; you could use it to manage access to https interfaces or other
things). Then again, if you have a large group of servers, you should have
those ports blocked to the world and only allow access through a VPN and/or
jump boxes.

~~~
willis936
Is there a practical benefit to pubkey or certificate auth? I feel like using
ufw with limit on port 22 and using fail2ban covers most cases, assuming good
password decisions.

~~~
gruez
Against a securely generated password in password manager? No. The only real
advantage I can think of is ergonomics: you can safely reuse a public
key/certificate across multiple hosts, and even if one of them becomes
compromised, your key isn't.

~~~
fiddlerwoaroof
This isn’t true: because the secret never leaves your client machine, mitm
attacks can’t be trivially leveraged to access other systems you have access
to.

~~~
gruez
Isn't that what I said? How are you going to get access to other machines if
each machine has a unique password?

~~~
fiddlerwoaroof
Sorry, missed that: the problem with a password is that, it the session is
mitmed, your password is compromised and the person running the attack can
login to the machine whenever they want. ssh keys prevent this attack: mitm
only grants access for the duration of the session.

------
thatha7777
This is very similar to port knocking, although more complicated. Although a
proper JWT token is “more secure” than a sequence of 0-65535 integers, I
contend that having complicated and/or unvetted logic as your first line of
defense is more problematic than secure.

~~~
jrochkind1
Yeah, I'd want to see a mention of port-knocking in the README, indicating
that the author was familiar with this technique, and the known critiques of
it, and why they think this method (or port-knocking in general?) withstands
it. Cause it's definitely in the vicinity.

With no mention, it makes me wonder if the author is familiar with the history
of this approach, which is not a good sign.

And yeah, I'm skeptical. Is this new, uncommon/unvetted, and somewhat complex
"access" app just another place for potential vulnerabilities that give
someone the keys to the kingdom?

~~~
Legogris
> With no mention, it makes me wonder if the author is familiar with the
> history of this approach, which is not a good sign.

Seeing as the word “proactive” keeps reappearing in a similar fashion to
“decentralized” does for people promoting their blockchain app (invariably
hosted web app on google cloud or aws, using Ethereum nodes from Infura, and
administered by a single permissioned smart contract) yeah, sounds like OP is
excited they invented this new principle.

------
ryanjkirk
Setting this up makes much less sense than setting up a tested vpn, such as
wireguard or open, or even a persistent ssh tunnel using autossh to your home
rpi.

I would never allow my prod systems to be potentially exposed by an api that
runs as root. (And the documentation is incorrect on that; it should run as an
unprivileged user with sudo privs to only run a wrapper script that runs
firewall-cmd).

This also makes little sense in the context of configuration management, which
should be enforcing a static set of iptables rules.

~~~
stqism
Personally I’d make a binary run as a non-root user with just CAP_NET_ADMIN
and CAP_NET_RAW permissions to cut the scope down even more.

~~~
ryanjkirk
Having not used setcap, would that also require the setuid bit to be set with
ownership as root (like sudo)? And reside on a filesystem without the nosuid
mount option?

~~~
Reelin
> would that also require the setuid bit to be set with ownership as root
> (like sudo)?

Doesn't setuid just change to whichever user owns the file? So with setuid
root only needs to own it if you want full administrative privileges.

I also haven't played with setcap myself. My understanding is that
capabilities don't rely on file ownership but instead on extended file
attributes (which can't be changed by ordinary users on a correctly configured
system). So root doesn't need to own the file, but of course granting any
capabilities to a binary for which a non-root user has write permission seems
like it would be a really bad idea in general.

> And reside on a filesystem without the nosuid mount option?

Yes, it appears the nosuid mount option disables file capabilities. (As one
would hope!) But I'm not really seeing how that's an issue since (for example)
bind mounts and btrfs subvolume mounts both allow changing the nosuid option.

(man 7 capabilities)

> 1\. For all privileged operations, the kernel must check whether the thread
> has the required capability in its effective set.

> 2\. The kernel must provide system calls allowing a thread's capability sets
> to be changed and retrieved.

> 3\. The file system must support attaching capabilities to an executable
> file, so that a process gains those capabilities when the file is executed.

([https://wiki.archlinux.org/index.php/Capabilities](https://wiki.archlinux.org/index.php/Capabilities))

> Capabilities are implemented on Linux using extended attributes (xattr(7))
> in the security namespace.

(man 2 execve)

> The capabilities of the program file (see capabilities(7)) are also ignored
> if any of the above are true.

------
nyx_
I like to run fail2ban in conjunction with a non-standard SSH port on which
only public key auth is available.

This way, most of the junkware that does rude things to port 22 is banging on
a closed door; the slightly more effective junkware that actually finds the
SSH port gets banned immediately, because I know anyone trying to log in with
a password is full of it.

~~~
dmos62
Wouldn't it make more sense then to keep SSH on 22? Think of it like a
honeypot. The more accessible you make it, the more bans you get. There was a
guy on here saying that fail2ban banning SSH bruteforcers also reduced the
number of HTTP bruteforcers, because they overlap.

~~~
greenie_beans
This is a noob question, but could you not setup fail2ban to ban bad IPs on
both ports, even if you're not using SSH on one? I'm wondering if it's
possible to close port 22 and set fail2ban rules to ban any IPs that try to
SSH there, while changing your SSH port and also setting up fail2ban on that
port, too. Does a closed port work like that?

~~~
kelnos
fail2ban works by tailing authentication logs and matching regexes to find
failed attempts and the IP address of the source. So if nothing is listening
on the port, there won't be anything to spit out logs that fail2ban can read.

You could potentially create a fake daemon that listens on port 22 and just
logs every access attempt, and set up fail2ban to block any IP that even opens
a connection to the port, regardless of what they try to send over it.

Actually, I think iptables has a LOG target that can log connection attempts
even without something listening, so that could work too and be much simpler.

~~~
Legogris
Just make sure to exclude that port from service checks ;)

------
wrkronmiller
I usually just disable password auth and update regularly, when I have an SSH
server open to the internet.

Short of an 0day in the SSH service, I expect brute forcing the private key(s)
to take longer than I have years to live.

~~~
tambourine_man
But it takes CPU, fills up log files, etc.

fail2ban is the first thing install. It’s also easy to understand, which I
love. It’s basically regex on log files.

~~~
necheffa
It really doesn't make that big a difference. I have single core, 515MiB VMs
that idle at <1% utilization with background noise. Most distros come with
logrotate anyways.

~~~
drdaeman
> Most distros

Most distros those days come with journald, and to best of my awareness
journald does not have per-unit (or better, per unit per message type)
configurable retention.

Still in the TODO list today:
[https://github.com/systemd/systemd/blob/908dbc70d6abeb9f6562...](https://github.com/systemd/systemd/blob/908dbc70d6abeb9f65624d89fb5ca021815d69ae/TODO#L1037)
(since at least 2016), with corresponding issue gathering no traction:
[https://github.com/systemd/systemd/issues/9519](https://github.com/systemd/systemd/issues/9519)

~~~
jlgaddis
Yes, but don't most of them _also_ come with syslog forwarding enabled and a
syslogd (rsyslog, syslog-ng, etc.) and logrotate installed?

------
fl0wenol
The idea is better than port knocking, in the sense that you take active
action to associate your host with the server, but there's a few issues.

* Something this simple shouldn't need k8s, unless it was intended as an exercise by the developer for that reason

* It combines the idea of using a non-standard port with certificate based authentication, which you can already do with SSH-- it's functionally the same with more steps

Hypothetically, this approach can be more powerful as a centralized service
running elsewhere, (i.e. cloud, your remote DC) and used for a whole bunch of
jump boxes and end-users. End-users could run a "check-in" script wrapping
around SSH that notified the service that a user was imminent, and then could
check the server to see if 1) The bearer token is accepted for the destination
server 2) That the destination server checked in, saw the new incoming
request, and has already opened the port -- And then proceeds to run the SSH
command if all is well, or fails with appropriate error.

------
joana035
Nope nope nope.

This has serious quality issues and any competent and nice sysadmin should say
_nope_ to run this in any serious environment, for your own good ;-)

IP addresses are just strings? At least parse/validate for IPv4/IPv6.

Why yet another database? Can't the rules be loaded from the running system?

Why not just ipset-persistent + knockd + portsentry? I know it is easy to get
overexcited with a new pet project, but just be careful to not put this kind
of stuff in a production system kidos.

~~~
prashantgupta24
Thank you for the feedback!

~~~
joana035
Anytime.

I saw you are already implementing some ideas from my inputs, that is great,
please keep it going.

Eventually, in notorious systems, you may face issues of having too many /32
rules, a good idea is to implement some mechanism to defragment them into
single/bigger contiguous cidr blocks. There are other firewall implementations
that can serve as inspiration.

OR

I recommend using ipset with a single iptables rule to match/block from the
set. This way you don't need to reload stuff everytime. (Just add the ips into
the set)

PS: I hope you don't mind my snark comments, HN is fun.

~~~
prashantgupta24
I do not mind at all. I appreciate constructive criticism. Plus you gave me
points to work on, I really appreciate that. So much to learn!

------
yabones
I don't like this approach very much because it's much more complicated than
it would have to be. Security is about layers, and this is essentially one
layer that acts as the sole guardian for sshd.

The way that I like to do this is to have a common 'entry point' for all my
cloud systems. Instead of whitelisting IPs on every VPS or cluster I build, I
just add them to the ACL on my management server. All the other systems only
allow SSH connections in from the bastion server. In practice, it works like
this:

* Add IP to the whitelist file in my change control

* Run the Terraform script to update the DigitalOcean ACL

* Start an SSH agent locally and add the key for the bastion, as well as the key for the destination

* Connect to the destination server by using ProxyJump

So, connecting to a box would always route through my bastion system first,
like this:

    
    
        ssh -J mgmt.mydomain.net cool-app-server.mydomain.net
    

I've been doing this for a couple years, and it works great. I practically
never have attempted login attempts on my systems. And, since I use an ssh
agent to forward the keys through the bastion without ever actually storing
them, a compromise of that system doesn't really give the attacker anything
other than access to port 22 on a bunch of systems that they wouldn't know
where to find. Only the most sophisticated attack
([https://xkcd.com/538/](https://xkcd.com/538/)) would lead to a real
compromise.

~~~
jiggawatts
I love that XKCD, but this particular cartoon gives a false impression in
order to be humorous. (You can't fault a comic artist for this of course.)

In my experience the #1 _real_ attack vector is the shouty middle manager
demanding that: "Everything be opened right now because some outsourced admin
says he needed it".

It's nigh impossible to use technological measures to defeat a CxO or a
department head.

------
manexploitsman
I like this "proactive" solution :) Endlessh: an SSH Tarpit
[https://nullprogram.com/blog/2019/03/22/](https://nullprogram.com/blog/2019/03/22/)

------
CamJN
6\. Possible enhancements

Rate limiting the number of requests that can be made to the application

So this just moves the brute forcing target from ssh to a web app. A lot of
work for no added security.

------
oehtXRwMkIs
If I have public key authentication set up for ssh, should I even bother with
fail2ban, firewalld-rest, port knocking, etc.? There's no way anyone is brute
forcing my ed25519 key, so what's the point? Sure security should be layered
and all that but it seems like public key auth is so strong by itself anything
on top seems unnecessary.

~~~
fmajid
It's very easy for brute-force attacks from botnets to consume all your SSH
process slots and then you are locked out.

~~~
roveo
Can confirm, had this problem the other day. CI started failing and after a
short investigation I figured out sshd was just dropping my connections.
Installed fail2ban first, but then just used a firewall since all valid IPs
are known.

------
gregod
If you do not need the more granular firewall configuration options there is
also classic port knocking
([https://en.wikipedia.org/wiki/Port_knocking](https://en.wikipedia.org/wiki/Port_knocking))
where the daemon sits behind the firewall so all ports can be closed by
default.

------
peterwwillis
A dynamic IP filtering list is still reactive because you haven't actually
secured the box or its services. You've just made it slightly inconvenient to
brute force. You might as well use port knocking, because even with a fancy
schmancy authentication system (and the attack surface of a custom web app...)
I can MITM your active connections just fine either way, or spoof your
whitelisted IP.

I know everybody likes Fail2ban, but these two iptables rules (or something
just like them) actually work better and don't fill up your logs:

    
    
      iptables -t mangle -A PREROUTING -p tcp --dport ssh -m conntrack --ctstate NEW -m recent --set
      iptables -t mangle -A PREROUTING -p tcp --dport ssh -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
    

To actually protect the network layer, use some kind of VPN with MFA.

------
geocrasher
This is interesting in that it combines the concept of port knocking with a
REST interface, which I'm assuming is up to the user to create a front end
for.

Unfotunately it also relies on Kubernetes which means that using it for a
single system isn't practical. At least, not for _this_ server owner.

My own approach is simply security by obscurity (a non standard port) with
APF/BFD doing the needful for locking bots out if they figure out the port.
I've had to change ports only once in 6 years, so it's working to keep bots
out rather nicely.

And really that's all these things are- a way to keep bots out. A determined
attacker will figure this stuff out anyway.

------
ed25519FUUU
Secure SSH and you won't have to worry about rogue login attempts, which
happen to everyone. If it really bothers you then move it to another port
where it will happen less.

But install a new firewall management system? Sounds like it will definitely
introduce more risks than the problems it solves, which for the SSH example
isn't really a problem at all.

------
ff7c11
Have you tried knockd
[https://linux.die.net/man/1/knockd](https://linux.die.net/man/1/knockd) ? You
send a special sequence of "knocks" to the server (packets to different ports)
and it executes a command such as allowing your IP for a time period. No JWTs.

~~~
thedanbob
My go-to is fwnop
([https://www.cipherdyne.org/fwknop/](https://www.cipherdyne.org/fwknop/))
which is actually similar to the OP’s thing but battle tested. Only downside
is it’s not available on iOS, so recently I setup WireGuard for my iPad.

And yes, ssh pubkey + fwnop + WireGuard + fail2ban is ridiculous overkill, but
hey, it’s my homelab server. That’s how I learn this stuff.

------
jlgaddis
If you're gonna go through all of this work -- including creating and
maintaining private keys -- why not just restrict the SSH server to only
permit key-based authentication (optionally, signed by your CA)?

If having 22/TCP open to the world is an issue, then set up Wireguard on the
host and only allow SSH connections that are coming in over the Wireguard
interface.

Got a bunch of machines to deal with? Set up a jumpbox or two running OpenBSD,
lock it down, give your users access to it via SSH (optionally, over a
Wireguard connection) and then only allow SSH access to all of those other
hosts from the jumpbox(es).

Then there's the fact that I have a _whole lot more_ trust in the security of
OpenSSH than I do some random web application!

To me, this just seems kinda pointless -- there's a bunch of other, better
(IMO) ways to deal with this -- but I guess if it fits your needs ...

~~~
MrStonedOne
Step 1: assume every service listening on your network has an auth exploit
that has not been found.

Step 2: deploy this.

Step 3: ...

Step 4: profit!

The line between something like this and a locking the service behind a VPN is
very narrow. They both achieve the same thing, protect high risk/value targets
by not even allowing unauthorized people to talk to them at the network level.

------
deathanatos
> _you can go to jwt.io and generate a valid JWT using RS256 algorithm (the
> payload doesn 't matter). You will be using that JWT to make calls to the
> REST application, so keep the JWT safe._

The JWT you got after you plugged the private key into a random website is
going to protect access to your machine?

------
koolba
Adding a JWT authenticated API layer to something is not a first choice for
adding additional security.

If you want something like this look into platform level firewalls (ex: AWS
security groups) or run spiped in front of your SSH server. I’d trust that a
hell of a lot more than a REST API.

------
deeblering4
Requiring ssh keys, disabling password auth, and using ssh bastion hosts is my
preferred approach.

~~~
robomc
Yeah it's not clear to me why exposing the ability to configure your firewall,
protected by a keypair, to the internet, is safer than just letting someone
bang their head against sshd, protected by a keypair, and presumably having
had a lot more eyes on it for security.

~~~
donor20
And the distributions can also do the SElinux stuff that I don't use day to
day.

------
abpavel
This looks like reincarnation of "Lock & Key" feature of Cisco routers
available since the late 90s. There were two major issues that led to
abandonment and hindered adoption. The first is that it's an extra step. Extra
steps for installation, for complexity, for single point of failure, for
availability of key services. The second is that instead of thwarting
attackers, you're thwarting yourself, every single time. It breaks so many
usecases, for example if you have a new machine, or if you want to access
through a jump host, or from ssh client on phone, etc.

------
john-shaffer
On EC2, I've had a great experience using AWS Systems Manager [1]. I don't
need any ports open, and it works great with normal shell tools and emacs
given a .ssh/config like this:

host i-* mi-*

ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-
StartSSHSession --parameters 'portNumber=%p'"

[1] [https://docs.aws.amazon.com/systems-
manager/latest/userguide...](https://docs.aws.amazon.com/systems-
manager/latest/userguide/what-is-systems-manager.html)

------
MrStonedOne
Documentation and idea framing is too snowflaked.

This a JWT based port knocking over https framework that would be useful when
used in a much broader sense.

Framing it as proactive fail2ban is technically correct, it also masks the
other and more powerful use-cases.

I could see this in use as a vpn bypass for a prosumer production system,
where normal operational commands go over a secured vpn but in a pinch, you
can disable that restriction for direct control during a partial failure.

------
swinglock
A proactive way of increasing your attack surface no doubt.

~~~
joana035
A proactive way to shoot yourself in the foot for sure.

------
netsectoday
Please don't use this software. From a security perspective this is horrible.

Option A) Expose all firewall rules via some hacky web-based API.

Option B)
[https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf](https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf)

------
solatic
If you're going to take this approach, a bigger question is why bother running
the SSH jumpbox 24/7 in the first place? Shut down the SSH jumpbox when you
don't need it (thus achieving SSH isolation) and start it up with your public
IP in user-data to enable access to you and only you when you do need it.

------
Jnr
This sounds like a complex port knocking setup with bigger overhead. If you
want better security and UDP is not a problem, consider use of Wireguard VPN.
It is passive and silent, random attackers won’t even know it is there.

------
Tepix
Running SSH on a non-standard port gets rid of the brute force login attempts,
too.

~~~
zootboy
In my experience, a non-standard port reduces the number of brute force
attempts, but it doesn't eliminate them. I just do a simple rate limit on syn
packets to the port and use SSH keys for auth, and that feels like plenty of
security.

------
nine_k
(1) The auth interface is per IP address, as visible by the listening server.
(2) The word "NAT" is found 0 times in the text. (3) There is no point 3 from
the practical standpoint, please move along :(

------
gitgud
Much simpler to stick with SSH and an IP Address whitelist...

Then you can't even get login attempts, unless they're from the same IP... am
I missing something?

------
cperciva
I still think that running ssh over spiped is more secure. Every byte being
passed to sshd is cryptographically verified (and encrypted).

------
wruza
Can anyone explain what's wrong with "pubkey only, ignore attempts" method?

------
johnisgood
I just whitelist IPs. You can do this easily using iptables or sshd config.

------
hitpointdrew
Why is SSH open to the public internet at all, ever? Use WireGuard as VPN
tunnel to your servers network, then SSH via internal IP.

~~~
joana035
> Why is SSH open to the public internet at all, ever?

Wasn't SSH made for that?

~~~
fiddlerwoaroof
Yeah, I trust the OpenSSH developers a lot more than almost anyone else I can
think of: if I’m going to trust any services open to the public internet, I’ll
trust OpenSSH

------
johnghanks
Wow - why didn't I think of just setting up a firewall instead of running
fail2ban. Great idea.

~~~
_joel
eh, fail2ban uses iptables to ban?

~~~
seized
Fail2Ban can execute scripts etc to do things. I have mine making API calls to
OPNsense to block IPs there.

