
Practical Linux Hardening Guide - covertress
https://github.com/trimstray/the-practical-linux-hardening-guide
======
nisa
Not a fan of this guide: No explanations, no context and worse: missing
critical stuff for your typical hosted $webapp server usecase:

run services as normal user and confine them using apparmor (or selinux if you
are masochichst)

the NSA will probably hack you anyway, that drive by script-kiddie that kills
your company JIRA instance by DDoSing some gameserver while exploiting some
old Wordpress plugin _can_ be stopped by apparmor!

What's missing is some kind of kernel live patching - if someone is not a
script-kiddie on your machine that Wordpress shell is a nice tool to elevate
to root - your kernel is rotting and exploits are plenty... so consider some
kind of live-patching if your are paranoid.

Also no discussion about updates, the need to restart services for reloading
updated .so files (like openssl)...

You can obey that guide and be busy with life and some skiddy root exploits
your box anyway...

so /rant - there must be some better guides? found those cis benchmarks
([https://www.cisecurity.org](https://www.cisecurity.org)) a mixed bag - is
there anything better out there?

~~~
ryanlol
Yeah, getting strong 90s vibes from this doc, perhaps because most hardening
manuals like this are just collections of advice from other (bad and old)
hardening manuals.

~~~
DCKing
That's exactly it.

Lists like this exist to give administrators and their managers (1) a false
sense of security when implementing it and (2) a subconscious excuse not to
actually have to think about what you actually have to defend against. The
list of topics is almost entirely useless if you are running any sort of
containers on top of your Linux install, just as one example.

I'm sorry to be sounding so harsh, but I sincerely detest the security-by-
checklist approach. It barely protects against any real-world threats,
needlessly complicates matters, and gives credence to the perception that
"security slow things down" through over-engineered solutions to non-existing
problems. Moreover, it's a subconscious excuse to actually stop thinking about
one's actual risks. Reasoning about actual threat models may not be easy but
it is what one should be doing.

There's nothing "practical" about this at all, in my view.

~~~
kbenson
> I'm sorry to be sounding so harsh, but I sincerely detest the security-by-
> checklist approach.

Checklists are essential (or becoming so) in all professions that require a
level of safety or assurance in their operations. They don't _replace_ careful
though and action, but they do _supplement_ it be making sure there's a
minimum level of items that were checked and not omitted because someone
thought they didn't apply when then did, or someone just plain forgot. A
checklist is a good way to start some rational thinking about what's required
for your specific case (especially if it's so restrictive that you have to
(and are expected to) selectively alter portions of the checklist just so your
current use case functions.

Checklists are well known and save lives in some professions (such as
aviation), and are being applied to others even when there's push-back
(Surgeons and emergency room operations[1]) because the benefits are just so
large.

I think it's fair to dislike how people and organizations adopt a checklist in
lieu of careful thought about security, and to dislike poorly defined and
reasoned checklists themselves, but I for one would be much happier, and feel
much safer, if security checklists were much more common overall.

1:
[https://www.who.int/bulletin/volumes/86/7/08-010708/en/](https://www.who.int/bulletin/volumes/86/7/08-010708/en/)

~~~
DCKing
Checklists are fine if you have a fixed problem space. Just to go by your
example - aviation safety is a mostly fixed problem space you can make
mitigations in. Any item on the checklist would be there to mitigate safety
issues almost any aircraft would encounter (and I'd imagine there are specific
checklists for e.g. passenger vs. cargo aviation).

"Linux security" is not a fixed problem space. Not at all. And that's
precisely my problem with this - this checklist pretends that it's a fixed
problem space, and therefore grossly misrepresents the problem.

~~~
kbenson
My point is that most problem spaces can be split into fixed and non fixed
portions. Aviation safety has plenty of problems that require a real person to
make a call and react intelligently, which is why we still have pilots.
Checklists are used to cut down on the entirely avoidable problems that might
be missed some percentage of the time otherwise.

There are _plenty_ of things in Linux security that are static solutions that
can be employed almost all the time, such as not allowing direct access to
root accounts, always running a local firewall, making sure remote services
aren't run as root without dropping privileges, etc.

------
tjmc
> Maintaining a stable temperature and humidity within tight tolerances is
> critical to IT system reliability.

Data center HVAC engineer here. The part about tight tolerances is not
entirely accurate. IT equipment can handle a reasonably wide range of
temperatures. The ASHRAE guidelines [1] have a range of 18 to 27 deg C for all
classes of equipment.

What is an issue is rapid changes between temperatures. For a data center with
any kind of devices (eg tape storage), ASHRAE limits this rate of change to
5degC within a 15 minute period and obviously you also want to avoid any
situation where condensation could occur or static build up (high or low
relative humidity).

Data centers do try to keep the temp stable towards the top of the thermal
envelope for economic reasons but there's always some fluctuation due to
server load.

[1] ASHRAE TC9.9 Data Center Power Equipment Thermal Guidelines and Best
Practices
[http://tc0909.ashraetcs.org/documents/ashrae_tc0909_power_wh...](http://tc0909.ashraetcs.org/documents/ashrae_tc0909_power_white_paper_22_june_2016_revised.pdf)

~~~
taneq
Wow, your gear is treated so nicely. The stuff I work on has to deal with
anywhere between 0C and 80C while having the crap shaken out of it and
occasionally being hit with a high pressure hose.

Different worlds...

~~~
gerdesj
OK, high pressure hose in a data centre piques my interest 8)

I stuck a camera on a stick at the southern boundary of my property, about
three feet above the stream. The stream rose four feet for a while. I moved
the camera.

Now, 0 to 80C. That is just plain odd. Here (southern UK) the temp range is
something like -10 to +35C. Nowhere has 0-80C apart from a weird oven.

~~~
taneq
This is minesite automation stuff, so it usually lives in a stainless steel
cabinet, ambient air temperature can be over 50C, sometimes sharing the space
with high power switchgear. It's not unheard of for the inside of the cabinets
to be over 70C if it's mounted in direct sunlight, so most gear is rated to
80C. Then at other times in the year it'll get down to freezing.

------
outworlder
> For secure rooms make sure that the walls go beyond the false ceiling, and
> below the raised floor, large vents should also be covered with bars if
> possible.

Ah, the 'Aliens' threat model.

~~~
adrianratnapala
Well if feel the need to have a physically secure room, then presumably you
want to actually be secure.

~~~
bigiain
What, are you trying to say security theatre doesn't make everything safe???

Surely if we grope everybody's balls/boobs on their way in, and have a secret
list of "people with names that are vaguely similar to middle eastern sounding
names" that we won't let in, that'd make up for the walls that you can just
crawl under of climb over? Right?

------
fyjvd90
Not Linux specific but:

Don’t make your security encourage legitimate users to work around it due to
pointless friction

~~~
torvolt
I always felt like increasing security has some cost on convenience.

~~~
fyjvd90
It does, but there’s also what I call stupid security that doesn’t really add
any measurable improvement, but does decrease productivity. Users out smart
these systems all the time (e.g. forced password changes where you can’t use
the last 10 passwords, users just change their password 11 times so they can
continue to use a password that’s been configured on their devices. It’s
stupid then to force changes like that when there are much better ways like
mfa)

Smart security allows users to do what they need to do efficiently and safely.

~~~
llama052
Yeah I can't stand the forced password changes where you can't use the last X
passwords, or passwords that expire every 30 days. A lot of times it's
security compliance entities that push this down to companies, for instance
PCI, etc all require those. I think even the new NIST standards address these
practices, but the compliance entities are slow and far from pragmatic.

~~~
techslave
yes but the rule does enhance security.

the password rules force you to choose [heuristically] guessable passwords,
therefore they must be changed every 90 days. simple!

~~~
xorgar831
It doesn't if users are working around it.

~~~
cwyers
Right. The first rule of password security: if you have a large enough user
base, the odds of a user writing down a password increase, and as passwords
become sufficiently difficult to remember, the odds approach 100% at some
point that _some_ people are writing down passwords. No amount of defense in
depth can protect the "I have a Post-It note under my keyboard" problem, if
people can get into your building.

~~~
shaftoe
We've handled this by mandating password manager use and pushing length
requirements to absurd levels to where it truly is easier to just use the
manager, which has two factor.

------
fforflo
On that note, I would like to recommend the most complete book I've
encountered on the topic: "UNIX and Linux System Administration Handbook" by
Nemeth et.al. Just because I wish someone had recommended it to me earlier.

~~~
SSLy
I can vouch for the book also. It's the best technical book for system
operators.

~~~
fforflo
It's really amazing the stuff they cover succinctly and yet complete. From the
"classic" unix stuff and contemporary ones, like Docker, Ansible etc.

------
teddyh
Personally, I prefer the Securing Debian Manual
[http://www.debian.org/doc/manuals/securing-debian-
howto/](http://www.debian.org/doc/manuals/securing-debian-howto/)

Note: This is an official Debian manual, not some random third party.

------
viraptor
This looks good. I like that they have explanations / reasoning included,
rather than just "do this". If all the planned topics get filled, that will be
something to recommend.

Also, first time I hear about polyinstantiated directories, so TIL.

~~~
ASalazarMX
This was one of the neatest features of the computer formerly known as AS/400:
each job (batch or interactive) had its own temporary library (QTEMP). You
could create objects there knowing only you could see them, and the library
was destroyed the moment the job ended. Definitely will try polyinstantiated
directories.

~~~
gerdesj
What do you mean former? There is an AS/400 in our data centre that we look
after for a customer for their off site DR box, with constant DB replication.
It is about four years old, the platform is rather older. Bloody noisy beast
especially on boot up (IPL?) sounds like a helicopter winding up. It was
switched on after racking and that was it - I'm told it is simply available.

~~~
schoen
Although the platform still exists, IBM has renamed it:

[https://en.wikipedia.org/wiki/IBM_System_i](https://en.wikipedia.org/wiki/IBM_System_i)

------
crankylinuxuser
A lot of this can be eliminated (Not the physical security or install) by
doing the following:

1\. Run centOS or Redhat

2\. install openscap-workbench

3\. Use the centOS stig and choose which profile (I recommend US govt
configuration base)

4\. Uncheck the firewall rules (they set it to deny all incoming; change to
DMZ with basic rules)

5\. Click remediate and apply.

6\. OpenSCAP does the work for you to harden the system

You'll have to use other security appropriate tools for appropriate servers,
but you'll know which service and its ramifications.I know that MySQL has a
comprehensive security script to prepare. Other tools have similar built in
functions.

Also note you can download Nessus and get a 7 day free trial as well. It's not
perfect since the ticket price is $2400/yr . You could also use OpenSCAP for
compliance, and metasploit as a substitute for application. There's also
websuites like Burp and OWASP.

But regardless you pay or not for automated testing, you need something to
automatically find bad things so you can fix them.

~~~
bubblethink
As with a lot of other enterprisey software made by RH, it should all work in
theory, but it rarely works as well. I've never been able to get stuff like
openscap to work reliably with satellite (which is one of those things that
should probably not exist at all). I don't remember which openscap profile it
was (I think C2S), but it would just hang on nfs servers with lots of data. So
you can go and hack on these profiles, but it just gives the impression that
none of this stuff is that widely used/production ready.

------
bubblethink
So here's something I've always wondered. If you use linux as a
desktop/workstation in enterprise, how do you deploy it securely ? Other than
chromeos, I don't see any other distro trying to tackle this. With all
traditional distros, you'll likely need to give the user sudo on their machine
to be productive (for eg., installing packages). However, all bets are off
once you do that. You can restrict sudo to certain programs, but that also
isn't very practical (for eg., sudo for yum is basically sudo for the whole
system since you can install arbitrary packages). You can create a chromeos
like system, but is there no interest in it ? There are some developments like
fedora silverblue and nixos, but how is it that this has gone unsolved till
now ?

~~~
gerdesj
"you'll likely need to give the user sudo on their machine" \- no we have
sysadmins and application management systems to obviate that. If a user needs
an app then they ring up the helldesk and get added to a group (after
authorisation), which causes the app to appear.

~~~
bubblethink
>If a user needs an app then they ring up the helldesk

That seems very inefficient and friction prone. We have stuff like the nix
package manager that can install stuff locally without sudo. It also has other
nice features from what I've read. What I'm saying is that we need general
purpose distros where you don't need sudo to use them.

~~~
romeisendcoming
Interesting you bring this up. Regardless of the provenance of packages and
facility for installation you seem to be hinting that there should be a method
to install specific service/feature sets to a system securely and without
systems access.

I designed such a system which runs with appropriate permissions to a task, is
modular, and accomplishes the provisioning via 'job' files submitted to a
service endpoint. A user could describe the software they want installed. The
SA performs due diligence on vetting software and designs a standard install
module then generates a skeleton jobfile and provides it to the requesting
user to fill in necessary details and submit. Wash , rinse, repeat.

------
Havoc
Also...stick the SSH on a non-default port. Cuts login attempts down to near
zero.

~~~
ynezz
That's really terrible recommendation. You should add firewall rules and allow
SSH access only from trusted hosts, ideally over VPN. It's not about login
attempts but about possible bug in SSH (and other network services), because
it's all just software, written in C, so believe it or not, but it's there,
it's not if but when.

~~~
wahern
The firewall isn't immune to exploitable bugs:
[https://nvd.nist.gov/vuln/detail/CVE-2017-18017](https://nvd.nist.gov/vuln/detail/CVE-2017-18017)

Likewise for VPN stacks:
[https://nvd.nist.gov/vuln/detail/CVE-2017-7521](https://nvd.nist.gov/vuln/detail/CVE-2017-7521)

If you're not prepared to expose a service to the world, you _probably_ should
run it at all. Ad hoc, non-standard configurations add substantial complexity
and maintenance burdens. Complexity is the enemy of security, and having less
time to manage more complex configurations is not a good recipe.

If you really need a service, then choose the best one and move on. The rule
of thumb is that if you can reach a service, you should assume anybody else
can, as well. This is especially true regarding SSH. I've seen plenty of
servers p0wned via SSH, but never by breaking SSH. Instead the vector was
always through an SSH user's computer infected by malware.

You want secure SSH? Disable password authentication and force everybody to
use smartcard authentication like a Yubikey. I do rate limit SSH access using
OpenBSD PF, but only because the authentication failures fill up and pollute
the logs.

~~~
vinay_ys
Have you seen yubikey/smartcard being used for sshin a large team? Would love
to hear about your setup and learnings.

------
zerotolerance
I'd absolutely love a section that covers UEFI, secure and measured boot, and
TPM integrations.

~~~
gerdesj
You can install Ubuntu with UEFI and secure boot enabled without any problems
(personal experience). I believe that several other distros work as well, out
of the box. All distros can do UEFI with minimal effort.

To be honest I consider things like BMC and the like and whatever goes on
inside the BIOS and the like as far greater opaque problems than some of the
more esoteric security "assurances" like measured boot.

------
vinay_ys
Even if you have a single server to run a bunch of services (web, mail, ssh,
file server etc), best way to raise the security bar quickly is to isolate
these services from one another. Use one of the modern
virtualmachine/container technologies with minimal footprint to do this.
Ideally, you would want physical separation between a node running
firewall/loadbalancer and the servers behind them. If you are running a large
cluster of many servers, then you definitely want to plan and separate your
architecture into different layers of separations.

Don't forget your own laptop from where you login to administer all this is
the best chance an attacker has to breaching all this security. So, ensure
there is good role separation for administrators and good security hygiene on
all the individuals machines (use password managers, do only official work on
these laptops, don't visit random websites, run vetted authorised packages
only).

For highly sensitive stuff (like CA or any root of your chain of trust), setup
a SCIF and use only terminals inside a SCIF to access/administer them.

------
vinay_ys
Since this guide started with datacenter and hardware security, it is
important to discuss the merits of having a secure enclave (hardware security
module) on each server to hold the security keys.

Securing the chain of trust is a critical element of securing any system. If
the root of the chain of trust is not securely bootstrapped and continuously
verified to be still secure, then all bets are off.

The provenance and integrity of binaries, config and data running on your
servers should be fully known – their integrity verified with cryptographic
methods tied to your chain of trust. There should be no way for these entities
to be mutated without cryptographic hardware tokens also being broken. This
raises the security bar significantly. (mostly targeted supply chain attacks
mounted by highly motivated likely state actors can overcome this bar).

Without this, you are always standing on shaky suspicious ground no matter how
much you harden the layers above.

------
auslander
I just keep my prod systems clean, updated and SELinux in enforcing mode.
Behind strict firewall of course. This is sufficient for practical reasons.

In proper devops teams, there is no even ssh access. System is deployed from
image and configures itself. And is killed in scaling events.

Grsecurity is for hardcore stuff. Or openbsd, ultimate solution :)

~~~
xorcist
Yeah, let's not debug anything, ever.

That'll teach'em not to make mistaeks.

------
jboy55
Oversaw a CTO 'harden' a linux box that was used as a firewall and exploited
with;

chmod -R 600 /

I wouldn't recommend doing that.

~~~
xorcist
Well, it certainly is secure, there's no denying that.

Especially after the next reboot.

------
amelius
Suggested additions: 1. Don't use a CPU equipped with Intel ME or equivalent,
or other system-level management engines like HP's iLO. 2. Look out for keylog
devices.

Also, from a practical point of view: sshguard is very useful and easy to
install.

~~~
viraptor
You mean, you want to drive to your data centre every single time something
fails to boot, or have the awkward debugging session over the phone with
remote hands? Or come in with a physical copy of the firmware update every
time? Remote management is necessary in some situations. KVMs, ILO, and others
have their issues, but it's a tradeoff. "Don't use" is not a great suggestion.

~~~
amelius
Ok, if you _have_ to use something like iLO, then make sure to connect it
_only_ to an internal network. And update the thing frequently.

~~~
Spivak
People expose IPMI to the internet?

------
systematical
Why put /boot on a separate partition? It becomes a pain on Ubuntu if the size
is too small, if you forget to run apt autoremove (or whatever the exact
command is) that thing fills up with old kernals. Updates can fall because of
this and your system breaks in weird ways.

I know this because the original developer of our system used a /boot
partition of 500 MB. That's right, 500 MB. I noped out of that. Don't even get
me start on 2 GB total disk space for everything else on our API server. This
was built in 2013 mind you, plenty of disk space by then...

~~~
gcb0
I think the advantage is to have /boot read only? so you can't abuse fs
security bugs to infect boot files

------
hsnewman
No mention of 2 factor authentication. Humm

~~~
jlgaddis
Have fun getting smartcard logon working on a desktop Linux machine (unless
you've got a full-blown enterprise PKI in place -- and, even then, it's still
gonna be fun).

 _(cue the "but I did it with a Yubikey" responses)_

~~~
Spivak
Or you could use a hosted solution like Duo which has a PAM module which is
dead simple to use.

Or pam_google_authenticator.

------
devwastaken
Seems to be in progress. In the future it will cover 'automatoc security
updates'. Anyone have insight into wether this is a good idea and how it's
done on something as rolling as an ubuntu server?

~~~
shocks
On Ubuntu and Debian there is a package for this: unattended-upgrades.

The default configuration should work fine for most.

~~~
nomadluap
I've had problems on some servers where unattended-upgrades would install new
kernel versions without removing the old ones, and ending up filling my /boot
partition.

~~~
systematical
Ditto, for this reason I leave it off. Also, I want to be able to verify
patches in staging. Kill me if you want, but we only patch once a month for
this reason. For critical things like heartbleed, meltdown, spectre we fast
track them obviously. I find it helpful to subscribe to all security lists of
core services to know if something really nasty is out there.

------
wyclif
This seems good technically, but the writing needs a good once-over. Some
directions are awkwardly written and obtuse:

 _For more security-focused situations is as follows_

------
w8rbt
This is like having a car that you work on more than you drive. Great as a
hobby but that's about it.

------
pjmlp
One thing I miss from that guide.

What about compiling everything with GCC security enabled like Google does on
Android?

~~~
dijit
The fact is; your distro is likely doing this already.

There is a script which reads elf binaries and outputs security info:
[https://unix.stackexchange.com/a/89214](https://unix.stackexchange.com/a/89214)

~~~
pjmlp
Not all of them enable ALSR for example, right?

Thanks for the script, I wasn't aware of it.

~~~
dijit
ASLR is implemented in the kernel; although some unscrupulous programs are
able to bypass it entirely (or force it to be disabled system-wide as is the
case for Dropbox on mac)

~~~
aaronmdjones
I think what they were getting at is that ASLR doesn't work at all (for the
main program) if you're not running a PIE binary. The libraries still get it
(they usually have to be PIC by design) but that's not much benefit when the
application using those libraries has entirely predictable addresses.

So yeah, your binaries need to be built with CFLAGS="-fPIE" and linked with
LDFLAGS="-pie".

------
pi-victor
incomplete and yet has 1600 stars now. good job. also it's top on HN.

------
cypherg
this is not practical...at all.

~~~
ozim
I have to agree. Table of contents is a bit too ambitious. Anything that wants
to cover "Air conditioning", "Fire protection" and configuring Nginx and
Apache is not going to be practical.

------
blakesterz
This would make a great Ansible role too!

~~~
viraptor
This is the beginning of a great list, but to apply everything without careful
understanding / adapting to your situation would just end up confusing people
and causing them problems. Some things are also very distro/environment
specific (like the mounts layout).

There are tools which will verify the situation for you, but not force a
specific solution.

------
fxfan
Does anybody knowledgeable know how hard would it be to patch the same linux
kernel with per-application permissions? I know there is some 'OS' doing it
but I am looking for a WSL like solution- 'patch' the existing kernel so I can
run the binaries et al unmodified.

There would be a permissioning matrix where I can list allow-all-permissions
and deny-all-permissions by default and then change the granularity

~~~
viraptor
That sounds like selinux / apparmor / rbac. (Or alternatively I don't
understand what you mean by "patch Linux kernel with per-app permissions")

~~~
fxfan
You understood exactly and apparmor clearly fits what I need (and possibly the
other two too but I haven't had time to look...). I used to (incorrectly)
believe that SELinux only dealt with networking...

