
My First 10 Minutes on a Server - codelitt
http://www.codelitt.com/blog/my-first-10-minutes-on-a-server-primer-for-securing-ubuntu/
======
brokenwren
This one is pretty decent but if you want the ultimate guide check out this
one:

[https://www.inversoft.com/guides/2016-guide-to-user-data-
sec...](https://www.inversoft.com/guides/2016-guide-to-user-data-security)

It covers 10x what all the other guides cover in terms of server and
application security. It was posted a few weeks ago on HN but didn't make the
front-page.

~~~
atonse
My worry here is that, in posting what seems to be a book, people just won't
even do it because we don't have time to do it, unless this is a primary part
of their jobs.

If a 10 minute guide gets users 90% of the way, then they're more likely to do
it. And that's good enough to cover a majority of automated attacks.

Update: I take it back – they've provided scripts to run this stuff. I will
explore these. Thanks for the link.

~~~
peterwwillis
Back in the day, The Linux Documentation Project had a trove of hundreds of
HOWTOs covering every facet of using Linux. By today's standards they seem
like "books", but in reality they were step-by-step instructions for anything
you could ever want to do in Linux. No digging through forums, no combing
through man pages, no following broken outdated blog posts that didn't explain
what you were doing. I find it sad that these are seen as detrimental today.

~~~
viraptor
I feel like arch wiki is that thing today. It's a bit disorganised and things
get outdated while nobody's looking, but there's almost anything you'd every
want in there.

~~~
peterwwillis
It seems like there a bunch of articles there but they're more like notes on
how to use a tool specifically with Arch, rather than a guide for everyone.

Compare this page
[http://www.tldp.org/HOWTO/Quota.html](http://www.tldp.org/HOWTO/Quota.html)
to this page
[https://wiki.archlinux.org/index.php/Disk_quota](https://wiki.archlinux.org/index.php/Disk_quota)
. TLDP organized HOWTOs like mini-books; tables of content, multiple authors,
versioned releases, and of course, you could download them all and search
through them by category. And they didn't assume things like the distribution
setting up a bunch of the tools and system for you, so you learned how the
tools _actually_ worked.

~~~
viraptor
Sure, it's a different format. But you can always dig into man pages or other
documentation to learn the details. Arch wiki is great for practical solutions
- for example
[https://wiki.archlinux.org/index.php/HiDPI](https://wiki.archlinux.org/index.php/HiDPI)
is the place to go to for hidpi display configuration. TLDP has not been
seriously updated since hidpi even came out. Similar to
[https://wiki.archlinux.org/index.php/Power_management](https://wiki.archlinux.org/index.php/Power_management)
\- there's no recent TLDP equivalent.

I appreciate TLDP for what it is and the details they go to. But Arch wiki can
easily be read as a FAQ for all modern systems. And it will usually send you
to other places for details (sometimes TLDP as well)

------
jldugger
> We don't even have a password for our root user. We'll want to select
> something random and complex.

So you're taking something secure by default -- no password means no login
allowed, and making it less secure. And if you have hundreds of these servers,
you'll need to rotate them whenever someone on the team leaves. This is
painful.

Simple solution: leave root password blank, don't forget your sudo password.
If you can't get in, use grub or a liveCD. Or tie auth to ldap or kerberos so
you _can't_ forget. This is one area where Windows has a distinct advantage:
AD more or less requires admins to think at the level of network of servers,
and provides a baseline set of services always present.

~~~
taneq
This part struck me as odd, too, especially when you consider the
justification: That if you lose access to your sudo account/password, you must
have some other way to get into the system.

Backups should include everything required to rebuild every server in the
company. No server should be 'too critical' to wipe and start again if
required. So maybe I'm being too much of an armchair warrior here, but the
reason I don't like this one is less about the actual security implications,
and more because of what it says about the fragility of the overall setup.

~~~
corney91
>Backups should include everything required to rebuild every server in the
company. No server should be 'too critical' to wipe and start again if
required.

True. But compare the effort required for, say, solving an accidentally
misconfigured /etc/sudoers on a database server. You could argue that would
never happen with proper testing, but shit happens and and having a
particularly long root password in Keepass is a small price to pay to save
that sort of headache.

------
malingo
This is good advice on achieving the most secure SSH configuration:
[https://stribika.github.io/2015/01/04/secure-secure-
shell.ht...](https://stribika.github.io/2015/01/04/secure-secure-shell.html)

"My goal with this post here is to make NSA analysts sad."

~~~
newman314
Actually, I've improved on this somewhat by splitting configs to 6.5+ vs.
older. Corrections welcome. At some point, I will get around to publishing it.

 __Configs __

OpenSSH 6.5+ Server

UsePrivilegeSeparation sandbox KexAlgorithms
curve25519-sha256@libssh.org,diffie-hellman-group14-sha1 Ciphers
chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-
sha2-512,hmac-sha2-256

OpenSSH Server Legacy

#UsePrivilegeSeparation yes KexAlgorithms diffie-hellman-group14-sha1 Ciphers
aes256-ctr,aes128-ctr MACs hmac-sha2-512,hmac-sha2-256

OpenSSH 6.5+ Client

UseRoaming no IdentitiesOnly yes KexAlgorithms
curve25519-sha256@libssh.org,diffie-hellman-group14-sha1,diffie-hellman-group-
exchange-sha256,diffie-hellman-group-exchange-sha1 HostKeyAlgorithms ssh-
ed25519-cert-v01@openssh.com,ssh-ed25519,ssh-rsa-cert-v01@openssh.com,ssh-rsa
Ciphers
chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-
sha1-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1

Host * IdentityFile ~/.ssh/id_ed25519 IdentityFile ~/.ssh/id_rsa
HashKnownHosts yes VisualHostKey yes VerifyHostKeyDNS ask AddressFamily inet
ForwardX11 no ForwardX11Trusted no

OpenSSH Client Legacy

UseRoaming no IdentitiesOnly yes KexAlgorithms diffie-hellman-group14-sha1
HostKeyAlgorithms ssh-rsa-cert-v01@openssh.com,ssh-rsa Ciphers
aes256-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-512,hmac-
sha2-256,hmac-sha1

Host * IdentityFile ~/.ssh/id_rsa HashKnownHosts yes VisualHostKey yes
VerifyHostKeyDNS ask AddressFamily inet ForwardX11 no ForwardX11Trusted no

------
tjohns
> I check our logwatch email every morning and thoroughly enjoy watching
> several hundreds (sometimes 1000s) of attempts at gaining access with little
> prevail.

This is something that actually bugs me a bit. These attacks are so common,
getting emails like this every day contributes to alarm fatigue.
([https://en.wikipedia.org/wiki/Alarm_fatigue](https://en.wikipedia.org/wiki/Alarm_fatigue))

I'd love to see the Linux nightly security scripts replaced with something
that only sends out emails when there's an specific _actionable_ event I need
to pay attention to. Ideally in a way that can easily be aggregated over all
the machines I manage.

~~~
MichaelGG
Yep and this doesn't demonstrate anything about security. Showing brute force
scanners trying out "root/letmein123" doesn't teach anyone the importance of
good security, just the importance of not using super-common user/passes.

I cannot figure out why anyone would care or find anything useful in these
logs. Change the port, call it a day. Getting worked up about random SSH
attempts (or random HTTP "exploit" attempts) seems to be for admins with too
much free time.

------
chrisfosterelli
> sudo ufw allow from {your-ip} to any port 22

I'm surprised nobody mentioned this is a great way to shoot yourself in the
foot if you don't have a static IP.

~~~
codelitt
Ooo. Fair point. I'll add that now and link to your comment.

~~~
chrisfosterelli
Great! Thanks for the article, it makes a good reference when setting up
VPS's.

------
ryanmarsh
I don't mean to sound flippant but why can't these "lock down your new box"
tutorials just be a bash script? Shouldn't they be?

~~~
larrywright
Hardening (along with any other server setup/configuration) should be
implemented using some idempotent configuration management software (Chef,
Puppet, Ansible, etc). It's 2016, there's no need to configure servers
manually.

~~~
devishard
It depends.

I'm a security guy, so it would be embarrassing and possibly bad for my career
if any of my servers got hacked, but I actually don't set up servers that
often--it's not part of my job. These Web 2.0 configuration management
solutions change pretty fast and don't care about reverse compatibility. So
between my infrequent setups my configuration scripts pretty much always
break.

Contrast this with bash, which cares a whole lot about reverse compatibility,
and it's a no brainer. I've got scripts where the only modifications I've made
since 2005 were to add functionality or increase security, never to fix
existing functionality that was broken by a change to the system. I'll take
that over Chef or Ansible (both tools I've used) any day.

~~~
ropable
Would be very interested to review server hardening Bash scripts. Have you
published anything on Github or elsewhere?

~~~
devishard
I have, but I don't want to associate this HN account to my GitHub account,
sorry. :/

~~~
aalbertson
throwaway github account then? :)

------
Someone1234
Why do people install fail2ban then disable password based authentication
entirely? I legitimately don't understand the purpose.

Also, they complain about log spam (from failed SSH attempts) this is one
reason to move SSH to a different port. It does NOT increase security, but it
DOES reduce log spam from bots trying for easy targets.

~~~
treerunner
I always change the ssh port to something other than 22. It has always seemed
to work well for me for most automated attacks. Perhaps this is not advisable
for some reason?

~~~
carlisle_
There is a slight security concern to binding it to a port above 1024 in that
a non-privileged user can bind to that port and MITM.

Here is a good summary of the different options you have with ssh and choosing
a port:

[http://serverfault.com/questions/619898/should-i-change-
the-...](http://serverfault.com/questions/619898/should-i-change-the-ssh-port-
to-1024)

------
nblr
Fail2ban? sshguard? unnecessary. Just disable ssh passwd auth (which generally
is a good idea) -> done/done

If you don't like lognoise from ssh scanners (even if you disable passwd
auth), move your sshd port to some random high port and make note of it in
your ~/.ssh/config

Generally: if in doubt, take the more simple and elegant solution to a
problem.

~~~
josho
I agree that fail2ban for ssh seems unnecessary. But, it also provides
monitoring for other services like http and common exploits.

I'd be interested in learning from the community if fail2ban adds much value.
As I've looked into the service, it seems like simply running the latest
security patches obviates the need for fail2ban.

~~~
treerunner
I run a web server with some 50+ Wordpress installs on it. You better believe
Fail2Ban is necessary. Without it all resources would be consumed by brute
force attacks. If someone knows of a better way I would like to hear about it.

~~~
snowwrestler
We use the WordFence module to block brute force attacks, seems to work fine.

I cannot believe that Wordpress _still_ ships without basic rate limiting on
its login form.

------
teddyh
I prefer the “Securing Debian Manual” – it’s an _official manual_ from the
Debian project.

[https://www.debian.org/doc/manuals/securing-debian-
howto/](https://www.debian.org/doc/manuals/securing-debian-howto/)

------
raimue
Be aware fail2ban does not handle IPv6 at all with its default configuration
on Debian/Ubuntu.

[https://github.com/fail2ban/fail2ban/issues/1123](https://github.com/fail2ban/fail2ban/issues/1123)

~~~
codelitt
You're right. Took mwpmaybe's suggestion for the time being and only will
accept IPv4 from ssh for now till that's fixed.

------
seagreen
Great tactical advice, but what a sad situation to be in. "Run this command,
then run this command, then run this command ..."

There should be a single configuration file (or set of files) that
declaratively describes the whole state of the machine. That way the exact
situation of the server can be reviewed by just looking at files, instead of
trying to poke and prod at the machine to see what commands have been run over
the last X weeks.

~~~
benplumley
There is a set of files that describes the state of the machine, it's called
the filesystem. Anything less doesn't describe the whole machine. The 'poking
and prodding' is just a convenient way of querying the very small parts of the
filesystem that are relevant to that query.

That said, a script that pokes and prods the right places and reports a
machine's 'security factor' and prompts improvements would be cool (and
probably already exists).

~~~
rdrake
> That said, a script that pokes and prods the right places and reports a
> machine's 'security factor' and prompts improvements would be cool (and
> probably already exists).

[https://cisofy.com/lynis/](https://cisofy.com/lynis/)

------
p8donald
Since I changed the default SSH port of 22 to something else (like 4422), I no
longer get any of these drive-by attacks and don't need fail2ban anymore.

I also like to set up a simple Monit configuration to alert me about high cpu
usage or when the disk space is about to run out. Instead of emailing me these
alerts (and also weekly reports) I've configured Monit to post them to my
Slack team of 1.

[https://peteris.rocks/blog/monit-configuration-with-
slack/](https://peteris.rocks/blog/monit-configuration-with-slack/)

~~~
kingosticks
You should still use fail2ban.

[https://news.ycombinator.com/item?id=11854576](https://news.ycombinator.com/item?id=11854576)

------
adrianmsmith
What's the reason for using a firewall?

Assuming that services which shouldn't be accessible to the outside only
listen to localhost not the network (e.g. MySQL on a LAMP stack), isn't that
sufficient?

(Honest question, I don't have much experience with syadmin.)

~~~
raesene6
So there's a couple of reasons to add a firewall.

1\. If an attacker gets unprivileged access it can slow them down (if properly
configured) in getting new tools onto the system or adding a shell.

2\. If a configuration error results in a service being started on a network
accessible interface by accident the firewall gives you a bit of defence in
depth protection against unauthorised connections to that server.

3\. you can also use it for logging activity to feed into other systems.

~~~
MichaelGG
>1\. If an attacker gets unprivileged access it can slow them down (if
properly configured) in getting new tools onto the system or adding a shell.

That only works if you have an outbound firewall. Which is very onerous -
you'd either have to whitelist destinations (package repos, but what if you
want to validate arbitrary certificate's CRLs?) or whitelist applications (but
not wget etc.)

------
elbear
Here's an Ansible role (I made it) that automates the steps described in the
article: [https://github.com/LucianU/ansible-
secure](https://github.com/LucianU/ansible-secure).

~~~
edtechdev
See also [https://github.com/geerlingguy/ansible-role-
security](https://github.com/geerlingguy/ansible-role-security) mentioned
above

------
tobltobs
Can somebody help me out with this question: The default config for
unattended-upgrades seems to not enable reboot even if a reboot would be
required to activate the upgrades. Wouldn't that had made quite a few
important upgrades in the last years effectless if they server did never get
rebooted?

~~~
btgeekboy
Not sure if this is still true, but I've also seen cases where Ubuntu will
happily continue to install kernel updates as they come down the pipe, right
up until /boot is full of old kernels and ramdisks.

------
walrus01
For those saying "why fail2ban?", fail2ban can be used for a great deal more
than just watching the sshd log. You can activate fail2ban rules for apache
and nginx which help significantly with small DDoS, turning spurious
traffic/login attempts into iptables DROP rules. And a lot of other daemons.

~~~
KB1JWQ
At least one log parsing tool I've seen in years past was vulnerable to log
injection attacks. Hilarious proof of concept to own a box by way of PTR
record.

I haven't checked to see whether fail2ban suffers from this model or not.

------
rodolphoarruda
I'd be more curious to see a "My first 10 minutes on an Ubuntu desktop"
version of the article.

~~~
Shorel
Or alternatively: "My first 10 minutes on an Ubuntu desktop (for users who
don't hate Unity, which includes myself)".

In my case: Change the Switch workspace keys from Ctrl+Alt+Arrow keys to
Super+ Arrow keys. Remove LibreOffice, install WPS Office. Remove
Transmission, install Deluge. Install indicator-multiload, indicator-sound-
switcher. Install Kodi. Install Steam.

A lot more stuff, but I have not written it down. =)

~~~
Scarbutt
First time I hear about wps office, why do you prefer it over libreoffice?
mobile support?

~~~
Shorel
It's faster (C++ vs Java) and more compatible with the documents I have to
open.

It also pleasantly surprised me once:

I received a Powerpoint presentation, and went to a customer meeting, where
said presentation had to be presented.

I plugged the HDMI to VGA adapter for the VideoBeam to the laptop, started the
presentation, and the presentation was running in the external display, while
the laptop display was still showing the normal 'Powerpoint' view. I could
load the web browser in the laptop display to check some things while the
presentation was still running undisturbed and the speaker and the audience
was happily unaware of it.

That level of professional use in software was something I did not come to
expect in Ubuntu for any third party software. In fact I don't know if the
other Office suites (including MS) have that feature. I guess they do, but
still.

It is wonderful when everything just works as intended.

------
babuskov
> First we'll want to make sure that we are supporting IPv6

How does that help security?

~~~
stqism
That comment was in regards to ufw as it doesn't support adding IPv6 rules by
default in commands in older Ubuntu.

If you don't enable IPv6 and the server supports it (odds are it does) all of
the benefits of using ufw at all are totally ignored on IPv6.

------
jboynyc
I'm finding that another important step is this one:

apt-get install etckeeper && cd /etc && etckeeper init

Keeps your /etc under version control so you know what kinds of configuration
changes you've perpetrated.

~~~
eropple
I used to use this, but I found that it's significantly less useful than a git
repo with my server-specific Chef cookbook in it.

Manually modifying servers was never a good idea; it's worse now, even with
tools like this.

~~~
jboynyc
I suppose that's true, but in the field where I work (social science), servers
are mostly spun up to scratch an immediate and idiosyncratic itch, so
configuration tends to happen organically.

I agree that's probably not a good idea, and learning Pupchefsible is well
worth the effort. In the meantime, though, there's at least some degree of
reproducibility with etckeeper.

~~~
eropple
I do this stuff professionally, and I've learned the hard way that you either
have a reproducible environment or you don't. etckeeper _isn 't_ reproducible.
Actually rolling back with something like etckeeper is much, much more likely
to break something (by deleting a config file used by a newer service, say)
than to save you. If it did something like separate branches for each service
or component within /etc I might be more sympathetic...but at that point you
have half of a CM system already and might as well just go the rest of the
way.

If you need reproducibility without a CM framework, keep backups of your
machines.

------
tmaly
I have been meaning to write up a similar guide.

I would like to recommend using just iptables instead of ufw, I had a case on
my vps where an update to ufw failed and then the firewall was not working.

With iptables, install iptables-persistent package so they are saved when you
do restarts. Do not try to block entire country ip ranges as this slows the
machine down substantially.

fail2ban is great, I would recommend looking at some of your system logs to
figure out new rules to add.

~~~
vidarh
Ferm [1] is wonderful as an iptables frontend. Apart from making it simpler to
read, it can avoid a lot of repetition with handy shortcuts such as ability to
group things. E.g.:

    
    
        proto tcp dport (smtp ssmtp qmtp pop3 pop3s imap2 imap3 imaps) ACCEPT;
    

.. creates rules for each of the ports listed. You can use multiple groups in
the same statement as well (lets say all the services above for some crazy
reason also listened to udp - you'd just replace "tcp" with "(tcp udp)").

Being able to set variables is also fantastic. E.g.:

    
    
        @def $DEV_PRIVATE = (eth0 eth1);
    
    

[1]
[https://github.com/MaxKellermann/ferm](https://github.com/MaxKellermann/ferm)

~~~
zhynn
Also, FireHOL [1] is an interesting option. It also has a DSL for managing
firewall rules. The custom service port definition is a little weird, but
overall I like it better than ferm.

[1] [https://firehol.org/](https://firehol.org/)

------
jtchang
Why don't they disable root logins with password period and only allow SSH key
authentication?

Also if you put a passphase on your SSH key does that mean you have to enter
it every time you want to SSH to the server (in order to unlock the key) or
does it stay cached on most SSH clients (ssh on mac terminal, putty on
windows, etc).

Isn't watching failed logins kind of useless? I think it is more important to
see what successful logins were made.

~~~
Lockyy
If you put a passphrase on your ssh key you only have to enter it when you
initially add it to your ssh-agent.

~~~
vmarsy
yes, ssh-agent will let you enter the password once, and then it won't prompt
you anymore.

(see [https://help.github.com/articles/working-with-ssh-key-
passph...](https://help.github.com/articles/working-with-ssh-key-passphrases/)
)

------
drzaiusapelord
Its a tradition to nitpick these kinds of lists. Here's my take.

>I generally agree with Bryan that you'll want to disable normal updates and
only enable security updates.

Hmm, fairly certain the Ubuntu (and others) don't do major product updates or
API breaking updates via apt-get. You shouldnt have to worry about breaking
anything if you use normal updates. This seems a bit too conservative for me
and leads to problems down the line of being on an ancient or bugged library
and then having to do the update manually later, usually after wasting a
couple hours googling why $sexy_new_application isn't working right on that
server.

He setup an email alert, but not an smtp to actually send it. Also, OSSEC
takes a few seconds to install and is much nicer than emailing full logs.

Lastly, fail2ban is becoming a sysadmin snake-oil/fix-all. Its use is
questionable in many circumstances. There's a real chance of being locked out
of your own server with this. If people are recommending it, they should be
giving noob-friendly instruction to whitelist their IP at the very least.

------
taf2
Not sure if others feel this way but adding this line to sudo never felt right
to me...

deploy ALL=(ALL) ALL

I usually instead limit the deploy user to a smaller subset of commands e.g.
the init.d script to control a service.

obviously if someone gained access to deploy user we're probably sol anyway...
but it just makes it seem safer... we have a to login as an ops user to
install or update things on the boxes.

~~~
codelitt
Someone on /r/netsec rightly pointed out that you shouldn't ever add a user
directly to sudoers anyways. You should add them to the sudo or wheel group.
I've since updated the article.

What I've described is a more of a base, but according the Principle of Least
Privilege you could go even one step further and do what you're suggesting.
You'd probably want to have a couple of users though. An admin user, a deploy
user, and a maintain user all with different privileges.

~~~
vacri
> _you shouldn 't ever add a user directly to sudoers anyways_

What was the reason for that? I have the deploy user able to run a couple of
individual commands without a sudo password (scripts that run canned updates,
to be initiated from a buildserver), but I don't see how it would improve
things to use a group instead that only holds that user.

------
kikimeter
I created a script that does almost everything automatically using Ansible and
Ansible Vault : [https://github.com/guillaumevincent/Ansible-My-
First-5-Minut...](https://github.com/guillaumevincent/Ansible-My-
First-5-Minutes-On-A-Server)

------
dawkins
I always worry that adding 2FA could make your machine inaccessible if
anything happens to google-authenticator in this case. Maybe it's a little bit
of paranoia but I don't like the idea of giving control over my ability to log
into my server.

~~~
throwanem
It'd be perfectly reasonable if libpam-google-authenticator relied on Google's
infrastructure, but despite the infelicitous name, it does not; it just
implements the server side of TOTP. The authentication flow is identical to
any other correct TOTP implementation, and you can use any compatible client;
no integration with Google services or infrastructure is required. (In fact, I
don't think it's even possible.)

Speaking of TOTP without Google, if you use iOS and find the Google
Authenticator app unsatisfactory, try Authy. It's good stuff, and well worth a
few bucks.

~~~
stephenr
Hurricane Electric's network tools iOS app is free and has an OTP client built
in, with iCloud Keychain sync.

OTP Auth is another excellent, free OTP client.

Edit: clarified both are free.

~~~
throwanem
While I'm not in the market for an OTP client, this is the network tools app
I've been looking for since I bought my first iPod Touch. If you have remote
beer-buying support, point me to it.

~~~
stephenr
You're welcome. I only discovered it myself recently when testing their IPv6
tunnel service.

no beer required :-)

------
amelius
For protecting against brute-force login attempts, I use sshguard [1]

I really think this should be installed by default on distros like Ubuntu.

[1] [http://www.sshguard.net/](http://www.sshguard.net/)

~~~
MichaelGG
I can't see any benefit, what am I missing? Put SSH on a port that's not 22
and done, no more mass scanning. The only thing SSHGuard has ever done for me
is to lock me out when I was accidentally using the wrong key.

~~~
oofabz
If SSH is on a non-standard port, it is still possible to brute-force access
to the server. You will see fewer automated attempts but you are still
vulnerable to a motivated attacker who port scans you and finds the SSH port.
Such an attacker is less common than automated scans but is more of a threat.
With Sshguard, you are no longer vulnerable to this type of attack at all, no
matter which port you run SSH on.

~~~
MichaelGG
But brute forcing any reasonable password or key is so far-fetched as to not
be something to even consider.

~~~
oofabz
If you can make hundreds of login attempts per second, and you can keep at it
for days/weeks/years, you can get through some pretty big password
dictionaries with lots of variants (e.g.
password/p4ssword/passw0rd/p4ssw0rd/etc.).

------
catmanjan
One of the suggestions is to make sure your public key has the .pub extension,
and they imply that if someone didn't include the extension they would be
reprimanded - any reason for this in particular?

~~~
AlexCoventry
Having a reliable convention like that reduces the risk of someone
accidentally copying their private key to a server.

------
usaphp
> "You should never be logging on to a server as root."

Can someone explain me, let's say I disabled password logins and only allow
login via a key, what are potential downsides of logging in as a root?

~~~
ghayes
Well for one, every command you run has root privileges (instead of requiring
sudo). Every process you spawn has root privileges. You're safer keeping your
privileges limited and sudoing when required. Also you lose your real-life
audit log if multiple people log in as root.

~~~
usaphp
But when I tried using a regular user to run anything on a server - it still
requires me to do sudo, so the process will anyway have root privileges, no?
So I end up typing sudo most of the times becauase most of processes do not
work without root priveleges

------
SadWebDeveloper
Forgot to check if the server isn't backdoored. You will be surprised how many
providers add many backdoors and monitoring systems you don't need (m looking
at you AWS guys).

------
Theodores
I would be annoyed with a cryptic Audi password. I would prefer
'BatteryHorseStaple' passwords. Anything I can't remember gets written on a
post it note and put next to my screen with what it is for. This is my
behaviour and the problem with cryptic passwords is that there are others like
me, willing to keep a good password secret and not willing to be so secret
about a clumsy, easy to crack by machine but impossible to remember password.

~~~
codelitt
Get a PW manager instead. No passwords should be stored in your head (because
every one should be different) and they should be stored behind encryption --
definitely not plain text nor sticky notes.

[http://keepass.info/download.html](http://keepass.info/download.html)

------
javajosh
It may be useful, at step 0, to check out the server and see _basic server
orientation_. Which Linux is it (cat /etc/*-release)? How much ram and disk
(htop, df)? How is the filesystem setup (mount)? What packages are already
installed (dpkg -l)? What processes are running (ps aux, htop)? What did the
last root, including me, do (history)? I also like to know where is the box
physically, roughly (tracert, run locally).

------
archon810
My biggest concern with being on a VPS like Linode, once you're all done
securing yourself and binding services to the local LAN IP, is an attack from
within the network. The VPS you own is also accessible by others on the same
subnet, contrary to what you might assume.

I'd love to see a ufw guide for whitelisting only your own internal IPs to be
allowed access to any services for ultimate security.

~~~
nisa
Not sure what you mean but ufw by default blocks everything on your interface
so other machines in the local subnet shouldn't have access.

If you want to have more security and no (or just a single) outgoing service
configure OpenVPN with TLS and put all your local services in a local subnet
for your machine. So not even a portscan can find something.

------
overcast
Very useful, most of this stuff is pretty common for anyone who has done any
regular sysadmin work, but definitely good to have a checklist.

------
cleeus
echo "set background=dark" > /etc/vim/vimrc.local

~~~
raimue
All users in your team are forced to use dark terminal backgrounds?

~~~
JoshTriplett
You can actually autodetect whether the terminal background is light or dark.
For any xterm-compatible terminal, write '\x1b]11;?\x07' to the terminal, and
it'll write back a string telling you the foreground color (for instance,
'\x1b]11;rgb:0000/0000/0000\x07', which if written back would set the
foreground color). If the color matches 'rgb/RRRR/GGGG/BBBB', compute the
luminance of that color, and assume a dark background if <0.5 and light
otherwise.

~~~
moosingin3space
I didn't know about this, but from now on, when writing CLIs that use color,
I'm going to take this into account!

~~~
JoshTriplett
Awesome; more tools should do that. Some caveats, though:

* You might not get a response from every terminal, so limit how long you wait.

* If you don't already have echo turned off, turn if off before sending the sequence, because otherwise it'll be visible as though the user typed it.

* You don't know that the color will use the "rgb:RRRR/GGGG/BBBB" format (a terminal can return anything XParseColor can understand); just read the string from the escape to the terminator, look for 'rgb:', and ignore formats you don't understand.

* To calculate whether a color is "light" or "dark", see [https://en.wikipedia.org/wiki/Luma_%28video%29](https://en.wikipedia.org/wiki/Luma_%28video%29):
    
    
        dark = (0.299*red + 0.587*green + 0.114*blue) < 0.5;

~~~
moosingin3space
Have you measured how long typical terminals take to respond?

Regarding the third point, it might be a good idea to just feed it to
XParseColor and process it from there.

~~~
JoshTriplett
> Have you measured how long typical terminals take to respond?

Arbitrarily long. Consider that a user might run your application over SSH via
a high-latency network connection. Better to just handle it asynchronously.
Your input loop needs to watch for escape sequences anyway, so watch for that
one and process it when or if you see it.

Sadly, that only works for interactive screen-oriented applications, not run-
and-exit command-line applications that want to use color.

> Regarding the third point, it might be a good idea to just feed it to
> XParseColor and process it from there.

That assumes you have libX11 and an X Display available. The former is a heavy
dependency for a CLI application, and the latter requires you to connect to
the X server.

I'd suggest just manually handling the common case of "rgb:R/G/B" (where each
component may use 1-4 digits and requires scaling accordingly), and then deal
with anything else if your users actually encounter it in the wild.

~~~
moosingin3space
I assume XParseColor would therefore limit portability to OS X or Wayland
Linux. I stand corrected.

I was speaking of this regarding screen-oriented termbox/curses applications,
not run-and-exit applications.

------
VLM
Technically you don't need the root password, you can always password recovery
if you have access to the box. And how exactly did you lock yourself out of
every account with sudo? Of course there's always "messed up my ldap or
general network settings, can't log in to fix them". There's nothing wrong
with setting your root password to a random string and throwing it away, after
verifying your sudo works, I guess.

I will admit to being lazy, and with full automation its faster to spawn a new
virtual image and let ansible run its course than to do root password recovery
where you boot and tell the bootloader to make the init system /bin/sh and
hand edit /etc/shadow and /etc/passwd and then reboot again, etc etc. I mean I
can set up a new image almost as fast as I can reboot an old image, and I set
up images a lot more often than I do password recovery, so...

Scrap the ssh commentary and set up ssh company wide as per stribika plus or
minus local modifications:

[https://stribika.github.io/2015/01/04/secure-secure-
shell.ht...](https://stribika.github.io/2015/01/04/secure-secure-shell.html)

"On large scale, you'll be better off with a full automated setup using
something like Ansible"

At ANY scale you're better off, unless you're experimenting or time isn't
money. It'll take longer to add the time to document and test what you're
doing by hand than to convince ansible to do it for you. If you don't document
or test you're just doomed, so its not like you can avoid that effort. With
automation this is like "first two minutes on a server" not ten.

Some people like to drop a .forward in root's homedir sending mail to your
sysadmin mailing list or yourself. I THINK but might be wrong that if you do
that you don't have to tell logwatch whom to email to, it'll go to root then
forward to the right people. More than logwatch assumes
root@something.whatever exists as an email address.

You're missing setting up your centralized rsyslog or local equivalent, your
munin/nagios/zabbix or local equivalent... I still configure zabbix by hand
because I'm old fashioned but its possible to automate that.

NTP is also missing. You can make Kerberos a very sad faced puppy if time
isn't synced. And its easy to set up to point to local trusted servers.

(Note, a post that's nothing but complaining still means the linked article is
at least 99.9% correct, it is a nicely written wide ranging TODO list)

~~~
JoshTriplett
> And how exactly did you lock yourself out of every account with sudo?

A single typo in /etc/sudoers or any /etc/sudoers.d file will lock you out of
all sudo usage. visudo helps with that, but a single mistake (including in a
sudoers.d file installed by a configuration management system or package) will
lock you out.

~~~
VLM
Yes sir, and that's why the paranoid sysadmin ssh's in, sudo su's up, THEN
runs ansible and tests that it works before unleashing the ansible (or puppet)
across the entire network.

Also if you do the "group auth" thing in sudoers then you edit that file
approximately once per employment and never touch sudoers again. Of course
that abstracts the problem into "I deleted the wheel (or sudo, or ...) group
on the ldap server and now I can't sudo up to fix it". And that's why you make
snapshot backups on the NAS, so you can roll back the image of the LDAP server
(or whatever you use locally) (and edited to add, and don't do something dumb
like use the ldap image running on the openstack to authenticate logins into
the openstack... that would be painful indeed)

~~~
hrez
I prefer lazy sysadmin that avoids all that manual ssh/sudo by making
puppet/etc validate sudoers before updating it. You can syntax check with
visudo -c.

------
thatusertwo
I have a VPS, when I first got it, it had an additional user setup for some
unknown reason. I didn't know it was there until my server was hacked by a
bot. I'd suggest adding one step of checking the /home directory or other
places to make sure no 'unknown' accounts have been set up.

------
timroy
Thanks for this article - very clear, well-motivated, and concise. I'm saving
this for myself and others.

------
windsurfer
I guess I'm a pretty big noob, but why do people recommend so strongly on
password protecting your private key? Losing it pretty much dooms you whether
or not it's password protected. It might get you a few hours or so to react
and invalidate the public key, I guess...

~~~
ryanlol
If you've got a good password on your key, then nobody will be able to use it
in years. It most certainly helps.

~~~
windsurfer
Years? How long a password would you need to make GPU cracking take years?

~~~
ryanlol
A very short one. Reasonable 10+ character passwords should remain out of
reach for years.

Also, AFAIK there doesn't currently exist any very effective GPU cracking
software for SSH passphrases.

~~~
chrisseaton
I went to lookup the algorithm GPG uses to encrypt private keys, to help
answer the original question, but couldn't seem to find that information
anywhere. Do you know what it is?

~~~
windsurfer
According to a quick stack exchange search, OpenSSL uses 3DES for encryption
of private keys.

------
feross
This is very similar to my "How To Set Up Your Linode For Maximum Awesomeness"
guide:

[http://feross.org/how-to-setup-your-linode/](http://feross.org/how-to-setup-
your-linode/)

------
z3t4
If you open up access from/to port 80 or 443, you also open up access to all
trojans/spyware/telemetry/auto-update created in the last ten years. You'll
want to limit access per user and process.

------
agentgt
It might nice if there were some cloud vendor specific addendums. For example
on rackspace you almost always want to install the monitoring daemon (it's
actually fairly decent and small foot print).

------
dmourati
Anyone remember Bastille Linux?
[https://help.ubuntu.com/community/BastilleLinux](https://help.ubuntu.com/community/BastilleLinux)

------
chrisper
Instead of using unattended-upgrade, I prefer to subscribe to mailinglists and
see when there are new securtiy updates.

One could combine that with something like rundeck where you run apt-get
upgrade.

------
bikamonki
Why not make a _certified_ secured best practice 99% covered snapshot and
share it as part of the one-click installs that most VPS providers offer
nowadays?

------
a_imho
I think 2FA is generally bad practice and quite sad it is ubiquitous in e.g.
banking and people try to shove it everywhere. It is analogous to password
rules, 8-14 characters, numbers, capital letters and other signs. Yet it is
very rare you can use a 40+ character passphrase. It gives a false sense of
added security, while being annoying at the same time imo. It is very common,
for me at least, not to have access to my phone all the time, because I left
it at home, in the car etc. Not to mention if you lose it (or someone steals
it) you have a huge pita to deal with.

~~~
tjohns
2FA doesn't have to be annoying. Take a look at Yubikey devices as an example
of how to do this right. The reality is that it is actually really useful at
preventing some common attack vectors: password reuse, keyloggers, etc.

It's even better if you're using a hardware dongle that supports U2F (or can
be used as a smartcard for SSH), because that can even prevent active MITM
attacks.

~~~
a_imho
no, my point is exactly that the 2 in 2FA is inherently annoying, because you
need to have physical access to 2 different devices at the same time.

How does it prevent password reuse? You can use the same (weak) password to
lock your phone and login to your banking account (which is again, a false
security). However it could be easily circumvented by random generating secure
passwords for users (which needs clever advertising like 2FA, because they
prefer convenience otherwise). In this case your phone is a single point of
failure. You could even argue it increases the attack surface.

~~~
pfg
> How does it prevent password reuse?

It does not _prevent_ password reuse, it mitigates the risks of password reuse
in that it adds the requirement of having physical access to a device, which
is a show-stopper for _most_ attackers.

If you're using a password manager with sufficiently complex passphrases, the
biggest remaining risk factor are targeted malware attacks (something like a
keylogger), which is something that typical SMS- or TOTP-App-based 2FA
implementions won't help you with, fair enough. Implementations where certain
security-sensitive activities require separate confirmation and where the
details are transmitted through a separate channel would mitigate this attack
to a certain degree as well. As an example, some banks in Europe provide their
customers with card readers with a PIN pad that shows transaction details on a
separate display. Banks routinely include transaction details in SMS-based TAN
mechanisms, which works as well, but is obviously not quite as good.

> You could even argue it increases the attack surface.

How?

~~~
a_imho
Meant in the general sense. More complexity, more opportunity for attacks
and/or implementation bugs. For starters you have a phone number associated
with an account already. I would wager losing your phone is nearly impossible
to prevent, while picking your passphrase is up to you. Losing your phone
could alone compromise your security, but the very least SMS leaks the info
where you bank.

I'm aware I am a minority with this opinion, but I would be really grateful if
I could at least opt out from phone based 2FA.

------
PerfectElement
Is there a similar guide for Windows servers out there?

~~~
garthk
[http://decentsecurity.com/](http://decentsecurity.com/)?

------
ec109685
It would be useful to discuss what prevents the server from being rooted
without a trace during the 10 minutes it takes to execute these steps.

~~~
VLM
Production boxes are not allowed to be plugged into bare internet or DMZ
unless they were seasoned and tested on the LAN first, and the LAN allows no
external traffic in (stateful firewall). If people on your own LAN are trying
to pown you, you got bigger problems to solve before installing another box.

In the old days this was manually moving ethernet cables, now a days this is
changing which VLAN the virtual image talks to or if you use something like
openstack that implements its own firewall at the virtualization level you
allow no external traffic in until the config and testing is done.

Also you need to verify your install media is not powned, which means you need
access to the md5sum of the media (and how do you know someone didn't MITM the
correct md5sum?) and you need to verify your md5sum program isn't powned which
means you need to verify your verification strategy isn't powned which means
this gets recursive real quick.

------
mmgutz
Hmmm ... why does root need a password? `sudo su`

~~~
molecule
_> You'll only need this root password if you lose your sudo password._

Also, good to have in case sudoers file is corrupted, _e.g._ via bad edit,
typo, _etc._.

~~~
merurr

        cd /etc
        git init
        git add *
        git commit -am "before i screwed up"

------
Hello71
1.

    
    
        useradd -m deploy
    

2\. "PasswordAuthentication no" probably won't work as you expect if UsePAM is
on.

------
brndnmg
May I suggest Ansible or whatever other provisioning tool, you can subtract 9+
minutes from the title...

~~~
cfieber
please don't.. provision once and snapshot, and deploy the snapshot

------
plusbryan
What was wrong with 5 minutes? :-)

~~~
codelitt
Nothing! (Except I doubt I can manually get it all done in 5 minutes =) )
Thanks for your great article.

We just ended up adapting your approach with a couple modifications (like 2FA)
and extending it to be more of a primer and explain the steps a bit more so
that the younger engineers understood what each step performed was doing. I
found myself pointing them to your article, but then having to explain what
was being performed and it's purpose (not a bad thing - just different
audience).

As has been mentioned, in the real world, an Ansible Playbook should be
performing these, but teach a man to fish, etc...

~~~
plusbryan
I love the article, and thanks for the credit.

------
dewarrn1
Nice guide, better comments, leaving this here for later reference.

------
cfieber
sure makes me glad all that (and so much more) happens in the first negative
10 minutes on any server I deploy.

If you are doing this after your server has launched you are doing it wrong.

------
tdalaa
Pretty useful, thanks

------
stonogo
No production server should ever be manually configured.

~~~
jeremyt
How should this be done by, say, a small team of three with no SysAdmin?

~~~
merurr
Take a day off and get familiar with ansible, it will save you tons of time in
the long run and simplify your dev/staging/prod environment for years to come

------
ck2
Don't just change SSH key requirements, also change SSH port.

Port 22 is possibly the most heavily scanned port around.

------
nanis
_Sigh_ ... "principal of least privilege"

------
YngwieMalware
I'd been using this article for a couple years when I was a Linux server
neophyte and now some of these things seem obvious to me. A good article for
total noobs.

~~~
YngwieMalware
Just realized I'm getting downvoted because I thought this was the first 5
minutes article. Whatever!

