Hacker News new | past | comments | ask | show | jobs | submit login
How I spend my first 5 minutes on a server (plusbryan.com)
960 points by plusbryan on March 4, 2013 | hide | past | favorite | 341 comments

The premise of this thing is not good advice.

1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one.

2) Everything else listed in this document should be configured by a configuration management system.

3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure.

4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does.

This stuff isn't hard. It's worth doing right.

> This stuff isn't hard. It's worth doing right.

Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.

This is an excellent request. Reading through all the comments here, it seems like a lot of people are feeling frustrated with the variety of information available and no clear way to discern what is "good".

I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-)

In the meantime, a decent source of generalized (not succinct) Ops-type knowledge can be found here: http://www.opsschool.org/en/latest/

Thanks! I really appreciate it. All to often we are quick to criticize someone for missing information without any willingness to back it up with better, correct information. The jokes goes that the best way to get an answer is to publish the wrong one. =)

That site is useless, sample pages I looked contained brief overview paragraphs and no real content. Many contained "todo" items.

So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.

First, pick a CM package.

This depends entirely upon your level of masochism and the kind of language youlike scripting in, as literally anything will do. Even shell scripts. Even Makefiles. Whatever you're most comfortable with, just start writing out configs. (you will eventually come to hate your job if you allow non-programmers to script/program in your CM, but blah blah keep shipping blah blah) Break it all out intoa wide hirearchy so you can reuse some bits in other bits.

Hey look, I wrote a crappy one! https://github.com/psypete/public-bin/tree/public-bin/src/si...

Next we implement the OP's comments.

Hey look, I already implemented it in my crappy CM tool! https://github.com/psypete/public-bin/tree/public-bin/src/si...

First push your CM tool and configs to the host:

  scp -r simplecfm-0.2 remote-host:
Then run the main config file which calls the others:

  ssh remote-host "cd simplecfm-0.2/ ; perl simplecfm examples/first-five-minutes/main.scfm"
Aaaaand you're done. Of course I haven't tested these configs and something will probably break (most likely my crappy code) but you'll get the idea from looking at the examples.

If Bastille was still in working order, I'd recommend it as a very good starting point for locking down a configuration. Actual configuration deployment setups I don't put much stock in because I'm not managing enough machines for it to be worthwhile. Would love to hear from someone who manages a large farm/cluster/VM hosting on a very clean and straightforward way to manage configurations. Last I looked, most setups were custom, or some hobbled together packages that didn't "hang" together very well.

You are right, this is a better path

But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool.

Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.

Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.

It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet.

Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.

Ok, great, that's the spirit

You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)

The problem is that most of the recommendations are bad.

Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior).

The rest is just bad advice.

Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?).

User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server.

Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers.

Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses.

You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.

I agree on a lot of things, except LDAP. Of course, it sounds like the ideal way, but in practice, there's a matter of scale to take into account.

Frankly, you need a lot of developers and servers before investing the time to setup a ldap deployment, integrating it with logins, and spending the inevitable hours debugging why nobody can access anything anymore, becomes more worth it than "just rsync/pssh into all servers and edit /etc/passwd".

Actually, if I had to do it, I'd use chef to automate creation (and destruction) of user accounts over an LDAP any day. Chef can be a pain to learn and use, but any sort of LDAP is even worse.

I would probably stand up an LDAP server after I had 4 or 5 user accounts to deal with, or, more importantly, more than, say, 3 servers to deal with.

I think there's a lot of fear and hate surrounding LDAP, but that's mostly for historical reasons. LDAP has gotten a lot easier to set up. Even in 2009 a colleague and I set one up (using openldap) and had other machines authenticating off of it in an afternoon. It's gotten even easier than that since then.

And hell, you should be using Chef to set up your LDAP master and slave. So once you have the config correct once, you can bring up another machine without trouble when needed.

My friend and I are rebuilding an old, text-based browser RPG, occasionally with another friend helps out with art and game content.

We're on LDAP, which we use to SSH into our ec2 servers, and which we use for authentication when we deploy using `git push production master` to a GlusterFS cluster. We're running our LDAP, application, and file servers on Gentoo. We can easily add new accounts, and we have it set up with group permissions (so the friend can deploy game content to test but not prod, for example).

I refuse to believe that LDAP is "too complicated" or "has to scale before it's useful", when a couple of guys can, in their free time, set it up for themselves. It's saved us a load of time in managing servers that would otherwise take away from the limited time we have to actually write code.

It's also a whole lot cleaner than a bunch of Chef scripts running a script across a quantity of servers; using Chef can too often be a crutch to fixing the actual issue.

You are assuming that they have several developers. I bet it's one or two

If you're dealing with more, let's say, 5, your suggestions became relevant

Also you are assuming this servers are staying with the company, which may not be the case

"User account management is a pain, so that's why we have things like LDAP"

Which is a bag of hurt in itself.

You are assuming that they have several developers. I bet it's one or two

The article presents this as general advice and doesn't make any mention of a team size that's appropriate for the recommendations. Maybe this guy has only 1 or 2 developers working on boxes, but IMO for any number of people >1 you don't want shared accounts.

Also you are assuming this servers are staying with the company, which may not be the case

In many cases they will be, and, again, the article presents this as general advice. If a server is being configured for a client and sent out, then a general process of setting up local user accounts probably doesn't make sense anyway (unless you only have one client).

Agreed that LDAP isn't the most friendly of things to set up, but there are how-tos for common use-cases, and if you have any skill as a sysadmin, you can do it. As soon as it's set up, it's simple to maintain. Manually keeping accounts in sync across multiple machines is rarely simple. Maybe LDAP is overkill for a handful of hosts and user accounts, but if you expect to grow even a little, expect to need some kind of centralized user account system.

If you only have one or two developers, then you have all the more reason to automate the boring and mundane tasks, since the time savings are even greater as a percentage of total time available!

I just wrote an ansible script to do these sorts of things yesterday. Now, whenever I get a new server, I just run the script with its ip, and boom, provisioned.

>Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.

You're right -- many used cfengine. Still others used a custom 'config' rpm / deb that deployed all of these files everywhere.

Automated configuration makes sense not just for repeatability, but for auditability and documentation. Especially when you are going to 'hand the server over', the next admin should be able to know what you've changed.

Also, disallow password-based access to everything (use the keys, Luke.)

I guess it all depends on how often you bootstrap a new server. If you do it infrequently, keeping a configuration management tool updated is probably more work. I do it perhaps every two years and keep a similar list (I try to keep up to date if I change something in the server configuration) for that purpose.

Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.

This line of thinking represents a logical fallacy - no one claimed that anything other than Chef or Puppet is "doing it wrong".

I would love to see a link to a counter-post of how to do it your way. Not generalities, but a specific guide.

For #4, wouldn't the only change when someone leaves the organization be to remove their key from authorized_keys for the shared account? Why would anyone else have to be updated?

It's a basic principle of security. Each account represents one person so that you have a full audit of who did what by watching the activity of a given user account. If everything is run as "devops" user for example, you have no idea who actually performed a given task. Was it Bill, or was it an automated job? PCI-DSS requirements also affect your model for user accounts (hint: shared users are often not compliant).

From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.

Why do you say a VPN server is more secure? Which one?

I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.

I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits.

Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack.

Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least).

Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.

Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.

anyone who compromises [the VPN server] is in a pretty good position.

Sure. But without a VPN, anyone who compromises even one of your other hosts is in the same position. It's a lot easier to audit a single-purpose VPN server for possible security issues than it is to audit all the application code running on the rest of your production systems.

And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.

I wouldn't. And even if you're right, getting a VPN login still doesn't get you anywhere. You still have to be able to ssh to the rest of the hosts. That's why we do security in layers.

Ok, just in case it wasn't clear: I am asking about the advantages of having a VPN server as a bastion host instead of another SSH server.

You would ssh to the bastion host, and from there to internal hosts.

With the appropriate ssh config at the client end, the tunneling through the bastion can be scripted away (using the ProxyCommand directive and RSA keys).

Routing transparency. Sure, you can script a bunch of tunnels, but it's nice to handle routing at a lower layer. Having worked with both setups, I vastly prefer the VPN solution for ease of setup, use, and maintenance.

Also, different attack surfaces. Two layers of the same security measure (ssh) is, all else equal, not quite as good as two layers involving two different measures (VPN, ssh).

Same question - which open source VPN solution would you recommend?

I liked the idea of adding VPN layer to SSH, so would like to get as much advice as possible :-)

I have experience with many open source VPN servers. The purpose is a bit different -- we provide a VPN service to home users to encrypt their internet traffic. But the same problems should apply.

OpenVPN is the most compatible with a variety of clients. OpenVPN runs in userspace, so the clients for each OS and mobile platform interoperate well. The downside is, it does require a client program to be installed and configured. It's considered very secure, using SSL. Since it's userspace, moving large amounts of traffic means more context switching and higher cpu usage. Despite that, I've found it to be faster and more stable than the alternatives.

L2TP/IPSec is built in to most clients -- Windows, OS X, mobile. But every implementation is different and it's hard to configure a server to work with all of them. There are also more moving parts -- an IPSec server (openswan, strongswan, or racoon), and L2TP server (openl2tpd, xl2tpd) and a PPP server (pppd). IPSec seems to be a secure protocol but it's very complicated. I tend to distrust complicated security.

Pure IPSec has many of the problems of L2TP/IPSec with the added problem of difficult to configure in Windows and OS X.

PPTP is not performant or very secure. Other than the fact that almost every client supports it, I see no reason to use it for a new VPN.

I've had success with OpenVPN. I don't know that I'd specifically recommend it over other options, as I don't have much experience with anything else.

Yes, I agree, I was just giving an example of how an additional layer can help protect against automated attacks, even for highly-secure services like SSH.

I also agree that SSH is less likely to have flaws than most VPN software. But on a properly configured bastion host, by-passing the VPN would just put you in a position where you can attack SSH. You would still need to by-pass SSH to access production servers.

Which open source VPN solution would you recommend?

> and anyone who compromises it is in a pretty good position.

You are assuming the VPN host is trusted any more than most people trusts random servers on the internet.

I am, just from my experience in real-life companies. It takes an awful lot of discipline to treat servers as if they were exposed to the public internet when you know full well that they're not.

This is just faulty logic.

openssh is one of the most secure projects. It's developed by the security obsessed (and I mean it in a kind way) folks at OpenBSD.

I, for one, am ready to place for more trust in openssh than in any VPN daemon. The most commonly used ones are propitiatory.

What if there is a 0-day vuln (not exploit) for these VPN daemons? That far more likely. "Securing" ssh with a VPN is just one step beyond of security by obscurity.

If you are afraid of script-kiddies and scanners, let your sshd listen on a non-standard port.

Port scanners do generally scan non-standard ports too, you know...

I don't think that exposing SSH to the internet is that bad, but your argument is not sound - requiring a VPN does add security, because if there happened to be a vulnerability in it that allowed access, all it would do is expose SSH on the machines (I'm assuming you have proper firewalls set up), which you are advocating making public in the first place.

Saying it adds no security is false, because you'd require an unpatched vulnerability both in your VPN server and in the SSH server simultaneously. A zero-day in one is possible, but in both at the same time is far, far less likely.

> "Securing" ssh with a VPN is just one step beyond of security by obscurity.

You're not securing ssh with a vpn. You're adding another layer. ssh is still secured by all of ssh's existing protection.

ssh behind a vpn requires that someone both compromise the vpn _and_ compromise the ssh service to gain that access that, without a vpn, would require them to only compromise the ssh service.

Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited

So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.

When someone advocates using a VPN, that doesn't mean not using SSH too. VPN + firewall just restricts who has the potential to try to SSH to you, and provides additional protection and central access control/management.

Agreed, anyone who has touched PCI DSS would agree you need to associate access with a human user. This would not work. If you look at the security logs it won't differentiate between which keys were used for that generic account.

It's a good idea to use fail2ban even if you use VPN.

If they have had access to a shared account, how much work are you willing to put in to verify that there's nothing in that shared account that will get executed by another user later that will quietly reinstate a key?

A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater)

After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.

But if you give people any access to any system surely this is a concern. As a software dev, maybe I've inserted something into one of the build scripts that quietly re-opens my backdoor to the source control server...

I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?

It's commonly stated that 9 out of 10 security threats come from employees or other insiders. You should assume malicious employees. Sooner or later you will hire the wrong person.

Now, you must also have a functioning system, and so you may take risks by leaving things more open than you would like if you don't have the resources to thoroughly lock everything down.

But wherever locking things down further costs you very little, you should take the opportunity. And elsewhere you should asses what level of protection you can afford. Ultimately it is a cost-benefit analysis. Many risks are not worth spending money protecting against. Others are vital.

But even disregarding malicious users: Individual user accounts is not just a protection against malicious users, but against careless users. When someone sets a password that gets guessed, you want to be in a position where exploiting that persons credentials is as hard as possible, and tracking down actions taken via the account is as easy as possible.

And yes, you could insert something into a build script. But if the build script is committed, and the commit was pushed from a named, individual account, you're now at the risk of going to jail. Creating deterrents is often a sufficient risk mitigation strategy to be acceptable.

Quite the contrary: You're probably going to end up with documented procedures for deploying software that are simplified, follow existing standards/best practices, and don't rely on complex stone soup build/init scripts concocted by inexperienced developers (and I've seen some doozies).

A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.

Not quite sure I understand what you're saying here, I'm not talking about the software being produced, but the systems used to produce it.

What I was trying to say was that there's not really any way for you (server admin guy) to know if I (software dev guy) have inserted something malicious into a script that all the other software folks run constantly (software build system, NOT server build/init script, NOT deployment script).

This is not about the end-user's privileges, or server set up, just how in a team-base software dev environment you're probably going to have to have a measure of trust for your employees.

I see, but I think the same principle applies, even in this narrow case. As a server admin guy or fellow software dev guy, I have to trust that any code you've written has been properly reviewed before checking it into a repository that I pull from. Fortunately, version control tools make this trivial, but you're right, the policy and infrastructure supporting it has to be in place, otherwise you're depending only on voluntary peer review.

Note that as an attacker, there's a high risk of exposure and identification in the scenario you describe, and that's a good thing. A well secured system shouldn't merely prevent attacks, it should also protect innocent users from suspicion (another reason why shared accounts are discouraged).

But they still would know the 'deploy' password needed for sudo access. And while you could be relatively sure that they couldn't get access, you still couldn't be completely sure since they did have sudo access to begin with. So, the best thing would be to change the shared password. That could be avoided with non-shared accounts.

Are you insinuating that the user could have used sudo access to install a backdoor of some sort? If so, changing the password won't stop them either. Am I missing something?

Is there a free and recommended configuration management system that does all this?

Chef or Puppet are common choices. There are many others, but those two are modern, have large communities, and decent documentation. There's a decent chance someone has already open sourced a cookbook/module for each of many of these items!

On linode, which he is using, there is one built in - they allow shell scripts to run on first boot which could replay his commands as they are without many changes:


Yes, stackscripts are a great way to take advantage of automation. :-)

We use Chef, many people run Puppet.



Puppet, chef, cfengine, ansible, and salt are a few.

Salt is wonderful.

I just ready your post out of context. Made me chuckle.

No, chef/puppet/etc are configuration management tools. They automate the manual typing of server setup commands, and then verify that the system stays that way.

They are not security tools. So you're on your own on what to actually tell the tools to do. "Install chef" is not a security tip. It's a repeatability tip, so you can get your system up to a known state repeatedly.

For the security side of things, you're back to figuring out what the right steps are, no matter how they're installed.

Voting for Ansible or Puppet here.



Highly recommend Chef due to its Ruby DSL.

I recommend Chef - http://www.opscode.com/chef/

I'm a good developer, but setting up systems is something I do so infrequently that I always have to relearn it, and I am positive I could be doing things better. Do you have a link to an article that explains how to go about performing some of these ideas you mention?

Beginner or not, you should probably use visudo [1] instead of

  vim /etc/sudoers
for the sanity checks that it provides, if nothing else. A botched edit of /etc/sudoers that locks you (along with every other user) out of administrative access is an unpleasant way to learn this.

[1] http://linux.die.net/man/8/visudo

Similarly "ufw allow from {ipaddress-you-will-access-from} to any port 22" sounds like a good way to accidentally lock yourself out unless you have an out of band backup

While true, I'd like to note that the article's author does have OOB access.

FWIW I never understood UFW over straight IP tables, is it really easier to read?

IMHO, yes. At least on Ubuntu, it's never been too clear to me how I should save my rules so that they come back on startup.

The ufw man page is pretty decent.

I've not tried anything complex with UFW so I still use iptables on my bastion host that handles my vpn tap. It's not terribly complex to make rules come back on startup (but probably more involved than one would hope).

For anyone else that followed the thread to this point- this advice on bringing iptables back up on reboot worked for me http://rackerhacker.com/2009/11/16/automatically-loading-ipt... YMMV

This is how I run iptables on a sufficiently large network of machines.

The advice is not complete. IPv6 is real and really works most of the time these days. Back up your ip6tables to a file too. I like /etc/firewall-4.conf and /etc/firewall-6.conf but it's down to preference.

Know about iptables-apply too, lest you be caught unaware.

He does he's on a Linode he has LISH.

You're absolutely right. Thanks for catching that!

In certain cases you may not even need to visudo :) AFAIK, Ubuntu's default /etc/sudoers contains this :

%admin ALL=(ALL) ALL

So you just need to make sure your 'deploy' user account belongs to the admin group and you're good to go.

Sure, as long as you remember to use

    EDITOR=emacs visudo

On debian or ubuntu you can also include a user on the sudo group:

  usermod -a -G sudo username

I prefer the "adduser" way (easier to remember):

$ sudo adduser username sudo

Firstly, a nice checklist. Easy actionable steps, repeatable, and pretty much most of what you need.

Secondly, you are about 4-5 hours away from learning puppet (or Chef) and making this checklist into actual code.

Thirdly, you now have a checklist of items that you can use in a job interview if you get the oppertunity to gain a new-hire or an intern.

Lastly, good on you for submitting this to a peer-review on HN. We can be a picky lot.

TL;DR Checklists are a good first step for building a proper config management system.

We can be picky, but with good reason as security is an exact science and a costly one to get wrong. There's so much conflicting and out right bad advice posted online these days that sometimes it takes a picky community to help clarify the best practices.

For what it's worth, some of the advice given in that article was worth mentioning (eg fail2ban, it's a great tool). But the shared account suggestion was the complete opposite of how you should be managing user accounts as you lose audit trails. And for that comment alone, I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate.

That article has generated a lot of good discussion though. So even if just indirectly, it's been a valuable contribution to HN.

Picky is good! We never grow without constructive critisim. And a community of practice may yeild better results than an individual (at least for simple things).

> I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate.

Agreed. But the same could be said of everything; hackers and engineers, empiricists both.

Some of the advice in the article would seem apporpriate to someone who had never worked with another senior engineer. No fault of the OP.

System engineering and security are as much a learned craft as a science; a dialectic between sand castles (if you will).

The first five minutes on any of my servers involve giving it a name, installing puppet and adding the server name to my central puppet config.

You seriously do this by hand for every server? That seems error prone and a huge waste of time when tools like puppet and chef exist.

Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.

I was in the same position - too many distinct environments for bash/Fabric, too little time to learn Chef/Puppet/CFEngine. Ansible [1] seems like a good compromise: you get the simplicity (runs over SSH) and host targeting of Fabric with the declarative nature and idempotency of the more complex tools. You can start with all-in-one "playbooks" [2], then split out tasks, handlers, Jinja2 templates, files, and variables [3].

As an aside, I think the default fail2ban config is too loose and quiet. Here's [4] an Ansible task file that configures it to be more aggressive and send notification emails.

[1] http://ansible.cc/

[2] https://gist.github.com/dbarlett/5079802

[3] https://github.com/fdavis/ansible-best-practices

[4] https://gist.github.com/dbarlett/5079715

I'm using fabric quite frequently, and am trying to understand what makes a configuration management tools a much better choice. I'm currently using fabric for anything from bootstrap a new environment from scratch, via restoring a snapshot from backups, to pushing code updates stored on git. Perhaps I'm being really daft, but it always evades me why something as simple as

    sudo("apt-get install -y <name your packages>")
needs to be replaced with

    - name: Install prerequisites for PPA management
      apt: pkg=$item state=present update_cache=yes
      - python-software-properties
      - software-properties-common
I do use a pretty homogeneous environment, which makes things simpler, but this is a deliberate choice to avoid complexity. If I know all my hosts are e.g. debian 6, then what makes ansible/chef/puppet so much better than fabric?

I'm not trying to be provocative or negative. I'm really trying to understand the supposedly big difference between what's labelled a deployment tool, and configuration management tools.

Deployment tools like fabric are imperative, and configuration management tools are declarative. With configuration management, you define the final state you want the server to be in, and it will do whatever is needed to get it into that state. Some or all of the parts might already be done, and it won't change the parts that are already correct (the declarative configuration is idempotent).

Deployment tools just execute whatever script you hand them, and so the scripts are either more brittle (server must be in a precise state beforehand or it doesn't work right), or require more effort to duplicate the work that the configuration management software does to only make the needed changes.

If your deployment scripts get complex, it's more difficult to see at a glance what the end configuration is supposed to be.

Thanks for the clarification. I think I get the theory a bit better now.

I'm still struggling with seeing some practical examples of those differences.

for example, `apt-get` is pretty much idempotent already, no? if I run `apt-get install -y <package>` x 1000 times in a loop it won't install it 1000 times...

Fabric does give you the building blocks for those kind of checks elsewhere, such as `exists`, `contains`, `append` (which is supposed to also be idempotent) etc... I've designed my deployment / bootstrap scripts with fabric to take this into account. It does add a little overhead, but nothing that makes me feel I need a better tool... Maybe my deployment base is still relatively small and homogeneous.

So it's true that I have to put in those checks myself, and that I don't have a very easy way to discover what state a server is at.

I'll try to take another stab at one of those tools. Maybe things will sink in when I actually use them. Thanks again for explaining.

When you install the package you do it as a on-off. When a configuration system does it then it will keep re-happening if anything changes.

Say you install "less", and I remove it. With Chef this will be fixed the next run - with fabric? You'll need to re-run your magic script.

That's the main difference. Automation like Chef/CFEngine/Slaughter will keep fixing breakages - as defined by differences in state from their rules - whereas you'll be playing a game of catchup.

If you're only doing one or two things, the value is a bit more vague. But consider even the simplest interaction: the config file for a service should look like X, and if it has to be changed, the service needs to be restarted afterward. Oh and there are 10 config files, but you only want to restart the service once if any of them changes, after they've all been changed. That's not hard, but it's already starting to look non-trivial.

And what if you want to have the same logic for several services? I guess you abstract it out to a function. But then it turns out one of those services doesn't have a restart command, and you have to do stop+start. And another service won't start if you use restart while it's not running, so you have to check if it's stopped and use start, otherwise use restart.

It's much more than just whether the code is declarative or imperative, or whether it's idempotent or not. An imperative tool can change your system from known initial state A to desired state B. A declarative system can change it from whatever initial state it's in to desired state B, even if you never considered it might be in that state.

When you want to make a configuration change to your servers, and they aren't already in a "known state", why would you think any tool could put them into a known state? When a computer has a virus, say, you don't think it's been put into a "known state" after the anti-virus program gets done with it; the virus may have done any number of things you might be unaware of, altered any number of data or configuration files in subtle ways that the tool doesn't look for, but implicitly relies upon.

Example: what if, say, one of the provisioning requirements is "make sure this gem is installed", but on one of the servers /etc/gemrc has "install: --no-rdoc --no-ri" in it? Now on one server, the docs are missing, while everywhere else they're available. That sort of thing.

If you're already running on an IaaS like EC2, I think there's a simpler, better way: rather than trying to get "unknown state" to "known state", why not use the simplest known state of all--unprovisioned? Write an imperative script that reinitializes a freshly-provisioned IaaS node to your known state, and then do a rolling reprovision, terminating old nodes and provisioning new ones.

An equivalent comparison[1]: would you feel safe running an automated script that would SSH into a production machine and "git pull" a checked-out repo sitting on it, from whatever state it happens to be sitting at, up to refs/heads/master, so as to deploy code from it? Could you guarantee that that repo hadn't been moved to some state where refs/heads/master isn't a fast-forward commit? Or would you rather do a fresh "git clone"?


[1] A contrived comparison, though; although the former option is unreliable, the latter is terribly inefficient, and they're both horrible for keeping a .git directory inside of a directory that might very well be web-accessible.

apt-get is most definitively not idempotent without a lot of care. It will upgrade packages if a new version is available, and such upgrades can have widespread implications.

Otherwise I agree with you - Puppet/ Chef etc. are overly big on ceremony for me. We "only" use about 100 VM's on about 20 physical servers at the moment, so it's a "simple" setup.

I think where people get the most value out of systems like Puppet and Chef is if a lot of their servers or VM's have simple enough requirements that they can largely use "off the shelf" community provided configurations. The more customisation you need, the more the leve of ceremony starts biting you. The more you can rely on ready-made scripts, the more those apps help you because of the size of their communities.

Regarding apt-get -- You're right.

If you use a single distribution and can get all your stuff packaged using the distribution's packaging system, you're OK.

It can be more difficult when you get to system configuration (editing files), but you can do it manually like you do with fabric if you're careful.

I'm not that familiar with Fabric, so maybe it can do some of the things I mention here out of the box without a lot of custom code. I'm only familiar with Capistrano and I associate them in my head, perhaps naively.

Your example is great to illustrate why configuration management is so powerful.

Most importantly, your example will upgrade the package to the latest version every time you run it (which is probably not what you intended, but maybe it is). I'm sure there is an idempotent command for each of apt, yum, ports, homebrew, etc, but I have better things to do with my time than figure that all out.

In puppet your command would be:

    package { 'foo': ensure => installed }  # or 'latest'
What if you also have RedHat servers? Now you need to figure out the right yum command to use to perform the equivalent. Or your developers Macs? Homebrew. Chef/Puppet are aware of the environment they run in and install appropriate packages using the appropriate package management script with minimal alteration to the underlying recipe.

For RedHat systems, it would be:

    package {'foo' : ensure => installed }
On a Mac, it would be:

    package {'foo' : ensure => installed }
On that Arch Linux machine that your devs are running, it would be:

    package {'foo' : ensure => installed }
Now, let's throw some dynamism into the mix...

What if you want to build an haproxy configuration file that pulls the list of hosts from an authoritative source? Do you "just" build it up from scratch every time and replace the file on every run of your script? With chef/puppet, you can pull that information in from your node classifier using the same query for dev, staging, and production and have the same recipe work for multiple environments. In some environments, the haproxy config has now changed, so I want to tell haproxy to reload. But in other environments, the config stayed the same, so I don't want it to reload. The file and service don't get touched if they didn't change.

What if a service has multiple files associated with it that when one or multiple of them change, you have to restart or reload the service? You don't want to restart each service every time one of the multiple files changes, so you end up having to build a queue to handle the service notifications.

Say I've got a few dozen users on my hosts and someone leaves. Which is easier - writing a purge script which knows how to purge the user from all environments that you support, or this...

    user { 'joe' : 
      ensure => absent
I'm certain that that you can do these things with fabric scripts, but by the time you have, you've either written your own configuration management tool (which is unlikely to be as robust as chef or puppet) or you have a hodgepodge of unmaintainable scripts.

that same line in puppet is "package { 'packagename': ensure => installed }". Or you can use 'latest' to keep it updated.

multiple packages can be done this way if you have a lot:

$prep_packages = ['package1', 'package2', 'blahblah']

package { $prep_packages: ensure => installed }

You can of course use extra arguments if you need them.

Eh, I didn't like Ansible. Every command is done over SSH, so you don't need an agent/ssh to box and run locally, but... scripts would take much longer to execute than in puppet.

We're currently using local runs of puppet, and it's easy to do dry-runs and see what fails before applying when testing. I'm not sure how Ansible gives feedback on that (I assume it must) but I find puppet pretty good about it.

>but... scripts would take much longer to execute than in puppet.

You can bootstrap 0mq and run them over that instead. That's much faster.

What do you usually do with these fail2ban notifications?

Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.

Speaking from experience as a part time admin, it is definitely worth it to automate the process, even with one or two servers. Any build system will repay in spades the first time you have to set up a new server or rebuild an old one, esp. if you're in a hurry after a server failure.

That said Linode offers other easier options (Linode specific of course). You can take the commands usually executed, and put them in a shell script in the language of your choice which is uploaded and executed on first run (they call this StackScripts - there are lots of examples and helper scripts on their site, but the scripts are really very basic). It's perhaps easier than learning a DSL as it's just codifying the same commands you would run via ssh by hand. You can also clone nodes so you could set one up in a known good state and clone that each time.

This does tie you to linode, so long term one of the other options available would be worth looking in to.

You don't have to re-learn puppet/chef/cfengine/bash script every time. You learn it once. It requires much less effort to "maintain" than manually updating more than 1 box.

I think being a part-time admin with other responsibilities is a perfect reason to learn something like Chef or Puppet. It'll end up saving you a lot of time that you can devote to other things.

I think the author meant for this to be a simple set of security guidelines for beginners. But yeah, any reasonably large prod stack should probably be deployed and setup with puppet and what not. It's DRY and less error prone than a human low on sleep.

Without Chef there would be no way for me to rollout a new server in our cluster. Investing time into Chef was one of the greatest things I ever did. Chef is the best documentation of our infrastructure. The second best tool I'm using is fpm[1] to make custom debian packages.

[1] https://github.com/jordansissel/fpm

While not directly related to this article, last I checked Puppet/Chef don't have particularly useful windows support. Is there an chef like solution for windows?

Puppet has great Windows support nowadays; not sure about Chef.

1. You should do "apt-get dist-upgrade" to get new kernel packages as well, otherwise you are stuck on an old kernel. (You might want that. I prefer updated kernel for the security, firefoxen, etc.). "apt-get upgrade" will only update existing packages - but the kernel updates require new packages to be installed.

2. If you're on ubuntu, root already has no password, and your initial setup user (whether it is called "deploy" or "kilroy") is in the sudoers file.

3. Other things I install in the "5 minutes with server" are: htop molly-guard screen git-core etckeeper

git-core because I prefer my etckeeper in git, but if you want it in bzr you don't need git-core. INSTALL AND CONFIGURE ETCKEEPER AS SOON AS YOU CAN, seriously. You need it. You'll thank me when you try to figure out when and how something in /etc got borked. (you need to edit /etc/etckeeper/etckeeper.conf if you use gif. You need to do "etckeeper init" and then "etckeeper commit" to establish the baseline)

molly-guard stops you from rebooting the wrong server

screen (or alternatively tmux) lets you keep your session open through ssh session disconnects (e.g. when moving from wifi to 3G, or between 3G towers that give you different external IP). The most useful way to use screen is "screen -xR" which also lets you share your session with someone else should you need to.

Indeed dist-upgrade should be used - it isn't just kernel updates either.

The server should also be rebooted. Applying kernel updates makes no good if you never apply them!

screen -x is equivalent to screen -xR afaict:)

I'd also add @reboot screen to crontab, which will recreate a session on startup - in my bashrc I have :

if [ "a$STY" == "a" ]; then screen -x fi

Other useful things include actually setting up backups (duplicity is a useful first step here), installing munin/nagios to monitor the new box.

Realistically if you are doing this more than once per blue moon, then you should be using something like puppet to do this automagically.

> screen -x is equivalent to screen -xR afaict:)

"screen -x" requires a session to already exist, whereas "screen -xR" will join one if it already exists, but will create one if it does not. At least it does in v4.00 which I use. If you have a "screen" call in your @reboot, you already have a session, so they will work the same.

I love etckeeper, but the use of a configuration management system (puppet/chef/salt...) tends to reduce its usefulness.

On the contrary - it is useful exactly to figure out when and what manual changes were made outside of the automated system.

I'm not sure I get your "on the contrary". Any configuration change on a managed configuration file is transient and bound to overwritten next time state is restored. This makes etckeeper absolutely less useful compared to a system where there is no way to restore state and an unfortunate "rm /etc/passwd" has little remedy. It does not make it "useless", but it's definitely less needed.

Possibly my misunderstanding, but I thought chef and puppet only push changes on the files that they are configured to control - There are tens of files in /etc/, some possibly modified by an "apt-get upgrade", then reverted by chef, some that chef/puppet do not try to modify.

etckeeper is a net that gives you an idea of how /etc looked on a given date or apt-run, not just how it was supposed to look (Which is what you get from the chef/puppet logs).

Question to chef/puppet users: do you restore the state continuously? or only when you've made a change?

Yes, it's true that chef/puppet do not overwrite your entire etc, just a subset of it, but presumably, you'll have your critical components backed by it (otherwise, there is not much point in using a configuration management system). So it is still useful on such a system, but not to the same extent of a system without configuration management.

> but presumably, you'll have your critical components backed by it

Often, the way you discover something is critical is after the first time it breaks. Backups may help, but proper version history is probably going to give better insight into what happened.

Definitely less important, but not useless, IMO.

Isn't the point of Chef/Puppet that you never do that, and if you do, they get reverted automatically?

It is the point that you "never do that". However, it's been my experience that e.g. when troubleshooting, people always make local modifications, and don't always remember to revert them / restore from configuration management.

I have no experience with chef/puppet, so I might be mistaken, but I'm under the impression that they only push changes when asked to - which means a local change may survive for several weeks before it is overwritten. But I could be wrong about that.

When run in client-server mode, Puppet will run every 30 minutes by default, overwriting local changes with the configuration on the server.

> No secure server is complete without a firewall.

Comments like those are why I normally point people to actual security expects (like, say, Schneier), and why I recommend that new admins should ignore as much as possible the practices chanted by the industry. A secure server does not need a firewall. A firewall can be used to secure a server against a specific threat, but that's it. The days of ping of death are behind us.

I would like to point out that following the article's guide and firewalling away ICMP, you can end up with a lot of trouble. (see http://serverfault.com/questions/84963/why-not-block-icmp). Some ICMP messages are not blocked by default by ufw, so I'm unsure how damaging ufw is when used like this.

At any rate, a Firewall is a block. A new fresh server install won't have ports that needs to be blocked. By putting up a firewall, there is nothing to be gained. Before the firewall, the ports are closed. After adding the firewall, the ports are closed. All that is gained is a hurdle the next time one wants to install something like a monitor tool (like Munin), or a new service.

It might be useful as a last line of defense against malware regarding outgoing traffic. I am normally against that kind of thing however (as focusing on the cause is better than the effect). At best, one can catch a spam malware, but any bot net, web server, ddos or other type of malware are untouched by the rules (port 80 and 443 is allowed). If the server has email sending configured so root message can be sent, then the spam malware can use that route and the firewall will just sit there.

So let's take a newly installed machine. What threats can be identified and what risks are we trying to mitigate with the help of this firewall (as specified by the article)? The only thing I can think of is either a Zero day TCP/IP stack vulnerability (not a realistic threat), or that the admin doesn't trust the other admins when they install new services. Yes, if an admin installs a new email server and enables relaying to the whole world against the explicit recommendation in bold font by the install wizard and the configuration file, a firewall can block that admins' actions. Then again, that same admin could just as well have disabled the firewall to "get the mail to work", so I'm not sure it's a viable defense against bad admins.

Schneier recommends use of a firewall: http://www.schneier.com/blog/archives/2004/12/safe_personal_...

You are correct that a firewall will not magically solve all your problems, but it does help to protect against programs that open ports you didn't know about.

Recommending against them doesn't make sense, and implying that they are only useful to prevent TCP/IP zero day vulnerabilities is silly (especially since the firewall likely wouldn't protect against that anyway).

Schneier talks about NAT and 2004 windows laptops (ie, with winxp). I actually asked Peter Gutmann during an IETF meeting around 2005, and he confirmed that nat had improved the situation around and win98/win2k/winxp windows machines and bot nets. If I recall right, the gist of it was that windows machines needed something, and while nat is wrong and bad, it "worked" in this aspect.

This is about as far from a server installed with ubuntu in 2012 that one can get. You are not going to find any such article by Schneier promoting default firewall installations. I suggest here to check out Secrets and Lies by Schneier, as it is rather clear that a firewall need to be configured against the specific threats one can identify. If you fail at identifying threats, the firewall is likely not be useful at all, or will simply work identical to NAT. At worst, it will give a sense of false security.

That's from 2004 and is about firewalls on desktops, not servers.

IP hasn't changed much since then, and I'm not sure server vs desktop is a relevant distinction here.

> The only thing I can think of is either a Zero day TCP/IP stack vulnerability (not a realistic threat), or that the admin doesn't trust the other admins when they install new services. Yes, if an admin installs a new email server and enables relaying to the whole world against the explicit recommendation in bold font by the install wizard and the configuration file, a firewall can block that admins' actions. Then again, that same admin could just as well have disabled the firewall to "get the mail to work", so I'm not sure it's a viable defense against bad admins.

I think it's less about defense against "bad admins" than it is about protecting against accidental bone-headedness. :-) I typically set up a restrictive firewall policy even when I have a clear list of the services I'm running and/or I am the only admin. This comes in handy every once in a while, in cases where...

* A service is expecting more ports to be open than are documented. (Happens not-infrequently with license servers.)

* I'm re-using an old image and there are undocumented services enabled by default.

* A user decides to run a network service in their own account without informing the admins.

In all those cases, am I likely to change the firewall to "make it work"? Sure. But having to actually make that change helps keep an audit trail, and helps keep the admins explicitly aware of the attack surface. It's similar to why it's a good idea to periodically run nmap against your own servers.

In those use cases I agree. If for any reason one do not trust the software installed to behave as expected, a firewall can be a nice net to cover any strange behavior. If the job require Proprietary software services with an unknown/untrusted behavior, or the re-use of old images with strange stuff in them, I would too consider running a firewall. In the case of users however, I tend to apply some good-faith practice and just monitor. If something happen, I can always apply a firewall to deal with the situation. However, I do understand if that is not possible in every work place.

Here nmap do shine, and periodically running nmap is a technique that should be taught in universities. Great way for students to both learn about computer systems, and about learning how to debug problems.

  The days of ping of death are behind us.
Packet of death. Hmm.


Ping of death is a ping that claims to have a 65536 byte payload: http://en.wikipedia.org/wiki/Ping_of_death

Flashback quite a few years. I was working in IT and my coworker asked me if my WinXP (IIRC) machine was up to date. I said "yes". Next thing I know, it crashed hard. Oops, my buddy just hit me with a ping of death.

Software firewalls can't prevent that problem.

Well, yes they could - if the machine targeted is behind a FW that is blocking that type of traffic.

If the FW itself has that specific NIC, it could be brought down with this attack - but you could prevent SIP traffic from hitting machines which are vulnerable.

ACL based firewalls are the foundation of real network security. I'm not saying every server needs to be running its own firewall. But these security measures need to be implemented at some level, whether it's at a network level or on an individual machine basis.

There are so many reasons to put a decent firewall in front of your shiny server(s), but I'll leave here only one (you could find the rest on the actual security experts blogs for sure): Half open TCP connections flood.

In general, people should keep a broader idea of security to include resiliancy.

Robust, redundant deployments should be a part of an overall security policy, securing yourself from outages and downtime.

Maybe this is considered a preinstallation step, but the very first thing to do is set the system time on the hardware, before you even boot the OS for the first time. Then the first step after booting is to confirm the time and reset it, if necessary. This is essential for accurate and usable logs, file times, version control timestamps, etc.

It's also a good idea to ensure that sshd has fresh keys that are unique to that machine. Hopefully, your images are installed without sshd keys, otherwise you'll have multiple servers with the exact same keys, which is considered bad practice. During initial configuration before deployment, you might want to remove the keys so that sshd will create fresh ones when it starts:

    rm -rf /etc/ssh/ssh_host_*

Looking through the responses here, I'm hopeful that someone will launch a "Sysacademy" variant for system administration training. There are tutorials scattered around the web, but (at least for those of us who don't know where to look), there doesn't seem to be one place that puts in under one umbrella.

Hopefully there will be more than one!

I had fun this past November with Snori74's Grep101 course.

I don't think it's open right now, but when it runs, it's a great stab at exactly what you're describing.

Initial blog: http://grep101.blogspot.com/

HN post: http://news.ycombinator.com/item?id=4726827

I have high hopes for Ops School (https://ops-school.readthedocs.org/en/latest/).

Looks really awesome. Thanks! :)

> vim /home/deploy/.ssh/authorized_keys [and] Add the contents of the id_rsa.pub on your local machine and any other public keys that you want to have access to this server to this file.

Cool trick: use ssh-copy-id <server> from the client machine.

From the man page:

    ssh-copy-id - install your public key in a remote machine's authorized_keys
It is much easier than editing the remote .ssh/authorized_keys file since copying and pasting the key is error prone due to extra new lines typically added by the terminal emulator.

My big thing is making sure 1) the machine comes back up cleanly after a reboot and 2) is current on all patches 3) is running as little as possible.

Also a big fan of externally verifying what ports are open, and making sure the system is in monitoring, backup, config management systems. Config management is kind of optional if you have a small number of servers which don't duplicate configurations, though, and there's often no need to back up the OS, but any data should be backed up automatically.

Don't forget:

  netstat -ntap | less
  ps aux | less
Also check to see what's enabled to run at boot time via whatever your flavor uses.

Check for unusual daemons, ssh running on other ports (yes, the provider pre-loaded systems with a back-door ssh without disclosing it to us).

This is especially important when you are taking over admin on a server you didn't setup yourself. Other folks have weird ideas on how to admin things. Like webmin for example...

I also like epylog for finding unexpected stuff in the logs.

to see what's enabled to run at boot time

what's good beyond this:

  cat /etc/rc.local

chkconfig is redhat specific. I believe debian-varients use update-rc.d.

Generically (for Linux), take a look at /etc/init.d , /etc/inittab, or /etc/rc?.d.

I went through the article and then read every single post on this thread. I am not a security expert so I won't even try to contribute except to say that I see a lot of people offering criticism without taking the extra step of explaining how they would go about hardening a fresh Linux install (or a pile-o-servers in a rack, whatever is applicable).

It'd sure be nice for those of us who are not security experts to read alternative approaches rather than, paraphrasing and not picking on anyone, "using a firewall is dumb" or "blocking ssh is pointless".

I like isolated ideas such as using a script to completely automate the provisioning of new boxes. Kind of a no-brainer if you ask me. The problem is that such recommendations are not followed by something like "Here's the script I use on Ubuntu 12.04 LTS".

How about it guys? Would you care to attempt to produce a canonical HN "How to harden your server" reference?

Maybe one of the security experts on HN can start a repository on Github to evolve a canonical script. I'm pretty much 100% Ubuntu 12.04 LTS, so it is my hope that this is one of the platforms that is addressed.

I did some looking around and this is what I found (I am in no position to evaluate the merits of any of these at anything beyond an intermediate level):













You are right, and I have some blame here of not including positive criticism in my comments. I do not have a script to offer, but I can go through the process I work by.

#1: A good password is a must. During installation, have a second computer generate a good password and either memorize it, or GPG encrypt it somewhere on the second computer. pwgen is decent in generating passwords.

2#, I fully agree with the article on automatic updates if its a personal computer. For others, one can have root mails sent if you are a fast and and read mail daily. This is how many people read about vulnerabilities before they reach the news.

#3: When installing large package like web services, I keep in mind of the long term prospect of each project. I ask the questions: Is there a deb package? Is it being maintained by a large group of independent developers? Is it mentioned in discussion at Serverfault? Are there any recent updates? What does the Wikipedia page have to say?

#4: read the man page, and check any section labeled security. Some man pages will say things like "we have this port open. Its completely insecure, and we expect either the local network to be safe or that you use a firewall". Through this just happened once for me, its still a good practice to check the man page with new services.

#5, avoid php themes/mods that require you to manually patch things. They won't be updated by Ubuntu, so things will either end with you uninstalling it or forgetting that it exist and thus get hacked. Sometimes ubuntu will just install over the mod, dealing with the issue for you.

Other than that, harderning depend on use case. A wiki/forum will need some form of anti-spam protection. A media center need access control or firewall to only allow local network. Unsecured protocols like nfs and nis need something like ipsec. A mail server needs authenticated smtp.

> have a second computer generate a good password

Why a second computer?

The Ubuntu installer does not ship with pwgen, and its good to have the password done right, right there while installing.

I'm throwing my hat in the ring. I took the author's original post and implemented it as an Ansible playbook in a little less than an hour.


Happy to answer questions about it. :)

I don't know Ansible. I learned about it through this thread. It sounds very interesting. I'll have to experiment with it as I am getting ready to launch a service that will start out with one server but will probably grow quickly to the five to ten range with a few devs involved.

Towards the end of your page you talk about keeping a manual log file. That's exactly the way I've handled this sort of thing for years. I usually call mine "project-log.txt". These files usually have several sections and it looks like this:

Header: Project name, start date and other relevant details

Date entry: What I did on a particular date

Working on now: Before I do anything I make this entry. After an interruption I usually have no clue what I was working on. By making a habit of writing a note to myself about what I am going to be working on immediately I can task-switch to that phone call or unwelcome question and mentally come back to what I was doing quickly.

To Do: Self explanatory.

Questions: If questions come up I log them here so I don't forget.

Ideas: Sometimes as you are working you say things like "hey I should do x". I log them here and go back to what I was doing.

References: Links, etc.

After launch: There are things that might not be critical at all and can be done after launch. I try to only note must-have's in "To Do".

This file lives at a self-named directory: "~/project-log". I do this because I can also save other relevant files there and keep the entire thing under source control.

  <Project Name>

  --------------------------------------------------------------------- 01MAR13
  <stuff I did on this date> format example:
  - Installed Ubuntu 12.04 LTS
  - Installed PHP
  - Installed MySQL
  - Installed Python
  --------------------------------------------------------------------- 02MAR13
  <stuff I did on this date>
  --------------------------------------------------------------------- 04MAR13
  <stuff I did on this date>

  --------------------------------------------------------------------- WORKING ON NOW
  - Configuring Virtual Hosting

  --------------------------------------------------------------------- TO DO
  - Configure vim for php development
  - Customize bash
  - Install imagemagick
  - Install php5-imagick

  --------------------------------------------------------------------- QUESTIONS
  - How to automate server bring-up and hardening?
  - Firewall or not?
  - Disable SSH?

  --------------------------------------------------------------------- IDEAS
  - Learn Ansible for automated deployment.

  --------------------------------------------------------------------- REFERENCES
  - Ubuntu 12.04 LTS docs
  - Ansible

There's nothing like good old plain text. :)

Nowadays I'm downright spoiled and use org-mode[1] to keep my systems journals. Org files are plain text as well, and org-mode takes care of setting up the tree by date. I can also add a journal entry from anywhere in Emacs with just a couple keystrokes, which makes it incredibly low-friction to use.

Like I said, the most important thing is to TAKE NOTES. Even pen and paper. It's one of Limoncelli's big points in Time Management for System Administrators.

Tooling doesn't really matter, the important part is being able to remember what the heck I did and when I did it. Invaluable for troubleshooting.

[1]: http://orgmode.org

The reason nobody shared their one true checklist for security is that it doesn't exist. Well secured systems get owned every day, and tons of bad ones never have trouble due to luck and lack of incentive.

Most extensive security procedures contain a lot of questionable advice and few will prevent human error which most compromise can be traced back to.

The only semi-universal list amounts to 1) use keys for ssh

2) block/turn off everything except your (web) service

3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)

4) Learn about web application security threats especially sql injection and consider them at design and implementation and have some kind of regular review (this can be very difficult to get right)

5) Avoid storing anything that makes you a particularly desirable target like bitcoins or secret defense plans as anything can be hacked with enough incentive.

Most everything else probably won't make too much of a difference to you.

As far as one true checklists go, I really like NSA Guide to securing RHEL.


and TLDR version: http://www.nsa.gov/ia/_files/factsheets/rhel5-pamphlet-i731....

It is starting to get a little bit dated (RHEL 5 is quite old), but general rules still apply and usually they explain their reasoning.

> 3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)

This isn't clear. Are you saying you think automating these updates is good or bad?

> 5) Avoid storing anything that makes you a particularly desirable

Well, in some cases your user's uid and pwd is the most valuable chunk-o-data a would be attacker wants.

> Most extensive security procedures contain a lot of questionable advice


> and few will prevent human error which most compromise can be traced back to.

I think this would be the power of having a canonical auto-provision script on Github that many can review and contribute to. The script could certainly take the form of sections that could be commented out as needed. In other words, a well documented and reviewed set of recommendations that someone could edit based on pier-reviewed information in the comments and then use to automatically configure a server. That, I think, could be of value.

This is great! I was thinking exactly along those lines, and started looking for puppet/chef scripts that are readily available.

It seems that using puppet has a pretty steep learning curve, at least from what I could gather skimming the docs.

Actually what I would really like to see is an interactive script, which then guides me through the process of hardening a fresh ubuntu server install, offering sane suggestions along the way.

I am pretty sure such scripts do exist, but having them maintained and regualarly updated somewhere would be quite neat.

I'm in the process of automating the configuration of my VPS using Ansible. This tutorial is an excellent introduction http://lextoumbourou.com/blog/posts/getting-started-with-ans...

Nice, I'm on the same road. I'm also looking at Ansible, it's just that I haven't got around to it yet.

ansible looks pretty good. thx for the link.

About a year ago I spent several days trying to get a full puppet system. It was a real headache to be honest and many of the modules have no or very vague docs. Coming back to my setup it's very hard to read or trace what my setup is or how to do one off deployments.

Today I have started with ansible and have already gotten a huge amount done and deployed. Very happy with it. It's all in a small git repo and its very readable and concise.

It hasn't been updated too much recently, but I've used Bastille for this in the past, and it's in the Ubuntu repos. http://bastille-linux.sourceforge.net/

Yeah, I forgot about bastille. On their news page they claim that development has been restarted recently. Lets hope that it gains enough momentum again.

Last I checked, Bastille was broken enough that you could fix it easily (I made a few small patches myself), if you knew what you were doing. Which kind of defeats a lot of the power of Bastille, but at least it was still kind of a handy automated checklist for seasoned sysadmins.

I've been using a product called scriptrock (http://scriptrock.com) recently that collects these kinds of best practices and ensures that they actually exist on nodes. I.e, the testing of configuration rather than the execution. I think this is really interesting for two reasons. 1, it's like monitoring for security on my nodes (it runs by default every 5 minutes), and 2, they could build some kind of open repo for people to contribute these best practices (I think they were asking for beta users a while back). Plus it allows me to configure my infrastructure in a kind of TDD way.

Add https://fedorahosted.org/scap-security-guide/, which I will at some point attempt to warp into an Ubuntu checklist, since I have always thought concordance on at least some security posture settings should be distro-neutral, with distro-specific implementations.

Writing a canonical server security document is difficult because it depends so much on what the server is doing (what kind of application is it running? how is it accessed?) and where you're starting from. (barebones install? something pre-rolled by your VPS service?)

Sounds as though we need a decision tree app to write the scripts for us. It asks questions such as, "Do you have your own IP block?" and explains the question thoroughly for those who don't understand it, explains the answers offered (for questions such as, "Which of the following types of logins would you like to allow (check all that apply)? Password, public key, ...").

It would offer you explanations, deeper explanations, defaults, recommendations for when you would override the defaults, etc. for each item. It would be smart enough to prevent incompatible or contradictory settings.

Once this process was complete, it would generate the needed puppet / ansible / whatever script, which the admin would run and store for future reference and use. When installing another machine, the original script could be read in by the decision tree and the new script could be generated by modifying the original rather than starting from scratch.

I would start with Ubuntu 12.04 LTS on Linode and see what develops from there.

Shocking that no one has mentioned the grsecurity project.

The problem I had with the article was that it said what user accounts he set up, how their security was based, etc. Why!? Don't give away that info in a blog post! Even if it is something a hacker could easily find out if they wanted, by doing that, you paint a target on your back.

If you are walking past a dark alley in a big city, you don't say, "I am a blue-belt in karate, and I have a wallet with $300 in it." They have a gun, and they'll take that wallet, thank you.

I totally appreciate you putting all those links together, but I would hope that others remember that loose lips sink ships when it comes to security. And from a hacker's perspective, what's the fun in social engineering if someone just blabs it all in a post?

If your system can't survive an attack because the attacker learned from a blog post that you are using tool X then perhaps tool X is the problem and not the blog post.

Real would-be intruders are not dummies. They have a suite of tests they can run to "x-ray" your system to the extent it is possible and discover vulnerabilities.

To some degree it's like encryption code. The safest code has to be open source.

> The safest code has to be open source.

This is a commonly held belief that is not true. I've been doing open source development quite a bit over the last several years and have seen plenty of insecure open source projects that were even less secure than I would see in the private repository of every place I worked. Here's why:

* Open source code gets more eyes on it, when it is well-used. But there are loads and loads of projects that are hardly looked at, and they have a greater chance to be used before they are thoroughly vetted.

* Those projects were most likely thrown up there by a developer like me who just hacked something up quickly to solve a problem. Once done with the problem, the code stays up there and I just let it atrophy. That consists of 95% of my projects, at least.

You misunderstood. I was referring to encryption code. These are open source projects that have to survive the scrutiny of experts. For example, I guarantee you that if I attempted to contribute to an open source encryption code-base without a massive amount of work my submissions would be rejected. This is not my area of expertise. Could I do it? Sure. Nothing is impossible. However, I would have to devote a significant amount of time to fully understanding the state of the art before doing so.

Your first 5 seconds should probably be


Add "free" and "top" there too.

I've had df -h write out to the motd before for clients who insisted on tiny slices of disk, no automated monitoring and no manual checking of disk space even after it chewed itself up.

I'm not sure it helped, but it made me feel better.

It saddens me each time I see a security best practice guide that suggest turning off ssh access for root. Its a very useful feature, and the security industry should focus on the security problems rather than removing features without thinking about the actually benefits of doing so.

Sysadmins with root access should be able to handle a random 8 character or longer password, and that number is large enough for a secure public accessible ssh. If your not a sysadmin or unable to remember a random password, try go with a passphrase like "correct horse battery staple". If you have too many machines and thus can't remember passwords, then use a master password locally with a bit larger password and store certs there.

To do some math to illustrate the security of a random 8 character long password, someone would need to fill a 100/100 line for several hundred years to get through them all. By that time you should have noticed the constantly full connection, and be happy you haven't died of old age yet even after your 200th birthday.

However, 8 character passwords are not the suggested length by security experts. They suggest using a 10 character long password, as that is also secure in the case that your password hash somehow got lost.

For the several servers that I have, I have been more worried about the logs from failed attempts than I have been of anyone guessing the password. Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it. It also decrease the noise level in the server room.

You fail to understand that true security is not about passwords but control.

Disabling remote root login,

1. cheaply reduces attack surface

2. necessitates assigning administration rights to specific persons and roles.

3. greatly increases command visibility and value of audit trails.

4. combines extremely well with disabled terminal shell over SSH

5. can reduce (unnecessary) system resource usage.


There are very good reasons why this practice is standard. Unless you are specifically creating a test or honeypot box of some kind, you would be foolhardy to ignore it.

You fail to understand that true security is not about passwords or control; true secrity is about risk mitigation.

Control is an ignorant tyrants last redoubt. Security is only about: identifying risks and dealing with those risks. All other messures are worthless as actual security objectives.

I am not arguing with you in priciple, only in particular. All of your points are well taken, to reduce risk, but not to control.

As the other comment say, Security is about risk mitigation. Risk in turn is about identifying threats, and attack surface is avenues where that threat might cause damage.

The problem I have with the articles claim is that limiting roots access in ssh is not improved security. It does not acknowledge the threat or attack avenue at all, but rather tries a blind approach. The threat is script kiddies. The attack avenue is bad passwords combined with password authenticated use of ssh by root. One can then either fix A) the password, or B), the access root has in using ssh. I pick option A and thus have have good passwords that secure even against internal users, rather than having bad passwords and limiting roots use of ssh. Having good passwords, and disable ssh for root is redundant and a expensive limitation for no security benefit. Having bad passwords and disabled ssh is bad security practice, as bad passwords should by policy be impossible.

> 2. necessitates assigning administration rights to specific persons and roles.

If you need to have different assigned administration roles for a single installation, and guarantee a correct audit trail, you should look into physical tokens or at least a kerberos+ldap setup. "root" user should then not exist as something that anyone should use. however, if you do not care about guaranteed audit trail (and to be honest here, the common use case does not require kerbros+ldap setup), you can add something like environment variables to indicate which user is logged, and ask administrators to document changes to the system.

> 5. can reduce (unnecessary) system resource usage.

This goal can be mitigated in several way. Thus in a honest discussion, this should be the target of it and not the pretense of increased security.

.. and it saddens me each time I read a comment like this. It's best practice to disable remote root login. A whole industry supports this.

And, yet, the best practice seems to require an user account with full sudo priviledges.

If you know the difference between that and root, please explain it, because I can't see any.

Well root is the default full access account. It is a bit easier to root@serveraddress and brute force in that way than needing to guess which account in the system has sudo access.

It also becomes easier to detect when someone is attempting to break in when you can see logs of common user names in a row fail to log in.

I don't do much admin outside my hobby boxes. One guess would be, excluding the ability to enter a root shell, each sudo command is logged so you know who ran it. If the user is root, then it's harder to know what they did while they were root.

Disabling root also means you now have to guess the username and the password instead of just the password.

If someone logs in directly as root, how do you track root access? Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root?

Discouraging root usage is common in desktop and laptop environments, and for developers on production servers. For admins on servers, root use is basically required for all commands, and sudo would not improve the security at all.

Commonly, I track access through environment variables (configuration on the ssh client), and documentation that each administrator do after finishing a task. As one should not recruit administrators that do rm -rf /, tracking (blame) is of an lesser importance and documentation a much higher one. If more was required, I would go with kerberos+ldap approach without anyone using local root user accounts at all.

I am the admin. I use the root account as little as possible. I do employ admins that don't rm -rf / by employing admins who don't hack on live as root. Knowing who does what is paramount for resolving problems particularly in admins' work practices.

If someone logs in directly as root, how do you track root access?

By their SSH-key and IP address (both of which are logged to a remote syslog-server).

Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root?

That is a false premise. I'd never encourage anyone to use "sudo" because it's a source of errors; people get the escaping and context wrong all the time, doing things that they didn't intend, and wasting precious productivity on getting the incantation just right.

It also doesn't add any meaningful security, it's a red herring.

If you need a dependable audit-trail then sudo is worth zilch. In that case you're looking at SELinux, auditd and friends.

If you don't need that trail (which is most people) then sudo is a ball on a chain and a false sense of security. Local privilege escalation exploits are dime a dozen, any shell-account can be upgraded to root by a dedicated attacker anyway (sometimes via sudo itself, as seen in in the recent sudo exploit...).

Oh, and let's not forget: If you enable a user to do anything meaningful with sudo then that normally includes giving him a way to write arbitrary files as root...

Logging the ssh key used sounds interesting - not something I've tried before.

I think you're doing something wrong with sudo. I'm not talking about "reentering your password" in the Ubuntu or Mac sense. I'm talking about giving access to specific limited commands.

And what distro are you running where local privilege escalation is common?

I'm talking about giving access to specific limited commands.

By definition the set is either so limited as to be essentially useless, or it opens (usually multiple) straightforward paths to a root-shell. Pretty much all sudoers-files I have seen fall in the latter category ("but he can only write the apache config-file!").

And what distro are you running where local privilege escalation is common?


"only write the apache config-file" Ha!

Local privilege: yeah, that's a lot but over a long time span and none should be current on decently maintained box. I've never even seen anyone attempt a 0 day on my boxen.

> Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it.

Or change the ssh port.

Another "security through obscurity" pointless trick. If I'm attacking your server and it's not running SSH on port 22, it's a very short matter of time until I've found it anyway.

I don't understand this objection.

If you've got a physical safe that you keep money or valuables in, do you put it out at the kerbside so people can have a go at cracking it, or do you hide it inside in an obscured location to make it more difficult to have a go?

Surely it's the same with services on your servers, why needlessly advertise their presence. It's a tiny veneer of security but it is still more secure surely.

Sure. It's only meant to stop automated script-kiddie attacks that target all open SSH servers on the internet. Less logs, less disk usage, ...

Wouldn't the script-kiddie through those hosts into a queue to run a more advanced port scan on? I mean, I see your point, but it seems highly useless compared to, say, simply disabling the password based authentication.

Why would they bother? Script-kiddie is just playing the numbers, any machine with a slightly non-default configuration is not worth the effort.

Disabling password authentication is better for "real" security, but doesn't stop those dictionary attacks spamming your logfile. (I still feel there should be better solutions to that problem than changing the port though)

> I still feel there should be better solutions to that problem than changing the port

Well, there is port knocking. http://www.thoughtcrime.org/software/knockknock/

I guess my response to all of these is... using key based auth is so much more secure than changing the port that changing the port seems to me, completely, pointless. Especially considering that:

- password auth = me having to type my password everytime, likely to have a less secure password

- key based auth = me never having to type it in, with a key length so long that it's completely inconceivable it'll be cracked or collided or brute forced

and then when you change the port, the only "advantage" I get is that I now have to remember the port because they're already not getting in by virtue of key-based auth.

> using key based auth is so much more secure than changing the port

Is it really? Once you get to a sensible password length (say, >14 characters) haven't you effectively made brute force and dictionary attacks impossible anyway?

> password auth = me having to type my password everytime, likely to have a less secure password

If you use a password often enough, I find the time difference between an 8 character and a 16 character password immeasurably small. It's all muscle memory anyway.

>key based auth = me never having to type it in

Perhaps I'm clinging to a false sense of security, but I find the idea of anyone with access to my local machine being able to access my server with 0 additional credentials really disconcerting.

Perhaps in the theoretical world if they can access your machine it's game over anyway because they can install a keylogger. In the real world, I think there's a fairly large population of people who might access your server who wouldn't/couldn't install a keylogger to steal your password.

Your private key should have a strong passphrase. I agree that using a passphrase-less key is bad practice. As for keyloggers, once they obtain your password, they have succeeded. But after obtaining a key's passphrase (which you normally only type in once a local session if using ssh-agent), they still need the key, which may or may not be marginally harder. It's still another step to overcome. So, even if it can be shown to be only a narrow improvement over passwords under certain circumstances, it's a big win in convenience and even security under normal circumstances if done properly. Give it a try, finetune your ~/.ssh/config and you might end up loving it.

> but I find the idea of anyone with access to my local machine being able to access my server with 0 additional credentials really disconcerting.

You could hook whatever locks your screen to also clear your ssh-agent then. This is probably something like alias xlock=$(ssh-agent -k && xlock) in linux.

You do lock your screen, right?

The answer is really easy: You should do both.

Disable password auth for root, and change the SSH port.

The advantage of changing the port is to reduce the spam in your logs. Checking your server logs regularly really does increase your real-world security.

Imagine how long it would take to scan every port between 1 and 65535 on millions of machines.

Versus how long it would take to just try port 22 on millions of machines.

It wouldn't stop a determined hacker for more than a few minutes, but it will reduce the number of automated attacks on ssh essentially to zero.

this. Everytime I check my logs there's tons of failed tries to port 22 yet none to the port ssh actually runs on. Which keeps the log with failed logins clean.

They don't seem to. Plus, if they hit more than 5 ports their ip gets blocked. Plus it keeps crap out of my logs...

“correct horse battery staple” — the new hacker’s choice of password.

Nice writeup. Although I prefer to automatically configure using Chef, I didn't know of fail2ban and logwatch -- I will definitely look into those.

Also, since this post is for beginners, you should mention about restarting the sshd over the lish tty and not over the ssh ptty.

1. I don't think it's a good idea to use the same account by multiple people.

2. And you're not using a configuration management tool (like SaltStack, which is also a remote execution engine) this will give you: - a central point to manage all your server - predictable configuration on all servers with the same role - a configuration documentation place (and even history if you git the confs) - will make managing multiple users a breeze

3. Use VPN and private services on private IP.

So I'll agree with many of the commentators that several of the practices suggested aren't 'ideal.' However, they are easy and possibly better than having no 'practice' at all.

Just as an example, the shared user account with unique SSH keys per user. Sure, it's obnoxious in some respects, but many of the criticisms I'm reading in the comments like "but they could reinstate their access with a cron job that re-adds their key when they leave" and such are silly - presumably those who are using the shared account are developers/sysadmins with sudo privileges. Regardless of whether they have a shared account they have privileges to do whatever they feel like to the systems in question. Hence I'd argue it's a fairly reasonable solution for the situation when you don't have the time/resources to configure something more complex and you have to trust all parties anyway.

I think there are two larger takeaways:

First - Managing multiple users across many servers and dev systems is not easy enough, particularly for smaller organizations, and only gets worse when you try to get more granular about who can do what.

Second - umm... no idea anymore. Forgot what was second. Automation is good?

I find it very bad practice to blindly pass -f to commands.

  chown deploy:deploy /home/deploy -Rf

"-Rf" not found... :P

flag arguments should go before any other arguments to be compatible with the most Unix systems.

Thanks for the catch! Fixed

I'd propose using OSSEC over logwatch & fail2ban. Ossec seems to be a bit of an obscure tool, but a thoroughly functional one at that. Logwatch gives a bit too much info at once to interpret properly, while OSSEC will only alert you when something is actually up.

Ossec provides (among some other things):

* Log file monitoring, with severity levels, occurence counting, time-based rules and auto-response. This means (for instance) you get to watch your auth.log for failed login attempts and after 10 failures fire a script to ban him, alert sysop by email or have hubot alert your sysops. Or whatever floats your boat.

* File integrity monitoring. Make sure noone's been mucking around with your files. It has real-time support (through inotify), but if you don't want use that, make sure you store the database it keeps off-server for forensics if need be. Pro-tip: FIM and auto-updates are a tad unnerving.

* It can watch command output. You can use that to make sure the `netstat -tnap` output doesn't change, for instance.

* For larger/more compliant instances, it has a client<->server setup available.

I have heard good words about OSSEC but never tried it because of it's perception being:

- heavyweight;

- not actively developed.

Are my perceptions right? If no, how would you recommend to start using it? Any good tutorials or other pointers?

Last version dates from 2012-11-19, not the fastest of releases - but seems ok. Trend Micro is involved (and offers commercial support).

It doesn't feel heavyweight to me. It does start a bunch of daemons for all of its processing and it has the client->server bit built right in. That may make it feel heavyweight, I guess. But, you don't have to use the client->server stuff if you don't want to. It'll still do all of its magic for you.

Ossec will do a lot out of the box. So, I suggest installing it with the default ruleset and the active-response stuff turned off (the installer will ask you). Then dive into the rules and ossec.conf (knowledge of regex is required).

Documentation @ http://www.ossec.net/doc/index.html

Ossec is actively developed. They just had a new release a few months ago. Realistically it doesn't get updated as much as it used to, but it still is actively developed.

I think more than anything it is just kind of at a near end-state where there isn't a whole lot left to ADD other than some signature type updates, or nice-to-have features.

It is a program that I HIGHLY recommend having and it is very simple to get up and running, even if you don't have packages of it for your OS version. Also you won't pass PCI-1 without a IDS like this.

I wouldn't bother with fail2ban considering password based SSH logins are disabled (which is good).

Since the author is using ufw to control iptables, better to just use "ufw limit" rules for SSH port 22 to slow down the rate of any automated SSH bots trying to give your server a workout.

Indeed. I have always been a bit worried about such approaches, since they parse log files and attackers have some control over what is written to log files (user names and host names).

To prevent having to log into root via remote console, set up a second backup account with an ssh key and sudo access, and then put that key somewhere safe.

The chances of both that and the deploy user getting corrupted at the same time is unlikely.

That's a good alternative. At least for Linode, remote console is fairly easy, but some don't offer this feature.

I was wondering about this the other day, is

    PasswordAuthentication no
really that necessary if a strong password is used?

Honestly, being a private key screw-up away from never being able to log in again scares me a little.

Encrypt the key (ssh-keygen offers you to do so) and backup it to bazillion clouds.

That's never the case. Any decent colo facility has some way for you to reboot in single user mode and remote in to that, IP KVM is one such tool.

I dunno about this guy but apt-get update/apt-get upgrade takes upwards of 15 minutes on new installs (as does yum upgrade on rhel-compatible systems). Kinda eats up into my first 5 minutes on a server.

Look into apt-cacher-ng. I have a slow internet connection at home and it made updates go a lot faster for multiple machines. Newer versions (in wheezy, not squeeze) even work with yum.

The guide recommends blocking SSH access to anything other than your own IP address. The problem is that my IP number sometimes changes at which point I end up locked out totally.

So to get around this you either have to allow SSH from anywhere or you have to use some remote KVM system. Most of the remote KVM systems seem to be based on Java applets which is not really something you want to enable on your system.

So what is the best way to get around this? Just open up SSH and be diligent about your SSH security? implement port knocking?

I do this:

Set up a small AWS server to use as a Web proxy and SSH proxy.

Connect from you home with a remote forward port (for example 2222 to 22) additional to any other local forward ports. Put this tunnel in the startup scripts.

When you are away, connect to your AWS server, then connect locally to port 2222.

Now you've got SSH access to your home machine from anywhere.

I've never tried this, but could you enable access from a given hostname rather than IP address, and then use a dynamic DNS system like http://www.noip.com/ to automatically update a hostname when your IP address changes? Two downsides I can see are that a) you would be vulnerable to DNS poisoning attack on the server side, but that would be the equivalent to just opening it up to all IPs, and b) dynamic DNS systems can take a short while to refresh the DNS record and/or it can get cached by your servers up-stream DNS servers. Some tinkering with your server to make it always use the authoritive DNS servers for your dynamic DNS hostname would help with that.

I'm just thinking out loud, I've never tried any of that!

This may not work for you, but I have a similar situation when I'm at home or I'm travelling. To solve it I set up a VPN at the office (we use Meraki hardware, so it was literally just a click) and connect to this first - then connect to the server.

I did consider that approach, but the closest thing I have to an office is my home (with a dynamic IP) anyway!

I thought of setting up a VPN on a server, but all that would really do is move the problem from securing SSH to securing VPN.

Our provider allows us to modify firewall rules through their control panel. Perfect for when we have a dev working from home or a hotel room for a few days.

I like the KISS aspect of this, though for accountability purposes, I prefer each user to have their own account.

One correction for you: the sshd_config lines should have no "=" symbol.

Good catch! Fixed

I use "adduser" instead of "useradd". Its more convinient as it does the remainings steps automatically for your. It comes with ubuntu by default. Any downsides?

For some reason it's not working in last Arch's build. Manual "useradd" is such a pain in the ass...


I just use this bash script when setting up a new VPS. It pretty much takes care of it all for you, plus it sets up a ngnix/mysql stack already optimised for law RAM machines.

Also sets up 3proxy which is handy for viewing Hulu/Netflix :)

Imagine you are a fairly normal windows user or even sysadmin. Imagine you are considering Linux to replace some task that a windows server performs.

Now imagine the conclusion after reading this thread. As some one who can just about get something useful done in Linux, this thread makes me want to never use it again, it just looks too scary. Loads of disagreements which seems to have lots of dire consequences. OK, great discussion for deep geekery, but scary as hell for normals.

Now, when ever I see those annoying posts from smug Linux users who jump in every time a Win or Mac user highlights a problem, I now have this discussion to point to as to why Mac and Win users wont generally go near Linux.

Sorry chaps (and chapesses), I am on your side, Linux is a great thing, but this has to be the worst advert for Linux ever.

There is just one action on the post that will have dire consequences if ignored: disabling the ssh password authentication.

The rest are decisions that you don't need to rush into, and things you should learn when they make sense. Be advised that "the proper way to configure a server" is an elusive beast, you'll be always improving it, it's never done.

This IS great advert for Linux. It's just that newbies are not this ad's target audience.

Sure, you can put Ubuntu in your mom's laptop and she'll be up and running in a breeze, but hey, she's won't be reading Hacker News.

While I agree these practices seem pretty safe/standard, it's still enough manual work that many people will just skip half of it out of laziness for small projects.

Does anyone have good reusable Puppet (or bash scripts) published that they use on all new servers?

A personal script, focuses more on getting a server ready for my preferences than it does on security: http://pastie.org/pastes/6376503/text

Here's something I wrote very quickly (bash script): https://github.com/vahek/vSetup It does most of the stuff mentioned in the post, however doesn't setup automatic updates or Logwatch.

(to both corin and vahe): That kind of shell scripting is the horrorshow that motivated configuration management tools* in the first place. Shell scripts require a lot of added complexity to manage multiple heterogeneous servers, or to be idempotent.

* Puppet, Chef, Ansible, Salt, CFEngine, etc.

I agree. However, there is no learning curve to get started using this script and was put together in a few minutes.

Why install fail2ban and logwatch instead of csf?


Looks like an interesting piece of software nonetheless.

Does it still have all of those root exploits?

I wasn't aware of any. If a malicious user has root access though, you are fucked anyways. Please correct me if I am wrong.

A few gentle suggestions:

> The days of passwords are over. You’ll enhance security and ease of use in one fell swoop by ditching those passwords and employing public key authentication for your user accounts.

ssh keys are better than passwords only because they contain (and require) more information. On the other hand, if your dev's machine is lost or stolen or compromised, so is your ssh key. This is especially a problem in environments with a shared account with full access, as you have. So, it's probably a good idea to make sure you're using a passphrase with your ssh key (during ssh-keygen), unless you need a passwordless login for a shell script or other automated remote system.

> passwd deploy: Set a complex password - you can either store it somewhere secure or make it something memorable to the team. This is the password you'll use to sudo.

Not necessarily. Anybody with access to the "deploy" account can use "passwd" to change its password to anything they like. (Edit: I'm wrong on this! passwd does require your current password; I've just gotten used to doing it for other accounts via sudo, which doesn't.) Changing the passwd on your own account doesn't require sudo. For this reason, I think it's better to simply give deploy nopasswd access to everything, and then delete and lock deploy's password to prevent it from being used at all (passwd -d -l deploy). You'll have effectively the same amount of security, but this way nobody will need to remember or retrieve a complex password, and you'll prevent, say, some accident in /etc/ssh/sshd_config from making deploy remotely accessible via a password.

You can do something better than this though, but it takes a little effort. Deployment is often the same steps over and over again (an rsync or an occasional apachectl graceful in my case). You can give the deploy user nopasswd access to only a shell script that's writable only by root; this way, deploy can still do 90% or more of their job without ever being given system administrator rights. You do have to be a little careful writing shell scripts though -- $* and "$@" still trip me up once in a while.

> Logwatch is a daemon that monitors your logs and emails them to you. This is useful for tracking and detecting intrusion.

This seems of dubious security value to me -- probably better as a generic sysadmin tool, so that you get annoyed by noisy logs and seek out and fix minor problems instead of ignoring them. Thing is, if someone does get access to your server, you pretty much can't trust it at all anymore. With services like Linode, you're really better off just launching a clean new instance, re-running your setup script (if you have one), and moving your data over.

I had to deal with the occasional intrusion in some pretty icky servers at an ISP once upon a time. We used rkhunter for a while, but I learned pretty quick that successful attacks against Linux servers are plenty enough sophisticated to alter all the basic tools that you would use to detect and remove the rootkit.

There is one caveat: I've been playing around with the idea of setting up rsyslogd to route syslog messages to mysql, and then using mysql replication to have an up-to-the-second offsite copy. I'd combine that with Snoopy (https://github.com/a2o/snoopy) or something similar. The point isn't to try to clean up an intrusion, it's to see how the intrusion happened so that I could close that hole. I haven't gotten around to setting this up yet, so I can't say anything terribly smart about it.

Finally: if you're going to have a problem with unauthorized access to your Linux or BSD server, it's probably going to be via one of its services, not via brute force ssh or anything similar. So, if you're concerned about this kind of stuff (and if you're being paid to be a sysadmin, you have to be), then you need to spend most of your attention making sure that your various services are set up correctly (apache/php/mysql/postfix/dovecot/spamassassin/etc. in my case).

I love ssh keys. However:

The big problem with ssh keys is not being able to enforce ssh key passphrases on users. From the server perspective, you have no idea if the user has set up a passphrase. There are security standards which mandate certain kinds of passwords (complexity) and are silent on asymmetric keys, so you couldn't use keys in those environments.

The old solution was to do some post-login hack to require a password as well (e.g. to su), or do a VPN (which could have multiple forms of auth) and then ssh with keys after that, but the newest ssh (and I believe commercial ssh for a long time) now supports requiring multiple authentications per login, so you can do ssh key plus passphrase.

There are also DLP/etc. reasons why ssh can be problematic in some environments (i.e. where you're required to log/analyze actions taken by users, particularly admin users). The solution there is to use a bastion host and ssh in and then ssh out, with the user account locked down to log. SSH Communications (the commercial ssh people) have an interesting ssh MITM box which essentially does what all the SSL x509 MITM CA things do.

SSH keys also never expire and are easy to copy/steal. Key-based auth is officially discouraged at the day job (for laptop-to-desktop type things) for exactly this reason.

Expiring an ssh-key is easy, just remove the public key from authorized_keys. Dealing with theft is also straightforward: just add a passphrase to the key -- the time it takes to crack your pass phrase should be more than the time it takes for you to run ssh-keygen && ssh-copy-id.

The issue is lack of key management for ssh keys in a lot of environments, especially for accounts not always used. You can wrap protection around it, but you ay be better off just using a different authentication system (Kerberos? Something else?) in large environments.

Expiring an ssh-key on YOUR machine is "easy". Ensuring it's really gone on every, single UNIX-like box anywhere in your company is less easy. Oh, and what happens when you restore a home directory or entire machine from tape? Are you absolutely sure you remembered to delete every SSH key you wanted to "expire" at some point?

SSH keys do not satisfy a "fail closed" security model: they're there unless you explicitly remove them (and keep them removed). Certificates, tickets, and other expiring credentials eventually go away and lock users out unless they're explicitly renewed.

Thank you

People often consider ssh keys to be a panacea and to think of passwords as outdated

We throw an OTP (One time Password) into the mix using Yubikeys:


Yep, I've been meaning to get one of these. I really dislike being unable to enforce a password on your private keys.

I've also been using Duo Security as my new 2FA solution and I like it a lot (it also has support for using Yubikeys to provide your OTP.)

You require both key-based AND password auth with OTP? Or is the OTP at a higher layer, e.g. VPN.

Thanks for the link. :)

> There are security standards which mandate certain kinds of passwords (complexity) and are silent on asymmetric keys, so you couldn't use keys in those environments.

Every single auditor I've worked with has given a pass when they saw we were using keys instead of passwords. Do you also fail audits because your RSA token is only 10-12 digits?

You could, I suppose, demand users upload their private key (shudder) to the organization so you can check if it's encrypted; this has its own flaws (primarily that you've just taught your users to be giving with their private key), and you still can't enforce passphrase strength well.

Yes -- FISMA. There are both requirements that the implementations be FIPS 140 (although software is ok) and other requirements. I don't think I've ever seen anyone use a securid without a password, so the password complexity requirement is satisfied there. (It wasn't that an auditor failed it, but it was said we'd need to fix this to avoid any issues, so enh)

I don't have a particularly high opinion of most security audit standards, though.

Thank you for the solid suggestions! +1 on protecting your ssh key with a passphrase.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact