1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one.
2) Everything else listed in this document should be configured by a configuration management system.
3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure.
4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does.
Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.
This is an excellent request. Reading through all the comments here, it seems like a lot of people are feeling frustrated with the variety of information available and no clear way to discern what is "good".
I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-)
That site is useless, sample pages I looked contained brief overview paragraphs and no real content. Many contained "todo" items.
So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.
Thanks! I really appreciate it. All to often we are quick to criticize someone for missing information without any willingness to back it up with better, correct information. The jokes goes that the best way to get an answer is to publish the wrong one. =)
This depends entirely upon your level of masochism and the kind of language youlike scripting in, as literally anything will do. Even shell scripts. Even Makefiles. Whatever you're most comfortable with, just start writing out configs. (you will eventually come to hate your job if you allow non-programmers to script/program in your CM, but blah blah keep shipping blah blah) Break it all out intoa wide hirearchy so you can reuse some bits in other bits.
Aaaaand you're done. Of course I haven't tested these configs and something will probably break (most likely my crappy code) but you'll get the idea from looking at the examples.
If Bastille was still in working order, I'd recommend it as a very good starting point for locking down a configuration. Actual configuration deployment setups I don't put much stock in because I'm not managing enough machines for it to be worthwhile. Would love to hear from someone who manages a large farm/cluster/VM hosting on a very clean and straightforward way to manage configurations. Last I looked, most setups were custom, or some hobbled together packages that didn't "hang" together very well.
But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool.
Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.
Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.
It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet.
Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.
You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)
The problem is that most of the recommendations are bad.
Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior).
The rest is just bad advice.
Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?).
User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server.
Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers.
Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses.
You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.
I agree on a lot of things, except LDAP. Of course, it sounds like the ideal way, but in practice, there's a matter of scale to take into account.
Frankly, you need a lot of developers and servers before investing the time to setup a ldap deployment, integrating it with logins, and spending the inevitable hours debugging why nobody can access anything anymore, becomes more worth it than "just rsync/pssh into all servers and edit /etc/passwd".
Actually, if I had to do it, I'd use chef to automate creation (and destruction) of user accounts over an LDAP any day. Chef can be a pain to learn and use, but any sort of LDAP is even worse.
I would probably stand up an LDAP server after I had 4 or 5 user accounts to deal with, or, more importantly, more than, say, 3 servers to deal with.
I think there's a lot of fear and hate surrounding LDAP, but that's mostly for historical reasons. LDAP has gotten a lot easier to set up. Even in 2009 a colleague and I set one up (using openldap) and had other machines authenticating off of it in an afternoon. It's gotten even easier than that since then.
And hell, you should be using Chef to set up your LDAP master and slave. So once you have the config correct once, you can bring up another machine without trouble when needed.
My friend and I are rebuilding an old, text-based browser RPG, occasionally with another friend helps out with art and game content.
We're on LDAP, which we use to SSH into our ec2 servers, and which we use for authentication when we deploy using `git push production master` to a GlusterFS cluster. We're running our LDAP, application, and file servers on Gentoo. We can easily add new accounts, and we have it set up with group permissions (so the friend can deploy game content to test but not prod, for example).
I refuse to believe that LDAP is "too complicated" or "has to scale before it's useful", when a couple of guys can, in their free time, set it up for themselves. It's saved us a load of time in managing servers that would otherwise take away from the limited time we have to actually write code.
It's also a whole lot cleaner than a bunch of Chef scripts running a script across a quantity of servers; using Chef can too often be a crutch to fixing the actual issue.
You are assuming that they have several developers. I bet it's one or two
The article presents this as general advice and doesn't make any mention of a team size that's appropriate for the recommendations. Maybe this guy has only 1 or 2 developers working on boxes, but IMO for any number of people >1 you don't want shared accounts.
Also you are assuming this servers are staying with the company, which may not be the case
In many cases they will be, and, again, the article presents this as general advice. If a server is being configured for a client and sent out, then a general process of setting up local user accounts probably doesn't make sense anyway (unless you only have one client).
Agreed that LDAP isn't the most friendly of things to set up, but there are how-tos for common use-cases, and if you have any skill as a sysadmin, you can do it. As soon as it's set up, it's simple to maintain. Manually keeping accounts in sync across multiple machines is rarely simple. Maybe LDAP is overkill for a handful of hosts and user accounts, but if you expect to grow even a little, expect to need some kind of centralized user account system.
If you only have one or two developers, then you have all the more reason to automate the boring and mundane tasks, since the time savings are even greater as a percentage of total time available!
I just wrote an ansible script to do these sorts of things yesterday. Now, whenever I get a new server, I just run the script with its ip, and boom, provisioned.
>Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.
You're right -- many used cfengine. Still others used a custom 'config' rpm / deb that deployed all of these files everywhere.
Automated configuration makes sense not just for repeatability, but for auditability and documentation. Especially when you are going to 'hand the server over', the next admin should be able to know what you've changed.
Also, disallow password-based access to everything (use the keys, Luke.)
I guess it all depends on how often you bootstrap a new server. If you do it infrequently, keeping a configuration management tool updated is probably more work. I do it perhaps every two years and keep a similar list (I try to keep up to date if I change something in the server configuration) for that purpose.
For #4, wouldn't the only change when someone leaves the organization be to remove their key from authorized_keys for the shared account? Why would anyone else have to be updated?
It's a basic principle of security. Each account represents one person so that you have a full audit of who did what by watching the activity of a given user account. If everything is run as "devops" user for example, you have no idea who actually performed a given task. Was it Bill, or was it an automated job? PCI-DSS requirements also affect your model for user accounts (hint: shared users are often not compliant).
From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits.
Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least).
Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
anyone who compromises [the VPN server] is in a pretty good position.
Sure. But without a VPN, anyone who compromises even one of your other hosts is in the same position. It's a lot easier to audit a single-purpose VPN server for possible security issues than it is to audit all the application code running on the rest of your production systems.
And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
I wouldn't. And even if you're right, getting a VPN login still doesn't get you anywhere. You still have to be able to ssh to the rest of the hosts. That's why we do security in layers.
Ok, just in case it wasn't clear: I am asking about the advantages of having a VPN server as a bastion host instead of another SSH server.
You would ssh to the bastion host, and from there to internal hosts.
With the appropriate ssh config at the client end, the tunneling through the bastion can be scripted away (using the ProxyCommand directive and RSA keys).
Routing transparency. Sure, you can script a bunch of tunnels, but it's nice to handle routing at a lower layer. Having worked with both setups, I vastly prefer the VPN solution for ease of setup, use, and maintenance.
Also, different attack surfaces. Two layers of the same security measure (ssh) is, all else equal, not quite as good as two layers involving two different measures (VPN, ssh).
I have experience with many open source VPN servers. The purpose is a bit different -- we provide a VPN service to home users to encrypt their internet traffic. But the same problems should apply.
OpenVPN is the most compatible with a variety of clients. OpenVPN runs in userspace, so the clients for each OS and mobile platform interoperate well. The downside is, it does require a client program to be installed and configured. It's considered very secure, using SSL. Since it's userspace, moving large amounts of traffic means more context switching and higher cpu usage. Despite that, I've found it to be faster and more stable than the alternatives.
L2TP/IPSec is built in to most clients -- Windows, OS X, mobile. But every implementation is different and it's hard to configure a server to work with all of them. There are also more moving parts -- an IPSec server (openswan, strongswan, or racoon), and L2TP server (openl2tpd, xl2tpd) and a PPP server (pppd). IPSec seems to be a secure protocol but it's very complicated. I tend to distrust complicated security.
Pure IPSec has many of the problems of L2TP/IPSec with the added problem of difficult to configure in Windows and OS X.
PPTP is not performant or very secure. Other than the fact that almost every client supports it, I see no reason to use it for a new VPN.
Yes, I agree, I was just giving an example of how an additional layer can help protect against automated attacks, even for highly-secure services like SSH.
I also agree that SSH is less likely to have flaws than most VPN software. But on a properly configured bastion host, by-passing the VPN would just put you in a position where you can attack SSH. You would still need to by-pass SSH to access production servers.
I am, just from my experience in real-life companies. It takes an awful lot of discipline to treat servers as if they were exposed to the public internet when you know full well that they're not.
openssh is one of the most secure projects. It's developed by the security obsessed (and I mean it in a kind way) folks at OpenBSD.
I, for one, am ready to place for more trust in openssh than in any VPN daemon. The most commonly used ones are propitiatory.
What if there is a 0-day vuln (not exploit) for these VPN daemons? That far more likely. "Securing" ssh with a VPN is just one step beyond of security by obscurity.
If you are afraid of script-kiddies and scanners, let your sshd listen on a non-standard port.
Port scanners do generally scan non-standard ports too, you know...
I don't think that exposing SSH to the internet is that bad, but your argument is not sound - requiring a VPN does add security, because if there happened to be a vulnerability in it that allowed access, all it would do is expose SSH on the machines (I'm assuming you have proper firewalls set up), which you are advocating making public in the first place.
Saying it adds no security is false, because you'd require an unpatched vulnerability both in your VPN server and in the SSH server simultaneously. A zero-day in one is possible, but in both at the same time is far, far less likely.
> "Securing" ssh with a VPN is just one step beyond of security by obscurity.
You're not securing ssh with a vpn. You're adding another layer. ssh is still secured by all of ssh's existing protection.
ssh behind a vpn requires that someone both compromise the vpn _and_ compromise the ssh service to gain that access that, without a vpn, would require them to only compromise the ssh service.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited
So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.
When someone advocates using a VPN, that doesn't mean not using SSH too. VPN + firewall just restricts who has the potential to try to SSH to you, and provides additional protection and central access control/management.
Agreed, anyone who has touched PCI DSS would agree you need to associate access with a human user. This would not work. If you look at the security logs it won't differentiate between which keys were used for that generic account.
If they have had access to a shared account, how much work are you willing to put in to verify that there's nothing in that shared account that will get executed by another user later that will quietly reinstate a key?
A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater)
After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.
But if you give people any access to any system surely this is a concern. As a software dev, maybe I've inserted something into one of the build scripts that quietly re-opens my backdoor to the source control server...
I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?
It's commonly stated that 9 out of 10 security threats come from employees or other insiders. You should assume malicious employees. Sooner or later you will hire the wrong person.
Now, you must also have a functioning system, and so you may take risks by leaving things more open than you would like if you don't have the resources to thoroughly lock everything down.
But wherever locking things down further costs you very little, you should take the opportunity. And elsewhere you should asses what level of protection you can afford. Ultimately it is a cost-benefit analysis. Many risks are not worth spending money protecting against. Others are vital.
But even disregarding malicious users: Individual user accounts is not just a protection against malicious users, but against careless users. When someone sets a password that gets guessed, you want to be in a position where exploiting that persons credentials is as hard as possible, and tracking down actions taken via the account is as easy as possible.
And yes, you could insert something into a build script. But if the build script is committed, and the commit was pushed from a named, individual account, you're now at the risk of going to jail. Creating deterrents is often a sufficient risk mitigation strategy to be acceptable.
Quite the contrary: You're probably going to end up with documented procedures for deploying software that are simplified, follow existing standards/best practices, and don't rely on complex stone soup build/init scripts concocted by inexperienced developers (and I've seen some doozies).
A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.
Not quite sure I understand what you're saying here, I'm not talking about the software being produced, but the systems used to produce it.
What I was trying to say was that there's not really any way for you (server admin guy) to know if I (software dev guy) have inserted something malicious into a script that all the other software folks run constantly (software build system, NOT server build/init script, NOT deployment script).
This is not about the end-user's privileges, or server set up, just how in a team-base software dev environment you're probably going to have to have a measure of trust for your employees.
I see, but I think the same principle applies, even in this narrow case. As a server admin guy or fellow software dev guy, I have to trust that any code you've written has been properly reviewed before checking it into a repository that I pull from. Fortunately, version control tools make this trivial, but you're right, the policy and infrastructure supporting it has to be in place, otherwise you're depending only on voluntary peer review.
Note that as an attacker, there's a high risk of exposure and identification in the scenario you describe, and that's a good thing. A well secured system shouldn't merely prevent attacks, it should also protect innocent users from suspicion (another reason why shared accounts are discouraged).
But they still would know the 'deploy' password needed for sudo access. And while you could be relatively sure that they couldn't get access, you still couldn't be completely sure since they did have sudo access to begin with. So, the best thing would be to change the shared password. That could be avoided with non-shared accounts.
Are you insinuating that the user could have used sudo access to install a backdoor of some sort? If so, changing the password won't stop them either. Am I missing something?
Chef or Puppet are common choices. There are many others, but those two are modern, have large communities, and decent documentation. There's a decent chance someone has already open sourced a cookbook/module for each of many of these items!
On linode, which he is using, there is one built in - they allow shell scripts to run on first boot which could replay his commands as they are without many changes:
No, chef/puppet/etc are configuration management tools. They automate the manual typing of server setup commands, and then verify that the system stays that way.
They are not security tools. So you're on your own on what to actually tell the tools to do. "Install chef" is not a security tip. It's a repeatability tip, so you can get your system up to a known state repeatedly.
For the security side of things, you're back to figuring out what the right steps are, no matter how they're installed.
I'm a good developer, but setting up systems is something I do so infrequently that I always have to relearn it, and I am positive I could be doing things better. Do you have a link to an article that explains how to go about performing some of these ideas you mention?
We can be picky, but with good reason as security is an exact science and a costly one to get wrong. There's so much conflicting and out right bad advice posted online these days that sometimes it takes a picky community to help clarify the best practices.
For what it's worth, some of the advice given in that article was worth mentioning (eg fail2ban, it's a great tool). But the shared account suggestion was the complete opposite of how you should be managing user accounts as you lose audit trails. And for that comment alone, I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate.
That article has generated a lot of good discussion though. So even if just indirectly, it's been a valuable contribution to HN.
Picky is good! We never grow without constructive critisim. And a community of practice may yeild better results than an individual (at least for simple things).
> I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate.
Agreed. But the same could be said of everything; hackers and engineers, empiricists both.
Some of the advice in the article would seem apporpriate to someone who had never worked with another senior engineer. No fault of the OP.
System engineering and security are as much a learned craft as a science; a dialectic between sand castles (if you will).
Beginner or not, you should probably use visudo [1] instead of
vim /etc/sudoers
for the sanity checks that it provides, if nothing else. A botched edit of /etc/sudoers that locks you (along with every other user) out of administrative access is an unpleasant way to learn this.
Similarly "ufw allow from {ipaddress-you-will-access-from} to any port 22" sounds like a good way to accidentally lock yourself out unless you have an out of band backup
I've not tried anything complex with UFW so I still use iptables on my bastion host that handles my vpn tap. It's not terribly complex to make rules come back on startup (but probably more involved than one would hope).
This is how I run iptables on a sufficiently large network of machines.
The advice is not complete. IPv6 is real and really works most of the time these days. Back up your ip6tables to a file too. I like /etc/firewall-4.conf and /etc/firewall-6.conf but it's down to preference.
Know about iptables-apply too, lest you be caught unaware.
1. You should do "apt-get dist-upgrade" to get new kernel packages as well, otherwise you are stuck on an old kernel. (You might want that. I prefer updated kernel for the security, firefoxen, etc.). "apt-get upgrade" will only update existing packages - but the kernel updates require new packages to be installed.
2. If you're on ubuntu, root already has no password, and your initial setup user (whether it is called "deploy" or "kilroy") is in the sudoers file.
3. Other things I install in the "5 minutes with server" are: htop molly-guard screen git-core etckeeper
git-core because I prefer my etckeeper in git, but if you want it in bzr you don't need git-core. INSTALL AND CONFIGURE ETCKEEPER AS SOON AS YOU CAN, seriously. You need it. You'll thank me when you try to figure out when and how something in /etc got borked. (you need to edit /etc/etckeeper/etckeeper.conf if you use gif. You need to do "etckeeper init" and then "etckeeper commit" to establish the baseline)
molly-guard stops you from rebooting the wrong server
screen (or alternatively tmux) lets you keep your session open through ssh session disconnects (e.g. when moving from wifi to 3G, or between 3G towers that give you different external IP). The most useful way to use screen is "screen -xR" which also lets you share your session with someone else should you need to.
"screen -x" requires a session to already exist, whereas "screen -xR" will join one if it already exists, but will create one if it does not. At least it does in v4.00 which I use. If you have a "screen" call in your @reboot, you already have a session, so they will work the same.
I'm not sure I get your "on the contrary". Any configuration change on a managed configuration file is transient and bound to overwritten next time state is restored. This makes etckeeper absolutely less useful compared to a system where there is no way to restore state and an unfortunate "rm /etc/passwd" has little remedy. It does not make it "useless", but it's definitely less needed.
Possibly my misunderstanding, but I thought chef and puppet only push changes on the files that they are configured to control - There are tens of files in /etc/, some possibly modified by an "apt-get upgrade", then reverted by chef, some that chef/puppet do not try to modify.
etckeeper is a net that gives you an idea of how /etc looked on a given date or apt-run, not just how it was supposed to look (Which is what you get from the chef/puppet logs).
Question to chef/puppet users: do you restore the state continuously? or only when you've made a change?
Yes, it's true that chef/puppet do not overwrite your entire etc, just a subset of it, but presumably, you'll have your critical components backed by it (otherwise, there is not much point in using a configuration management system). So it is still useful on such a system, but not to the same extent of a system without configuration management.
> but presumably, you'll have your critical components backed by it
Often, the way you discover something is critical is after the first time it breaks. Backups may help, but proper version history is probably going to give better insight into what happened.
It is the point that you "never do that". However, it's been my experience that e.g. when troubleshooting, people always make local modifications, and don't always remember to revert them / restore from configuration management.
I have no experience with chef/puppet, so I might be mistaken, but I'm under the impression that they only push changes when asked to - which means a local change may survive for several weeks before it is overwritten. But I could be wrong about that.
Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.
I was in the same position - too many distinct environments for bash/Fabric, too little time to learn Chef/Puppet/CFEngine. Ansible [1] seems like a good compromise: you get the simplicity (runs over SSH) and host targeting of Fabric with the declarative nature and idempotency of the more complex tools. You can start with all-in-one "playbooks" [2], then split out tasks, handlers, Jinja2 templates, files, and variables [3].
As an aside, I think the default fail2ban config is too loose and quiet. Here's [4] an Ansible task file that configures it to be more aggressive and send notification emails.
I'm using fabric quite frequently, and am trying to understand what makes a configuration management tools a much better choice. I'm currently using fabric for anything from bootstrap a new environment from scratch, via restoring a snapshot from backups, to pushing code updates stored on git. Perhaps I'm being really daft, but it always evades me why something as simple as
I do use a pretty homogeneous environment, which makes things simpler, but this is a deliberate choice to avoid complexity. If I know all my hosts are e.g. debian 6, then what makes ansible/chef/puppet so much better than fabric?
I'm not trying to be provocative or negative. I'm really trying to understand the supposedly big difference between what's labelled a deployment tool, and configuration management tools.
Deployment tools like fabric are imperative, and configuration management tools are declarative. With configuration management, you define the final state you want the server to be in, and it will do whatever is needed to get it into that state. Some or all of the parts might already be done, and it won't change the parts that are already correct (the declarative configuration is idempotent).
Deployment tools just execute whatever script you hand them, and so the scripts are either more brittle (server must be in a precise state beforehand or it doesn't work right), or require more effort to duplicate the work that the configuration management software does to only make the needed changes.
If your deployment scripts get complex, it's more difficult to see at a glance what the end configuration is supposed to be.
Thanks for the clarification. I think I get the theory a bit better now.
I'm still struggling with seeing some practical examples of those differences.
for example, `apt-get` is pretty much idempotent already, no? if I run `apt-get install -y <package>` x 1000 times in a loop it won't install it 1000 times...
Fabric does give you the building blocks for those kind of checks elsewhere, such as `exists`, `contains`, `append` (which is supposed to also be idempotent) etc... I've designed my deployment / bootstrap scripts with fabric to take this into account. It does add a little overhead, but nothing that makes me feel I need a better tool... Maybe my deployment base is still relatively small and homogeneous.
So it's true that I have to put in those checks myself, and that I don't have a very easy way to discover what state a server is at.
I'll try to take another stab at one of those tools. Maybe things will sink in when I actually use them. Thanks again for explaining.
When you install the package you do it as a on-off. When a configuration system does it then it will keep re-happening if anything changes.
Say you install "less", and I remove it. With Chef this will be fixed the next run - with fabric? You'll need to re-run your magic script.
That's the main difference. Automation like Chef/CFEngine/Slaughter will keep fixing breakages - as defined by differences in state from their rules - whereas you'll be playing a game of catchup.
If you're only doing one or two things, the value is a bit more vague. But consider even the simplest interaction: the config file for a service should look like X, and if it has to be changed, the service needs to be restarted afterward. Oh and there are 10 config files, but you only want to restart the service once if any of them changes, after they've all been changed. That's not hard, but it's already starting to look non-trivial.
And what if you want to have the same logic for several services? I guess you abstract it out to a function. But then it turns out one of those services doesn't have a restart command, and you have to do stop+start. And another service won't start if you use restart while it's not running, so you have to check if it's stopped and use start, otherwise use restart.
It's much more than just whether the code is declarative or imperative, or whether it's idempotent or not. An imperative tool can change your system from known initial state A to desired state B. A declarative system can change it from whatever initial state it's in to desired state B, even if you never considered it might be in that state.
When you want to make a configuration change to your servers, and they aren't already in a "known state", why would you think any tool could put them into a known state? When a computer has a virus, say, you don't think it's been put into a "known state" after the anti-virus program gets done with it; the virus may have done any number of things you might be unaware of, altered any number of data or configuration files in subtle ways that the tool doesn't look for, but implicitly relies upon.
Example: what if, say, one of the provisioning requirements is "make sure this gem is installed", but on one of the servers /etc/gemrc has "install: --no-rdoc --no-ri" in it? Now on one server, the docs are missing, while everywhere else they're available. That sort of thing.
If you're already running on an IaaS like EC2, I think there's a simpler, better way: rather than trying to get "unknown state" to "known state", why not use the simplest known state of all--unprovisioned? Write an imperative script that reinitializes a freshly-provisioned IaaS node to your known state, and then do a rolling reprovision, terminating old nodes and provisioning new ones.
An equivalent comparison[1]: would you feel safe running an automated script that would SSH into a production machine and "git pull" a checked-out repo sitting on it, from whatever state it happens to be sitting at, up to refs/heads/master, so as to deploy code from it? Could you guarantee that that repo hadn't been moved to some state where refs/heads/master isn't a fast-forward commit? Or would you rather do a fresh "git clone"?
---
[1] A contrived comparison, though; although the former option is unreliable, the latter is terribly inefficient, and they're both horrible for keeping a .git directory inside of a directory that might very well be web-accessible.
apt-get is most definitively not idempotent without a lot of care. It will upgrade packages if a new version is available, and such upgrades can have widespread implications.
Otherwise I agree with you - Puppet/ Chef etc. are overly big on ceremony for me. We "only" use about 100 VM's on about 20 physical servers at the moment, so it's a "simple" setup.
I think where people get the most value out of systems like Puppet and Chef is if a lot of their servers or VM's have simple enough requirements that they can largely use "off the shelf" community provided configurations. The more customisation you need, the more the leve of ceremony starts biting you. The more you can rely on ready-made scripts, the more those apps help you because of the size of their communities.
I'm not that familiar with Fabric, so maybe it can do some of the things I mention here out of the box without a lot of custom code. I'm only familiar with Capistrano and I associate them in my head, perhaps naively.
Your example is great to illustrate why configuration management is so powerful.
Most importantly, your example will upgrade the package to the latest version every time you run it (which is probably not what you intended, but maybe it is). I'm sure there is an idempotent command for each of apt, yum, ports, homebrew, etc, but I have better things to do with my time than figure that all out.
In puppet your command would be:
package { 'foo': ensure => installed } # or 'latest'
What if you also have RedHat servers? Now you need to figure out the right yum command to use to perform the equivalent. Or your developers Macs? Homebrew. Chef/Puppet are aware of the environment they run in and install appropriate packages using the appropriate package management script with minimal alteration to the underlying recipe.
For RedHat systems, it would be:
package {'foo' : ensure => installed }
On a Mac, it would be:
package {'foo' : ensure => installed }
On that Arch Linux machine that your devs are running, it would be:
package {'foo' : ensure => installed }
Now, let's throw some dynamism into the mix...
What if you want to build an haproxy configuration file that pulls the list of hosts from an authoritative source? Do you "just" build it up from scratch every time and replace the file on every run of your script? With chef/puppet, you can pull that information in from your node classifier using the same query for dev, staging, and production and have the same recipe work for multiple environments. In some environments, the haproxy config has now changed, so I want to tell haproxy to reload. But in other environments, the config stayed the same, so I don't want it to reload. The file and service don't get touched if they didn't change.
What if a service has multiple files associated with it that when one or multiple of them change, you have to restart or reload the service? You don't want to restart each service every time one of the multiple files changes, so you end up having to build a queue to handle the service notifications.
Say I've got a few dozen users on my hosts and someone leaves. Which is easier - writing a purge script which knows how to purge the user from all environments that you support, or this...
user { 'joe' :
...
ensure => absent
...
}
I'm certain that that you can do these things with fabric scripts, but by the time you have, you've either written your own configuration management tool (which is unlikely to be as robust as chef or puppet) or you have a hodgepodge of unmaintainable scripts.
Eh, I didn't like Ansible. Every command is done over SSH, so you don't need an agent/ssh to box and run locally, but... scripts would take much longer to execute than in puppet.
We're currently using local runs of puppet, and it's easy to do dry-runs and see what fails before applying when testing. I'm not sure how Ansible gives feedback on that (I assume it must) but I find puppet pretty good about it.
Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.
Speaking from experience as a part time admin, it is definitely worth it to automate the process, even with one or two servers. Any build system will repay in spades the first time you have to set up a new server or rebuild an old one, esp. if you're in a hurry after a server failure.
That said Linode offers other easier options (Linode specific of course). You can take the commands usually executed, and put them in a shell script in the language of your choice which is uploaded and executed on first run (they call this StackScripts - there are lots of examples and helper scripts on their site, but the scripts are really very basic). It's perhaps easier than learning a DSL as it's just codifying the same commands you would run via ssh by hand. You can also clone nodes so you could set one up in a known good state and clone that each time.
This does tie you to linode, so long term one of the other options available would be worth looking in to.
You don't have to re-learn puppet/chef/cfengine/bash script every time. You learn it once. It requires much less effort to "maintain" than manually updating more than 1 box.
I think being a part-time admin with other responsibilities is a perfect reason to learn something like Chef or Puppet. It'll end up saving you a lot of time that you can devote to other things.
I think the author meant for this to be a simple set of security guidelines for beginners. But yeah, any reasonably large prod stack should probably be deployed and setup with puppet and what not. It's DRY and less error prone than a human low on sleep.
Without Chef there would be no way for me to rollout a new server in our cluster. Investing time into Chef was one of the greatest things I ever did. Chef is the best documentation of our infrastructure. The second best tool I'm using is fpm[1] to make custom debian packages.
While not directly related to this article, last I checked Puppet/Chef don't have particularly useful windows support. Is there an chef like solution for windows?
> No secure server is complete without a firewall.
Comments like those are why I normally point people to actual security expects (like, say, Schneier), and why I recommend that new admins should ignore as much as possible the practices chanted by the industry. A secure server does not need a firewall. A firewall can be used to secure a server against a specific threat, but that's it. The days of ping of death are behind us.
I would like to point out that following the article's guide and firewalling away ICMP, you can end up with a lot of trouble. (see http://serverfault.com/questions/84963/why-not-block-icmp). Some ICMP messages are not blocked by default by ufw, so I'm unsure how damaging ufw is when used like this.
At any rate, a Firewall is a block. A new fresh server install won't have ports that needs to be blocked. By putting up a firewall, there is nothing to be gained. Before the firewall, the ports are closed. After adding the firewall, the ports are closed. All that is gained is a hurdle the next time one wants to install something like a monitor tool (like Munin), or a new service.
It might be useful as a last line of defense against malware regarding outgoing traffic. I am normally against that kind of thing however (as focusing on the cause is better than the effect). At best, one can catch a spam malware, but any bot net, web server, ddos or other type of malware are untouched by the rules (port 80 and 443 is allowed). If the server has email sending configured so root message can be sent, then the spam malware can use that route and the firewall will just sit there.
So let's take a newly installed machine. What threats can be identified and what risks are we trying to mitigate with the help of this firewall (as specified by the article)? The only thing I can think of is either a Zero day TCP/IP stack vulnerability (not a realistic threat), or that the admin doesn't trust the other admins when they install new services. Yes, if an admin installs a new email server and enables relaying to the whole world against the explicit recommendation in bold font by the install wizard and the configuration file, a firewall can block that admins' actions. Then again, that same admin could just as well have disabled the firewall to "get the mail to work", so I'm not sure it's a viable defense against bad admins.
You are correct that a firewall will not magically solve all your problems, but it does help to protect against programs that open ports you didn't know about.
Recommending against them doesn't make sense, and implying that they are only useful to prevent TCP/IP zero day vulnerabilities is silly (especially since the firewall likely wouldn't protect against that anyway).
Schneier talks about NAT and 2004 windows laptops (ie, with winxp). I actually asked Peter Gutmann during an IETF meeting around 2005, and he confirmed that nat had improved the situation around and win98/win2k/winxp windows machines and bot nets. If I recall right, the gist of it was that windows machines needed something, and while nat is wrong and bad, it "worked" in this aspect.
This is about as far from a server installed with ubuntu in 2012 that one can get. You are not going to find any such article by Schneier promoting default firewall installations. I suggest here to check out Secrets and Lies by Schneier, as it is rather clear that a firewall need to be configured against the specific threats one can identify. If you fail at identifying threats, the firewall is likely not be useful at all, or will simply work identical to NAT. At worst, it will give a sense of false security.
> The only thing I can think of is either a Zero day TCP/IP stack vulnerability (not a realistic threat), or that the admin doesn't trust the other admins when they install new services. Yes, if an admin installs a new email server and enables relaying to the whole world against the explicit recommendation in bold font by the install wizard and the configuration file, a firewall can block that admins' actions. Then again, that same admin could just as well have disabled the firewall to "get the mail to work", so I'm not sure it's a viable defense against bad admins.
I think it's less about defense against "bad admins" than it is about protecting against accidental bone-headedness. :-) I typically set up a restrictive firewall policy even when I have a clear list of the services I'm running and/or I am the only admin. This comes in handy every once in a while, in cases where...
* A service is expecting more ports to be open than are documented. (Happens not-infrequently with license servers.)
* I'm re-using an old image and there are undocumented services enabled by default.
* A user decides to run a network service in their own account without informing the admins.
In all those cases, am I likely to change the firewall to "make it work"? Sure. But having to actually make that change helps keep an audit trail, and helps keep the admins explicitly aware of the attack surface. It's similar to why it's a good idea to periodically run nmap against your own servers.
In those use cases I agree. If for any reason one do not trust the software installed to behave as expected, a firewall can be a nice net to cover any strange behavior. If the job require Proprietary software services with an unknown/untrusted behavior, or the re-use of old images with strange stuff in them, I would too consider running a firewall. In the case of users however, I tend to apply some good-faith practice and just monitor. If something happen, I can always apply a firewall to deal with the situation. However, I do understand if that is not possible in every work place.
Here nmap do shine, and periodically running nmap is a technique that should be taught in universities. Great way for students to both learn about computer systems, and about learning how to debug problems.
Flashback quite a few years. I was working in IT and my coworker asked me if my WinXP (IIRC) machine was up to date. I said "yes". Next thing I know, it crashed hard. Oops, my buddy just hit me with a ping of death.
Well, yes they could - if the machine targeted is behind a FW that is blocking that type of traffic.
If the FW itself has that specific NIC, it could be brought down with this attack - but you could prevent SIP traffic from hitting machines which are vulnerable.
ACL based firewalls are the foundation of real network security. I'm not saying every server needs to be running its own firewall. But these security measures need to be implemented at some level, whether it's at a network level or on an individual machine basis.
There are so many reasons to put a decent firewall in front of your shiny server(s), but I'll leave here only one (you could find the rest on the actual security experts blogs for sure): Half open TCP connections flood.
Maybe this is considered a preinstallation step, but the very first thing to do is set the system time on the hardware, before you even boot the OS for the first time. Then the first step after booting is to confirm the time and reset it, if necessary. This is essential for accurate and usable logs, file times, version control timestamps, etc.
It's also a good idea to ensure that sshd has fresh keys that are unique to that machine. Hopefully, your images are installed without sshd keys, otherwise you'll have multiple servers with the exact same keys, which is considered bad practice. During initial configuration before deployment, you might want to remove the keys so that sshd will create fresh ones when it starts:
Looking through the responses here, I'm hopeful that someone will launch a "Sysacademy" variant for system administration training. There are tutorials scattered around the web, but (at least for those of us who don't know where to look), there doesn't seem to be one place that puts in under one umbrella.
> vim /home/deploy/.ssh/authorized_keys
[and] Add the contents of the id_rsa.pub on your local machine and any other public keys that you want to have access to this server to this file.
Cool trick: use ssh-copy-id <server> from the client machine.
From the man page:
ssh-copy-id - install your public key in a remote machine's authorized_keys
It is much easier than editing the remote .ssh/authorized_keys file since copying and pasting the key is error prone due to extra new lines typically added by the terminal emulator.
My big thing is making sure 1) the machine comes back up cleanly after a reboot and 2) is current on all patches 3) is running as little as possible.
Also a big fan of externally verifying what ports are open, and making sure the system is in monitoring, backup, config management systems. Config management is kind of optional if you have a small number of servers which don't duplicate configurations, though, and there's often no need to back up the OS, but any data should be backed up automatically.
Also check to see what's enabled to run at boot time via whatever your flavor uses.
Check for unusual daemons, ssh running on other ports (yes, the provider pre-loaded systems with a back-door ssh without disclosing it to us).
This is especially important when you are taking over admin on a server you didn't setup yourself. Other folks have weird ideas on how to admin things. Like webmin for example...
I also like epylog for finding unexpected stuff in the logs.
I went through the article and then read every single post on this thread. I am not a security expert so I won't even try to contribute except to say that I see a lot of people offering criticism without taking the extra step of explaining how they would go about hardening a fresh Linux install (or a pile-o-servers in a rack, whatever is applicable).
It'd sure be nice for those of us who are not security experts to read alternative approaches rather than, paraphrasing and not picking on anyone, "using a firewall is dumb" or "blocking ssh is pointless".
I like isolated ideas such as using a script to completely automate the provisioning of new boxes. Kind of a no-brainer if you ask me. The problem is that such recommendations are not followed by something like "Here's the script I use on Ubuntu 12.04 LTS".
How about it guys? Would you care to attempt to produce a canonical HN "How to harden your server" reference?
Maybe one of the security experts on HN can start a repository on Github to evolve a canonical script. I'm pretty much 100% Ubuntu 12.04 LTS, so it is my hope that this is one of the platforms that is addressed.
I did some looking around and this is what I found (I am in no position to evaluate the merits of any of these at anything beyond an intermediate level):
You are right, and I have some blame here of not including positive criticism in my comments. I do not have a script to offer, but I can go through the process I work by.
#1: A good password is a must. During installation, have a second computer generate a good password and either memorize it, or GPG encrypt it somewhere on the second computer. pwgen is decent in generating passwords.
2#, I fully agree with the article on automatic updates if its a personal computer. For others, one can have root mails sent if you are a fast and and read mail daily. This is how many people read about vulnerabilities before they reach the news.
#3: When installing large package like web services, I keep in mind of the long term prospect of each project. I ask the questions: Is there a deb package? Is it being maintained by a large group of independent developers? Is it mentioned in discussion at Serverfault? Are there any recent updates? What does the Wikipedia page have to say?
#4: read the man page, and check any section labeled security. Some man pages will say things like "we have this port open. Its completely insecure, and we expect either the local network to be safe or that you use a firewall". Through this just happened once for me, its still a good practice to check the man page with new services.
#5, avoid php themes/mods that require you to manually patch things. They won't be updated by Ubuntu, so things will either end with you uninstalling it or forgetting that it exist and thus get hacked. Sometimes ubuntu will just install over the mod, dealing with the issue for you.
Other than that, harderning depend on use case. A wiki/forum will need some form of anti-spam protection. A media center need access control or firewall to only allow local network. Unsecured protocols like nfs and nis need something like ipsec. A mail server needs authenticated smtp.
I don't know Ansible. I learned about it through this thread. It sounds very interesting. I'll have to experiment with it as I am getting ready to launch a service that will start out with one server but will probably grow quickly to the five to ten range with a few devs involved.
Towards the end of your page you talk about keeping a manual log file. That's exactly the way I've handled this sort of thing for years. I usually call mine "project-log.txt". These files usually have several sections and it looks like this:
Header: Project name, start date and other relevant details
Date entry: What I did on a particular date
Working on now: Before I do anything I make this entry. After an interruption I usually have no clue what I was working on. By making a habit of writing a note to myself about what I am going to be working on immediately I can task-switch to that phone call or unwelcome question and mentally come back to what I was doing quickly.
To Do: Self explanatory.
Questions: If questions come up I log them here so I don't forget.
Ideas: Sometimes as you are working you say things like "hey I should do x". I log them here and go back to what I was doing.
References: Links, etc.
After launch: There are things that might not be critical at all and can be done after launch. I try to only note must-have's in "To Do".
This file lives at a self-named directory: "~/project-log". I do this because I can also save other relevant files there and keep the entire thing under source control.
<Project Name>
<Date>
<Description>
<Notes>
--------------------------------------------------------------------- 01MAR13
<stuff I did on this date> format example:
- Installed Ubuntu 12.04 LTS
- Installed PHP
- Installed MySQL
- Installed Python
--------------------------------------------------------------------- 02MAR13
<stuff I did on this date>
--------------------------------------------------------------------- 04MAR13
<stuff I did on this date>
--------------------------------------------------------------------- WORKING ON NOW
- Configuring Virtual Hosting
--------------------------------------------------------------------- TO DO
- Configure vim for php development
- Customize bash
- Install imagemagick
- Install php5-imagick
--------------------------------------------------------------------- QUESTIONS
- How to automate server bring-up and hardening?
- Firewall or not?
- Disable SSH?
--------------------------------------------------------------------- IDEAS
- Learn Ansible for automated deployment.
--------------------------------------------------------------------- REFERENCES
- Ubuntu 12.04 LTS docs
https://help.ubuntu.com/12.04/
- Ansible
http://ansible.cc/
Nowadays I'm downright spoiled and use org-mode[1] to keep my systems journals. Org files are plain text as well, and org-mode takes care of setting up the tree by date. I can also add a journal entry from anywhere in Emacs with just a couple keystrokes, which makes it incredibly low-friction to use.
Like I said, the most important thing is to TAKE NOTES. Even pen and paper. It's one of Limoncelli's big points in Time Management for System Administrators.
Tooling doesn't really matter, the important part is being able to remember what the heck I did and when I did it. Invaluable for troubleshooting.
The reason nobody shared their one true checklist for security is that it doesn't exist. Well secured systems get owned every day, and tons of bad ones never have trouble due to luck and lack of incentive.
Most extensive security procedures contain a lot of questionable advice and few will prevent human error which most compromise can be traced back to.
The only semi-universal list amounts to
1) use keys for ssh
2) block/turn off everything except your (web) service
3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)
4) Learn about web application security threats especially sql injection and consider them at design and implementation and have some kind of regular review (this can be very difficult to get right)
5) Avoid storing anything that makes you a particularly desirable target like bitcoins or secret defense plans as anything can be hacked with enough incentive.
Most everything else probably won't make too much of a difference to you.
> 3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)
This isn't clear. Are you saying you think automating these updates is good or bad?
> 5) Avoid storing anything that makes you a particularly desirable
Well, in some cases your user's uid and pwd is the most valuable chunk-o-data a would be attacker wants.
> Most extensive security procedures contain a lot of questionable advice
List?
> and few will prevent human error which most compromise can be traced back to.
I think this would be the power of having a canonical auto-provision script on Github that many can review and contribute to. The script could certainly take the form of sections that could be commented out as needed. In other words, a well documented and reviewed set of recommendations that someone could edit based on pier-reviewed information in the comments and then use to automatically configure a server. That, I think, could be of value.
This is great!
I was thinking exactly along those lines, and started looking for puppet/chef scripts that are readily available.
It seems that using puppet has a pretty steep learning curve, at least from what I could gather skimming the docs.
Actually what I would really like to see is an interactive script, which then guides me through the process of hardening a fresh ubuntu server install, offering sane suggestions along the way.
I am pretty sure such scripts do exist, but having them maintained and regualarly updated somewhere would be quite neat.
About a year ago I spent several days trying to get a full puppet system. It was a real headache to be honest and many of the modules have no or very vague docs. Coming back to my setup it's very hard to read or trace what my setup is or how to do one off deployments.
Today I have started with ansible and have already gotten a huge amount done and deployed. Very happy with it. It's all in a small git repo and its very readable and concise.
It hasn't been updated too much recently, but I've used Bastille for this in the past, and it's in the Ubuntu repos. http://bastille-linux.sourceforge.net/
Yeah, I forgot about bastille. On their news page they claim that development has been restarted recently.
Lets hope that it gains enough momentum again.
Last I checked, Bastille was broken enough that you could fix it easily (I made a few small patches myself), if you knew what you were doing. Which kind of defeats a lot of the power of Bastille, but at least it was still kind of a handy automated checklist for seasoned sysadmins.
I've been using a product called scriptrock (http://scriptrock.com) recently that collects these kinds of best practices and ensures that they actually exist on nodes. I.e, the testing of configuration rather than the execution. I think this is really interesting for two reasons. 1, it's like monitoring for security on my nodes (it runs by default every 5 minutes), and 2, they could build some kind of open repo for people to contribute these best practices (I think they were asking for beta users a while back). Plus it allows me to configure my infrastructure in a kind of TDD way.
Add https://fedorahosted.org/scap-security-guide/, which I will at some point attempt to warp into an Ubuntu checklist, since I have always thought concordance on at least some security posture settings should be distro-neutral, with distro-specific implementations.
Writing a canonical server security document is difficult because it depends so much on what the server is doing (what kind of application is it running? how is it accessed?) and where you're starting from. (barebones install? something pre-rolled by your VPS service?)
Sounds as though we need a decision tree app to write the scripts for us. It asks questions such as, "Do you have your own IP block?" and explains the question thoroughly for those who don't understand it, explains the answers offered (for questions such as, "Which of the following types of logins would you like to allow (check all that apply)? Password, public key, ...").
It would offer you explanations, deeper explanations, defaults, recommendations for when you would override the defaults, etc. for each item. It would be smart enough to prevent incompatible or contradictory settings.
Once this process was complete, it would generate the needed puppet / ansible / whatever script, which the admin would run and store for future reference and use. When installing another machine, the original script could be read in by the decision tree and the new script could be generated by modifying the original rather than starting from scratch.
The problem I had with the article was that it said what user accounts he set up, how their security was based, etc. Why!? Don't give away that info in a blog post! Even if it is something a hacker could easily find out if they wanted, by doing that, you paint a target on your back.
If you are walking past a dark alley in a big city, you don't say, "I am a blue-belt in karate, and I have a wallet with $300 in it." They have a gun, and they'll take that wallet, thank you.
I totally appreciate you putting all those links together, but I would hope that others remember that loose lips sink ships when it comes to security. And from a hacker's perspective, what's the fun in social engineering if someone just blabs it all in a post?
If your system can't survive an attack because the attacker learned from a blog post that you are using tool X then perhaps tool X is the problem and not the blog post.
Real would-be intruders are not dummies. They have a suite of tests they can run to "x-ray" your system to the extent it is possible and discover vulnerabilities.
To some degree it's like encryption code. The safest code has to be open source.
This is a commonly held belief that is not true. I've been doing open source development quite a bit over the last several years and have seen plenty of insecure open source projects that were even less secure than I would see in the private repository of every place I worked. Here's why:
* Open source code gets more eyes on it, when it is well-used. But there are loads and loads of projects that are hardly looked at, and they have a greater chance to be used before they are thoroughly vetted.
* Those projects were most likely thrown up there by a developer like me who just hacked something up quickly to solve a problem. Once done with the problem, the code stays up there and I just let it atrophy. That consists of 95% of my projects, at least.
You misunderstood. I was referring to encryption code. These are open source projects that have to survive the scrutiny of experts. For example, I guarantee you that if I attempted to contribute to an open source encryption code-base without a massive amount of work my submissions would be rejected. This is not my area of expertise. Could I do it? Sure. Nothing is impossible. However, I would have to devote a significant amount of time to fully understanding the state of the art before doing so.
I've had df -h write out to the motd before for clients who insisted on tiny slices of disk, no automated monitoring and no manual checking of disk space even after it chewed itself up.
I'm not sure it helped, but it made me feel better.
It saddens me each time I see a security best practice guide that suggest turning off ssh access for root. Its a very useful feature, and the security industry should focus on the security problems rather than removing features without thinking about the actually benefits of doing so.
Sysadmins with root access should be able to handle a random 8 character or longer password, and that number is large enough for a secure public accessible ssh. If your not a sysadmin or unable to remember a random password, try go with a passphrase like "correct horse battery staple". If you have too many machines and thus can't remember passwords, then use a master password locally with a bit larger password and store certs there.
To do some math to illustrate the security of a random 8 character long password, someone would need to fill a 100/100 line for several hundred years to get through them all. By that time you should have noticed the constantly full connection, and be happy you haven't died of old age yet even after your 200th birthday.
However, 8 character passwords are not the suggested length by security experts. They suggest using a 10 character long password, as that is also secure in the case that your password hash somehow got lost.
For the several servers that I have, I have been more worried about the logs from failed attempts than I have been of anyone guessing the password. Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it. It also decrease the noise level in the server room.
You fail to understand that true security is not about passwords but control.
Disabling remote root login,
1. cheaply reduces attack surface
2. necessitates assigning administration rights to specific persons and roles.
3. greatly increases command visibility and value of audit trails.
4. combines extremely well with disabled terminal shell over SSH
5. can reduce (unnecessary) system resource usage.
Etc.
There are very good reasons why this practice is standard. Unless you are specifically creating a test or honeypot box of some kind, you would be foolhardy to ignore it.
You fail to understand that true security is not about passwords or control; true secrity is about risk mitigation.
Control is an ignorant tyrants last redoubt. Security is only about: identifying risks and dealing with those risks. All other messures are worthless as actual security objectives.
I am not arguing with you in priciple, only in particular. All of your points are well taken, to reduce risk, but not to control.
As the other comment say, Security is about risk mitigation. Risk in turn is about identifying threats, and attack surface is avenues where that threat might cause damage.
The problem I have with the articles claim is that limiting roots access in ssh is not improved security. It does not acknowledge the threat or attack avenue at all, but rather tries a blind approach. The threat is script kiddies. The attack avenue is bad passwords combined with password authenticated use of ssh by root. One can then either fix A) the password, or B), the access root has in using ssh. I pick option A and thus have have good passwords that secure even against internal users, rather than having bad passwords and limiting roots use of ssh. Having good passwords, and disable ssh for root is redundant and a expensive limitation for no security benefit. Having bad passwords and disabled ssh is bad security practice, as bad passwords should by policy be impossible.
> 2. necessitates assigning administration rights to specific persons and roles.
If you need to have different assigned administration roles for a single installation, and guarantee a correct audit trail, you should look into physical tokens or at least a kerberos+ldap setup. "root" user should then not exist as something that anyone should use. however, if you do not care about guaranteed audit trail (and to be honest here, the common use case does not require kerbros+ldap setup), you can add something like environment variables to indicate which user is logged, and ask administrators to document changes to the system.
> 5. can reduce (unnecessary) system resource usage.
This goal can be mitigated in several way. Thus in a honest discussion, this should be the target of it and not the pretense of increased security.
Well root is the default full access account. It is a bit easier to root@serveraddress and brute force in that way than needing to guess which account in the system has sudo access.
It also becomes easier to detect when someone is attempting to break in when you can see logs of common user names in a row fail to log in.
I don't do much admin outside my hobby boxes. One guess would be, excluding the ability to enter a root shell, each sudo command is logged so you know who ran it. If the user is root, then it's harder to know what they did while they were root.
Disabling root also means you now have to guess the username and the password instead of just the password.
If someone logs in directly as root, how do you track root access? Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root?
Discouraging root usage is common in desktop and laptop environments, and for developers on production servers. For admins on servers, root use is basically required for all commands, and sudo would not improve the security at all.
Commonly, I track access through environment variables (configuration on the ssh client), and documentation that each administrator do after finishing a task. As one should not recruit administrators that do rm -rf /, tracking (blame) is of an lesser importance and documentation a much higher one. If more was required, I would go with kerberos+ldap approach without anyone using local root user accounts at all.
I am the admin. I use the root account as little as possible. I do employ admins that don't rm -rf / by employing admins who don't hack on live as root. Knowing who does what is paramount for resolving problems particularly in admins' work practices.
If someone logs in directly as root, how do you track root access?
By their SSH-key and IP address (both of which are logged to a remote syslog-server).
Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root?
That is a false premise. I'd never encourage anyone to use "sudo" because it's a source of errors; people get the escaping and context wrong all the time, doing things that they didn't intend, and wasting precious productivity on getting the incantation just right.
It also doesn't add any meaningful security, it's a red herring.
If you need a dependable audit-trail then sudo is worth zilch. In that case you're looking at SELinux, auditd and friends.
If you don't need that trail (which is most people) then sudo is a ball on a chain and a false sense of security. Local privilege escalation exploits are dime a dozen, any shell-account can be upgraded to root by a dedicated attacker anyway (sometimes via sudo itself, as seen in in the recent sudo exploit...).
Oh, and let's not forget: If you enable a user to do anything meaningful with sudo then that normally includes giving him a way to write arbitrary files as root...
Logging the ssh key used sounds interesting - not something I've tried before.
I think you're doing something wrong with sudo. I'm not talking about "reentering your password" in the Ubuntu or Mac sense. I'm talking about giving access to specific limited commands.
And what distro are you running where local privilege escalation is common?
I'm talking about giving access to specific limited commands.
By definition the set is either so limited as to be essentially useless, or it opens (usually multiple) straightforward paths to a root-shell. Pretty much all sudoers-files I have seen fall in the latter category ("but he can only write the apache config-file!").
And what distro are you running where local privilege escalation is common?
Local privilege: yeah, that's a lot but over a long time span and none should be current on decently maintained box. I've never even seen anyone attempt a 0 day on my boxen.
Another "security through obscurity" pointless trick. If I'm attacking your server and it's not running SSH on port 22, it's a very short matter of time until I've found it anyway.
If you've got a physical safe that you keep money or valuables in, do you put it out at the kerbside so people can have a go at cracking it, or do you hide it inside in an obscured location to make it more difficult to have a go?
Surely it's the same with services on your servers, why needlessly advertise their presence. It's a tiny veneer of security but it is still more secure surely.
Wouldn't the script-kiddie through those hosts into a queue to run a more advanced port scan on? I mean, I see your point, but it seems highly useless compared to, say, simply disabling the password based authentication.
Why would they bother? Script-kiddie is just playing the numbers, any machine with a slightly non-default configuration is not worth the effort.
Disabling password authentication is better for "real" security, but doesn't stop those dictionary attacks spamming your logfile. (I still feel there should be better solutions to that problem than changing the port though)
I guess my response to all of these is... using key based auth is so much more secure than changing the port that changing the port seems to me, completely, pointless. Especially considering that:
- password auth = me having to type my password everytime, likely to have a less secure password
- key based auth = me never having to type it in, with a key length so long that it's completely inconceivable it'll be cracked or collided or brute forced
and then when you change the port, the only "advantage" I get is that I now have to remember the port because they're already not getting in by virtue of key-based auth.
> using key based auth is so much more secure than changing the port
Is it really? Once you get to a sensible password length (say, >14 characters) haven't you effectively made brute force and dictionary attacks impossible anyway?
> password auth = me having to type my password everytime, likely to have a less secure password
If you use a password often enough, I find the time difference between an 8 character and a 16 character password immeasurably small. It's all muscle memory anyway.
>key based auth = me never having to type it in
Perhaps I'm clinging to a false sense of security, but I find the idea of anyone with access to my local machine being able to access my server with 0 additional credentials really disconcerting.
Perhaps in the theoretical world if they can access your machine it's game over anyway because they can install a keylogger. In the real world, I think there's a fairly large population of people who might access your server who wouldn't/couldn't install a keylogger to steal your password.
Your private key should have a strong passphrase. I agree that using a passphrase-less key is bad practice. As for keyloggers, once they obtain your password, they have succeeded. But after obtaining a key's passphrase (which you normally only type in once a local session if using ssh-agent), they still need the key, which may or may not be marginally harder. It's still another step to overcome. So, even if it can be shown to be only a narrow improvement over passwords under certain circumstances, it's a big win in convenience and even security under normal circumstances if done properly. Give it a try, finetune your ~/.ssh/config and you might end up loving it.
> but I find the idea of anyone with access to my local machine being able to access my server with 0 additional credentials really disconcerting.
You could hook whatever locks your screen to also clear your ssh-agent then. This is probably something like alias xlock=$(ssh-agent -k && xlock) in linux.
The advantage of changing the port is to reduce the spam in your logs. Checking your server logs regularly really does increase your real-world security.
this. Everytime I check my logs there's tons of failed tries to port 22 yet none to the port ssh actually runs on. Which keeps the log with failed logins clean.
1. I don't think it's a good idea to use the same account by multiple people.
2. And you're not using a configuration management tool (like SaltStack, which is also a remote execution engine) this will give you:
- a central point to manage all your server
- predictable configuration on all servers with the same role
- a configuration documentation place (and even history if you git the confs)
- will make managing multiple users a breeze
So I'll agree with many of the commentators that several of the practices suggested aren't 'ideal.' However, they are easy and possibly better than having no 'practice' at all.
Just as an example, the shared user account with unique SSH keys per user. Sure, it's obnoxious in some respects, but many of the criticisms I'm reading in the comments like "but they could reinstate their access with a cron job that re-adds their key when they leave" and such are silly - presumably those who are using the shared account are developers/sysadmins with sudo privileges. Regardless of whether they have a shared account they have privileges to do whatever they feel like to the systems in question. Hence I'd argue it's a fairly reasonable solution for the situation when you don't have the time/resources to configure something more complex and you have to trust all parties anyway.
I think there are two larger takeaways:
First - Managing multiple users across many servers and dev systems is not easy enough, particularly for smaller organizations, and only gets worse when you try to get more granular about who can do what.
Second - umm... no idea anymore. Forgot what was second. Automation is good?
I'd propose using OSSEC over logwatch & fail2ban. Ossec seems to be a bit of an obscure tool, but a thoroughly functional one at that. Logwatch gives a bit too much info at once to interpret properly, while OSSEC will only alert you when something is actually up.
Ossec provides (among some other things):
* Log file monitoring, with severity levels, occurence counting, time-based rules and auto-response. This means (for instance) you get to watch your auth.log for failed login attempts and after 10 failures fire a script to ban him, alert sysop by email or have hubot alert your sysops. Or whatever floats your boat.
* File integrity monitoring. Make sure noone's been mucking around with your files. It has real-time support (through inotify), but if you don't want use that, make sure you store the database it keeps off-server for forensics if need be. Pro-tip: FIM and auto-updates are a tad unnerving.
* It can watch command output. You can use that to make sure the `netstat -tnap` output doesn't change, for instance.
* For larger/more compliant instances, it has a client<->server setup available.
Last version dates from 2012-11-19, not the fastest of releases - but seems ok. Trend Micro is involved (and offers commercial support).
It doesn't feel heavyweight to me. It does start a bunch of daemons for all of its processing and it has the client->server bit built right in. That may make it feel heavyweight, I guess. But, you don't have to use the client->server stuff if you don't want to. It'll still do all of its magic for you.
Ossec will do a lot out of the box. So, I suggest installing it with the default ruleset and the active-response stuff turned off (the installer will ask you). Then dive into the rules and ossec.conf (knowledge of regex is required).
Ossec is actively developed. They just had a new release a few months ago. Realistically it doesn't get updated as much as it used to, but it still is actively developed.
I think more than anything it is just kind of at a near end-state where there isn't a whole lot left to ADD other than some signature type updates, or nice-to-have features.
It is a program that I HIGHLY recommend having and it is very simple to get up and running, even if you don't have packages of it for your OS version. Also you won't pass PCI-1 without a IDS like this.
I wouldn't bother with fail2ban considering password based SSH logins are disabled (which is good).
Since the author is using ufw to control iptables, better to just use "ufw limit" rules for SSH port 22 to slow down the rate of any automated SSH bots trying to give your server a workout.
Indeed. I have always been a bit worried about such approaches, since they parse log files and attackers have some control over what is written to log files (user names and host names).
To prevent having to log into root via remote console, set up a second backup account with an ssh key and sudo access, and then put that key somewhere safe.
The chances of both that and the deploy user getting corrupted at the same time is unlikely.
I dunno about this guy but apt-get update/apt-get upgrade takes upwards of 15 minutes on new installs (as does yum upgrade on rhel-compatible systems). Kinda eats up into my first 5 minutes on a server.
Look into apt-cacher-ng. I have a slow internet connection at home and it made updates go a lot faster for multiple machines. Newer versions (in wheezy, not squeeze) even work with yum.
The guide recommends blocking SSH access to anything other than your own IP address. The problem is that my IP number sometimes changes at which point I end up locked out totally.
So to get around this you either have to allow SSH from anywhere or you have to use some remote KVM system. Most of the remote KVM systems seem to be based on Java applets which is not really something you want to enable on your system.
So what is the best way to get around this? Just open up SSH and be diligent about your SSH security? implement port knocking?
Set up a small AWS server to use as a Web proxy and SSH proxy.
Connect from you home with a remote forward port (for example 2222 to 22) additional to any other local forward ports. Put this tunnel in the startup scripts.
When you are away, connect to your AWS server, then connect locally to port 2222.
Now you've got SSH access to your home machine from anywhere.
I've never tried this, but could you enable access from a given hostname rather than IP address, and then use a dynamic DNS system like http://www.noip.com/ to automatically update a hostname when your IP address changes?
Two downsides I can see are that a) you would be vulnerable to DNS poisoning attack on the server side, but that would be the equivalent to just opening it up to all IPs, and b) dynamic DNS systems can take a short while to refresh the DNS record and/or it can get cached by your servers up-stream DNS servers. Some tinkering with your server to make it always use the authoritive DNS servers for your dynamic DNS hostname would help with that.
I'm just thinking out loud, I've never tried any of that!
This may not work for you, but I have a similar situation when I'm at home or I'm travelling. To solve it I set up a VPN at the office (we use Meraki hardware, so it was literally just a click) and connect to this first - then connect to the server.
Our provider allows us to modify firewall rules through their control panel. Perfect for when we have a dev working from home or a hotel room for a few days.
I use "adduser" instead of "useradd". Its more convinient as it does the remainings steps automatically for your. It comes with ubuntu by default. Any downsides?
I just use this bash script when setting up a new VPS. It pretty much takes care of it all for you, plus it sets up a ngnix/mysql stack already optimised for law RAM machines.
Also sets up 3proxy which is handy for viewing Hulu/Netflix :)
Imagine you are a fairly normal windows user or even sysadmin. Imagine you are considering Linux to replace some task that a windows server performs.
Now imagine the conclusion after reading this thread. As some one who can just about get something useful done in Linux, this thread makes me want to never use it again, it just looks too scary. Loads of disagreements which seems to have lots of dire consequences. OK, great discussion for deep geekery, but scary as hell for normals.
Now, when ever I see those annoying posts from smug Linux users who jump in every time a Win or Mac user highlights a problem, I now have this discussion to point to as to why Mac and Win users wont generally go near Linux.
Sorry chaps (and chapesses), I am on your side, Linux is a great thing, but this has to be the worst advert for Linux ever.
There is just one action on the post that will have dire consequences if ignored: disabling the ssh password authentication.
The rest are decisions that you don't need to rush into, and things you should learn when they make sense. Be advised that "the proper way to configure a server" is an elusive beast, you'll be always improving it, it's never done.
While I agree these practices seem pretty safe/standard, it's still enough manual work that many people will just skip half of it out of laziness for small projects.
Does anyone have good reusable Puppet (or bash scripts) published that they use on all new servers?
Here's something I wrote very quickly (bash script): https://github.com/vahek/vSetup
It does most of the stuff mentioned in the post, however doesn't setup automatic updates or Logwatch.
(to both corin and vahe): That kind of shell scripting is the horrorshow that motivated configuration management tools* in the first place. Shell scripts require a lot of added complexity to manage multiple heterogeneous servers, or to be idempotent.
> The days of passwords are over. You’ll enhance security and ease of use in one fell swoop by ditching those passwords and employing public key authentication for your user accounts.
ssh keys are better than passwords only because they contain (and require) more information. On the other hand, if your dev's machine is lost or stolen or compromised, so is your ssh key. This is especially a problem in environments with a shared account with full access, as you have. So, it's probably a good idea to make sure you're using a passphrase with your ssh key (during ssh-keygen), unless you need a passwordless login for a shell script or other automated remote system.
> passwd deploy: Set a complex password - you can either store it somewhere secure or make it something memorable to the team. This is the password you'll use to sudo.
Not necessarily. Anybody with access to the "deploy" account can use "passwd" to change its password to anything they like. (Edit: I'm wrong on this! passwd does require your current password; I've just gotten used to doing it for other accounts via sudo, which doesn't.) Changing the passwd on your own account doesn't require sudo. For this reason, I think it's better to simply give deploy nopasswd access to everything, and then delete and lock deploy's password to prevent it from being used at all (passwd -d -l deploy). You'll have effectively the same amount of security, but this way nobody will need to remember or retrieve a complex password, and you'll prevent, say, some accident in /etc/ssh/sshd_config from making deploy remotely accessible via a password.
You can do something better than this though, but it takes a little effort. Deployment is often the same steps over and over again (an rsync or an occasional apachectl graceful in my case). You can give the deploy user nopasswd access to only a shell script that's writable only by root; this way, deploy can still do 90% or more of their job without ever being given system administrator rights. You do have to be a little careful writing shell scripts though -- $* and "$@" still trip me up once in a while.
> Logwatch is a daemon that monitors your logs and emails them to you. This is useful for tracking and detecting intrusion.
This seems of dubious security value to me -- probably better as a generic sysadmin tool, so that you get annoyed by noisy logs and seek out and fix minor problems instead of ignoring them. Thing is, if someone does get access to your server, you pretty much can't trust it at all anymore. With services like Linode, you're really better off just launching a clean new instance, re-running your setup script (if you have one), and moving your data over.
I had to deal with the occasional intrusion in some pretty icky servers at an ISP once upon a time. We used rkhunter for a while, but I learned pretty quick that successful attacks against Linux servers are plenty enough sophisticated to alter all the basic tools that you would use to detect and remove the rootkit.
There is one caveat: I've been playing around with the idea of setting up rsyslogd to route syslog messages to mysql, and then using mysql replication to have an up-to-the-second offsite copy. I'd combine that with Snoopy (https://github.com/a2o/snoopy) or something similar. The point isn't to try to clean up an intrusion, it's to see how the intrusion happened so that I could close that hole. I haven't gotten around to setting this up yet, so I can't say anything terribly smart about it.
Finally: if you're going to have a problem with unauthorized access to your Linux or BSD server, it's probably going to be via one of its services, not via brute force ssh or anything similar. So, if you're concerned about this kind of stuff (and if you're being paid to be a sysadmin, you have to be), then you need to spend most of your attention making sure that your various services are set up correctly (apache/php/mysql/postfix/dovecot/spamassassin/etc. in my case).
The big problem with ssh keys is not being able to enforce ssh key passphrases on users. From the server perspective, you have no idea if the user has set up a passphrase. There are security standards which mandate certain kinds of passwords (complexity) and are silent on asymmetric keys, so you couldn't use keys in those environments.
The old solution was to do some post-login hack to require a password as well (e.g. to su), or do a VPN (which could have multiple forms of auth) and then ssh with keys after that, but the newest ssh (and I believe commercial ssh for a long time) now supports requiring multiple authentications per login, so you can do ssh key plus passphrase.
There are also DLP/etc. reasons why ssh can be problematic in some environments (i.e. where you're required to log/analyze actions taken by users, particularly admin users). The solution there is to use a bastion host and ssh in and then ssh out, with the user account locked down to log. SSH Communications (the commercial ssh people) have an interesting ssh MITM box which essentially does what all the SSL x509 MITM CA things do.
SSH keys also never expire and are easy to copy/steal. Key-based auth is officially discouraged at the day job (for laptop-to-desktop type things) for exactly this reason.
Expiring an ssh-key is easy, just remove the public key from authorized_keys. Dealing with theft is also straightforward: just add a passphrase to the key -- the time it takes to crack your pass phrase should be more than the time it takes for you to run ssh-keygen && ssh-copy-id.
The issue is lack of key management for ssh keys in a lot of environments, especially for accounts not always used. You can wrap protection around it, but you ay be better off just using a different authentication system (Kerberos? Something else?) in large environments.
Expiring an ssh-key on YOUR machine is "easy". Ensuring it's really gone on every, single UNIX-like box anywhere in your company is less easy. Oh, and what happens when you restore a home directory or entire machine from tape? Are you absolutely sure you remembered to delete every SSH key you wanted to "expire" at some point?
SSH keys do not satisfy a "fail closed" security model: they're there unless you explicitly remove them (and keep them removed). Certificates, tickets, and other expiring credentials eventually go away and lock users out unless they're explicitly renewed.
> There are security standards which mandate certain kinds of passwords (complexity) and are silent on asymmetric keys, so you couldn't use keys in those environments.
Every single auditor I've worked with has given a pass when they saw we were using keys instead of passwords. Do you also fail audits because your RSA token is only 10-12 digits?
You could, I suppose, demand users upload their private key (shudder) to the organization so you can check if it's encrypted; this has its own flaws (primarily that you've just taught your users to be giving with their private key), and you still can't enforce passphrase strength well.
Yes -- FISMA. There are both requirements that the implementations be FIPS 140 (although software is ok) and other requirements. I don't think I've ever seen anyone use a securid without a password, so the password complexity requirement is satisfied there. (It wasn't that an auditor failed it, but it was said we'd need to fix this to avoid any issues, so enh)
I don't have a particularly high opinion of most security audit standards, though.
Why install fail2ban? You already have SSH password auth disabled, and you only allow SSH connections from your office. Won't this just risk banning your own office if someone's SSH client is misconfigured?
If helps if a) you've misconfigured your "office only" rules, or b) someone has breached your office network/one of your local machines (but hasn't discovered any of your keys/passwords yet) and is trying to get into the server. It also helps with non-ssh services too. A belt-and-braces approach. If you accidentally lock yourself out using it, use your out-of-band access to re-enable your logins.
agreed, you can set up fancy jails for people scanning other services too, someone who probes SMTP/POP/IMAP doesn't need to hit SIP and SSH. Depending on the scenario you could choose to say block an entire netblock from hitting ssh after a single offensive IP probes a few services. Even a 10min jail time will cause most attackers to give up and move along to their next victim (unless you're being targeted.)
The first two things I prefer to do after I log in for the first time (on ubuntu):
> ufw enable
> ufw default deny
This way, after I log in, I will not allow anyone else to connect to my machine (I've had instances when by the time I changed my root password "bad guys" had already tried to connect to my machine).
Of course after I do the server setup (which is usually a script that will change ssh ports, install packages, etc) I will allow other services in ufw.
Once upon a time somebody wanted to give me access to a new system - they ran "adduser steve", and set the password to be "steve".
Two hours later, when I read the mail, that account had already been compromised. I knew it was a risk, but had no idea that it would happen so __quickly__.
Change the root password to something long and complex.
And bam! Not even a full paragraph in and security fail. root login should be disabled completely, and all use of privileges should be through sudo. Debian sets this up for you automatically upon install if you supply an empty root password. Of course, disabling root is just the beginning (and the first user created needs to be locked down, as they are now essentially root).
I was installing Arch on RaspberryPi and tried disabling root (since I had seen it done on other distros). Not long after I was fiddling with LVM. When I rebooted it halted on my fstab which couldn't find my VolumeGroups and required my root password to continue booting.
I ended up popping out the drive and mounting it on another box, changed my fstab to noauto, got things to boot and reenabled root.
I figure distros that disable root are setup for these corner-cases where root is required.
The other obvious case is if you're using LDAP and the network is down. Maybe I haven't looked in the right places, but I haven't heard these things discussed.
The foremost reason is that root is carte blanche to ruin a system. This may be fine for a developer seat or even a desktop that is not critical infrastructure (shared), but on a multi-user system (and this includes things such as email and web servers), you really should have a "think twice" prompt, along with logging of who did what and when (eg, sudo). Even if you leave root as ssh key only login, then once an attacker has gained that key (or found a remote exploit), they don't need a password; they're root. Setup administrator users who are given a limited set of sudo commands and only allow them to login via SSH with keys and require complex passwords for sudo, and that's multiple layers of protection and logging. Nothing's perfect, but it has long been recognized that root is a big gaping hole in UNIX security; that's why things like SELinux and RBAC were created, and it's why Windows for so long was so insecure (ie, the main user was essentially passwordless root).
The problem of sudo is that it's so easy to make a mistake in your command (and in sudoers). That's a security flaw in itself.
Also, you can always sudo bash, or sudo su.
You have a point about that extra password. But not about SELinux and RBAC being created because of that, nor in comparing the security of a system with root to one where eveybody is root.
All said, I'm still unconvinced. The logging isn't that usefull (I've tried it), and the extra password isn't relevant enough. Also, none of them will detain a malicious user (Linux has very week defenses against you grabing your own password).
Not if you lock down sudo properly. If you're just doing "username ALL=(ALL) ALL", then yes, you've got a big gaping hole. Even for my main administrator accounts sudo is locked down to a specific list of commands. As for logging, I get an email every time sudo is run on my systems. It's not built into sudo, but there are multiple packages which take care of such things. You can even set it up to do remote logging to an external source.
As I understand RBAC, it was specifically created to break up the responsibilities of root to different roles with privileges which were then assigned based on different access control mechanisms.
Linux has very week defenses against you grabing your own password.
This sounds like a major hole which I'm sure the security community would love to hear about; care to elaborate? Just to pick one example, how does one grab a password entered over an ssh session?
We do something similar, though we have a script that sets up the box. We use linode as well, so the script deploys a new box with a complex root password and my id_dsa.pub file. It also sets up /etc/skel and the profile files so that useradd uses them later.
We don't use a single deploy user though, instead, each user with deploy perms is in the sudo group.
Having a _shared_ account, with sudo privileges and a common password doesn't look smart.
Also, you're forcing devs to copy their public keys around.
I think you underestimate the benefits brought in by a centralized system, like LDAP, which would also allow you to manage the permissions with some more granularity
1) Many people seem to be recommending Puppet / Chef. How many servers or installs do you need before this is a good ROI? (Over using odd bash scripts or cPanel/WHM)
2) Am I right in thinking kernel updates don't get applied until the server is rebooted? If so, how / when do you manage this?
I've always thought 10 was the magic cut-off. I can imagine running "apt-get update && apt-get upgrade" ten times in a row before I get annoyed. Any more than that and I'd want scripting.
Really there's nothing stopping you from automating "some" machines and later pushing that out to the rest of your hosts. (Or just setting up trivial automation "configure NTP", "Disable root logins", "Append 'steve' to sudoers" - and keeping doing the rest by hand until you're confident.)
FWIW I use my toy configuration system, slaughter:
I wasted several days trying to get a good puppet setup. In the end it still needs more work, and the deploy system is complicated. I only have 4 boxes to manage.
Ansible looks great. Much lighter weight but it can still scale up.
Puppet and chef would both benefit by supporting single run light weight modes of usage.
I think it's more about future planning. If you only have a few hosts now, but know that in the future you'll have many, then you're just going to have to rewrite all your bash scripts anyway. Might as well do them "right" from the start. Learning Chef isn't that hard, and it won't take you that much longer for the first host.
Puppet/chef are certainly worth it for 3 machines. If it's an important machine they're worth it for even 1 machine (so that you have a disaster recovery procedure).
Given that there have been real kernel security issues recently I would reboot sooner. You can do a risk assessment per issue. Monthly automated doesn't seem the best option.
I do not recommend unattended upgrades. Every upgrade without any exceptions have the possibility to destroy any working system or your applications. Test and deploy would be wiser.
By the way, what the heck developers do on production systems?
Bear in mind that in small company, developer == dev op == sys op == network manager = testers == toilet cleaner. i.e. they do everything, dev or production.
As regards to unattended upgrades, again in small companies the choice is usually between them or no upgrades (no time for the test/deploy cycle across all platforms in use etc., less frequently used systems get forgotten and so on). As the article points out, its safest just to go for security updates only. If the security update breaks things, then yes, your system is down until you fix it. If a hacker breaches your system, your system is down until you re-deploy it, and then you have to deal with the PR fallout of lost credentials and data etc.
Its only anecdotal, but I've had no problems with unattended security updates on my ubuntu boxes in the 5-ish years. Security breach attempts are a daily occurrence.
It's nice if its not an either/or choice, but for many it is.
If you don't have time to test, you have yet another reason to not automate updates. It's better to run them only when you are around and have some time to fix whatever goes wrong.
Kinda concerned about the usage of ufw with fail2ban without the mention that you have to do a little more work for fail2ban to actually work with ufw (otherwise fail2ban causes interesting issues/clobbers ufw's rules)
I've never understood the point of fail2ban if you disable ssh password authentication. Yeah, it might eliminate some spam in the logs, but if you only allow key-based authentication that doesn't really matter.
fail2ban can deal with much more than SSH login attempts. I believe you can configure it to do arbitrary actions on arbitrary log events. Perhaps it comes with a block for HTTP, too?
Seeing as he blocks SSH and only allows it from his IP, there won't be any SSH attempts reaching sshd at all, so fail2ban wouldn't be doing anything if it was looking at SSH only.
My only thought is even with password auth. turned off hostile persons can still connect to the ssh daemon and might discover and unknown vulnerability or some type of dos. If you know someone is trouble why let them in even if you are wearing armor?
This will get buried but I recently posted a line-by-line guide of my first time setting up a VPS server, maybe it will help someone! http://cbracco.me/vps/
Is there a web app that I can just have control my servers from? I'd like to do puppet/chef, but I read that its not 100% working for Windows yet (I need to manage both) and it sounds overly complicated.
Start by changing your SSH port from 22 to something else. It stops 99% of automated attacks. Beware that it is obscurity, not security though. That said, it does help clear some of the "noise" to focus on the other 1% of attacks where the attackers are actually a bit more sophisticated than a simple script.
It's easier to get started with UFW, and you don't need to learn much about networking to be able to setup your firewall. I haven't checked this out (I prefer iptables and never actually UFW), but it's said UFW is a front-end to configure iptables.
I actually do this on my personal server because more than one public wifi (and one of my previous jobs, at a public high school) disallowed outbound 22 connections.
Sure, it'll not stop dedicated manual intrusion attempts, but it will actually prevent a ton of automated bots from even just trying to connect with common passwords through SSH.
2222 is a dumb choice as an alternative port, it's both obvious and quite commonly used. I'm using a port on 4XXXX-range that's normally not used for anything and therefore not scanned by the bots unless all the 65536 ports are. The automated login attempts disappeared almost immediately, except for a few that were quickly blocked.
Now the logs are clean from automated login bots, the only thing left are real dedicated hacking attempts that is worth pursuing further.
My take on all this is that you should spend 0 minutes on a server and be using a PaaS solution. There might be cases where you can't, but increasingly I can't find them.
I honestly do not understand why you need a deploy account with sudo access.
I much prefer a deploy accound which DOES NOT have sudo access. I add firewall rules transparently redirecting 80/443 to non-privileged ports that the webapp is actually listening to. Hence no need to be root / sudo'ed for the deploy account.
You then could get a bit fancier and set the login shell for the deploy account to /bin/false or something fancy (like a fake honeypot shell) to add a bit of "defense in depth" in case someone exploiting a hole in your web server manages tries to drop down to a shell. You'd then use another account to do the (automated) deploy/start/stop/update/patch whatever.
I'd also say that during the first five minutes you should set the default firewalling rules to REJECT anything and then only whitelist what is actually allowed.
> I'd also say that during the first five minutes you should set the default firewalling rules to REJECT anything and then only whitelist what is actually allowed.
At which point your 'net connection blips and suddenly you wish you'd paid extra for console access.
That's how I felt about the line where he recommends only allowing the deploy user to connect from a certain number of white-listed IPs. That's great and secure until you get a phone call while you're at the airport or on vacation, and then you hope that you can SSH into a machine at the office just so you can tunnel through from a white-listed IP.
Yeah, I've always felt nervous about doing whitelisting (and to some extent, things like fail2ban) for reasons like that. Having a couple of backup hosts / friends with shell-servers listed who you can bounce through might mitigate it somewhat though, whilst still avoiding 90% of the portscans-from-(china|mars) stuff.
One idea I had was to enable something annoying and kooky like port-knocking or OTP pass/connection enabled 'backdoor entrance' for emergencies, but ended up being too lazy, and realised it was just expanding the attack surface.
To my original point, we had an interesting setup for one network where the firewall changes were all manipulated via a script which required the change to be applied, and then confirmed after a short delay with a different command, otherwise after 5 minutes the ruleset would revert to the previous known-working one. It definitely saved some downtime/late night DC trips.
On FreeBSD (and other BSDs?), you can change the privileged port range with a sysctl setting, so you don't need to muck about with the settings, just let any user bind to 80 or higher.
1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one.
2) Everything else listed in this document should be configured by a configuration management system.
3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure.
4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does.
This stuff isn't hard. It's worth doing right.