My logs show increased break-in attempts after moving to Digital Ocean. Their IPs are hot targets for hackers, making it all the more important to properly secure my droplets, something I'm having a hard time doing in Linux. With Linux, it's lot of beating around the bush. Overcomplicated config files, a lot of disabling and removing of things one don't need, having to deal with a messy and outdated firewall. With OpenBSD, lo and behold I'd have one of the simplest, most secure and well-configured droplets the world has ever seen.
Yeah noticed that as well. First thing I do is (on debian), update packages and the kernel, install ufw (allow only 22), fail2ban, postfix and logwatch. Then setup a non privileged user, sudo, turn of root ssh and password auth and move to key based auth.
Then I can sleep/configure the rest of the system.
Been meaning to package this all up in an ansible playbook but I can't be bothered :(
I'd be using FreeBSD if they offered it. Much like OpenBSD it's a lot more pleasant for sysadmins!
It took me about an hour to put all of this into an ansible playbook. I think there are couple of examples out on blogs about this as well. If there is interest I'll share my playbook.
Writing this playbook paid off already for the second installation of the box.
The most I know about securing a Linux vps I picked up from this: http://feross.org/how-to-setup-your-linode/ . If there is an ultimate resource on how to start securing a *nix vps then please do let me know. If I knew what to search for I could but a lot of this is unknown unknowns to me.
Shameless plug: a small side project I did with my team at nodeSWAT [1]: CMify [2] - a web service to easily generate your ansible playbooks. Somewhat naive so far, if there is interest we'll extend it.
PSA: If you just update the kernel using system tools within the instance, it'll still boot the old one. (At least it did few months ago.)
The grub (or other bootloader) and kernels that are installed inside the instance's disk image are silently ignored and the vm uses a kernel stored outside of your instance's image. You can select which kernel image you want to boot in the control panel. The problem is that the kernels available in there are updated very, very rarely. For popular distributions like Ubuntu they can lag several months, for others it can get up to, say, _two years_ [1].
Since digitalocean "doesn't support custom kernels" it appears that the only way to use something up to date (and stay secure) is to write a script that loads a custom kernel using kexec. :S
edit: The fact that they use kvm (and that kexec works) which means real visualization but can't boot user's kernels is just very, very weird. The only technical reason i can think of is that they can't manage to make it work with their control panel. If that's the reason, then it's very worrying. (The "right" way is easier to code and it's the solution that you'd think of first and it makes custom kernels a non-problem, but they picked one that both has obvious problems and is harder to setup, i can't think of a reason why.)
Second shameless plug on this page: if running your own linux kernel or BSD is really important to you, http://prgmr.com can do that and we have out of band console access so you can debug why it's not working (I was having problems with that when I was checking whether some linux+xen bugs were also present on ec2.)
At this time we only support xen paravirtualization(PV) though, which almost, but not quite, all major distributions have. Notably we can't run 64-bit freebsd right now because they only have HVM support. I'd like to change that but it needs more testing first. Here is a partial list of distributions supporting xen: http://wiki.xen.org/wiki/DomU_Support_for_Xen Really any linux distribution with a modern kernel can be made to support it though as support has been in mainline for a long time now.
Actually, I really would love to use prgmr. The thing is, the price point is super far from Digital Ocean. Compare https://www.digitalocean.com/pricing/ to http://prgmr.com/xen/plans.html . The difference is a bit of a joke. If you're able to provide competitive pricing, I'd be thrilled though.
Yeah, I'm working on it. I announced upgrades to some of my customers several years back, and failed to deliver many of said upgrades, so I'm not going to give hard dates... but I recognize that competitive pricing is a prerequisite to the survival of the company, and I'm working towards that end.
As is industry standard, I'm upgrading existing customers (e.g. they stay on the same price plan, but they get more resources) before I offer those new price plans to the general public. I have maybe... 1/3rd of my customers upgraded to the plan I promised several years back (that has more disk and less ram than the plan I am upgrading to now.)
A big part of the problem is that I made a huge business mistake a year ago that cost me most of the money I would have used for upgrades. But, I've got a few options on the table, one of which is just consulting for a while;
if I go with used hardware, I'm within $50K of having enough hardware to upgrade everyone to "competitive, but not great" pricing. - that's maybe 3 months of full-time contracting work. Completely reasonable. Of course, used hardware is a lot like 'technical debt' - pay me now or pay me later, but it might make sense, just to get the monkey off my back. (That, and dealing with hardware problems is part of my core skillset; I can probably eek more reliability out of used garbage than most companies can.)
I'm talking 2-3x that for new hardware, which is a lot less realistic without some sort of loan or lease, something I should look into, but eh. I am thinking that used hardware until I'm competitive, then start buying new hardware once I start getting new customers on board might be the best way to go. The company is vastly easier to run when it's slowly growing rather than slowly shrinking.
I dug into this a while ago; I recall (but can't source at the moment) an eventual statement by the DO guys to the effect of:
1. their original "inject kernel into OS and boot from it that way" architecture was designed that way on purpose, so that users could downgrade kernels out-of-band as a way to rescue a bad upgrade;
2. but now they've got thousands of droplets configured that way, so switching architectures to something sensible will require either A. forcibly installing grub on, and restarting, every one of their nodes (not something people expect out of a VPS provider), or B. coming up with some sort of glue that can manage both the nodes expecting an injected kernel, and nodes that want to boot on their own, and providing a way to transition your nodes from one to the other.
Basically, it's a mess. A bit like the problems Heroku had with its Alpine stack.
Is it actually likely that someone would brute force a password? I always install fail2ban, but it's more just to stop bots from filling up my logs with junk and eating my bandwidth/CPU. Unless you use dumb passwords, the odds are phenomenally low.
I'm not sure, but I've come around to realize that it's always worth installing and configuring fail2ban just for the reasons you mentioned, if nothing else. I have a demo server that, for a while, was being hammered so hard by ssh brute-force bots that it kept getting knocked offline. I installed fail2ban and no more problems since. I personally can't recommend fail2ban highly enough.
fail2ban is based on the horribly flawed premise of correctly parsing arbitrary text logs from tons of different programs that wrote their log format without any concerns about parsing that output later. Not surprisingly there have been numerous vulnerabilities in fail2ban that let an attacker ban arbitrary hosts such as your DNS server, database, etc.
A better approach is something like pam_abl which is a pam module that will accomplish mostly the same thing but only for login attempts and without the crappy plain text log parsing.
I'm amazed noone ever mentions GRSecurity as must have. It will reduce the risks of an actual comprise significantly. Non-patched Linux is always full of rootkit potential.
Why would you use that over iptables? It's simple and doesn't require installing a package.
> There is a lot of functionality built into these
> utilities, iptables being the most popular nowadays, but
> they require a decent effort on behalf of the user to
> learn and understand them.
It's just a nicer interface to iptables. Instead of changing my iptables config file you can do things like 'sudo ufw allow 22' or 'sudo ufw allow http' for example.
You should still understand iptables but you do not need to config everything manually.
Simple is being able to simply say "allow all outgoing traffic and incoming traffic should only be allowed for HTTP(S) and SSH" and being able to figure out how to do it by just invoking "ufw --help".
Maybe someday I'll learn about iptables, I'm sure it's going to be worth it, but for now ufw does the job for me.
Indeed, it is. Even if you want to cargo cult that without understanding it, you might get bitten because running those commands again will not do what you expect, since they're not idempotent.
You will now reply telling me how to deal with this situation, for example if I want to now listen on a different port, or how I get FTP (or some other protocol that needs "-m state" to work. The need to do this proves that using iptables is more complicated that your example.
I've not played with iptables much, but when I do, I'm always struck by how easy it is to read rules, yet hard to write from scratch. This being said, a simple config like this is trivially googleable.
I think that's a little unfair... Your server needs to be hardened against drive-by SSH password guessing and similar nonsense regardless of how or where it's connected to the internet.
That is true but as per the poster above, I've seen 10x the attack traffic on a DO IP address so any zero-days and you're already herded into a predictable net block ready to be poked.
This is true for pretty much any VPS-only hoster, they all have well defined ranges and tons of targets to attack. I see thousands of attempts a day at Linode and hundreds at Leaseweb (dedi and VPSes) and DO. It's just part of doing business.
Rest of the world doesnt know I have ssh up and if I'm ever in a hotel or something I can tunnel through my home or login to DO's panel and allow another ip
Don't do that. I did that once and our static ip changed suddenly despite lots of money being spent on it. That was a painful day driving from London to Manchester for me to get to the console.
I could imagine this being quite dangerous, like being on holiday, there's a power outage, fire, ISP error, etc at home and no way to control your server.
I have essentially a static IP (A 'dynamic' IP with a 6month lease that stays the same on renewal and follows my account regardless of where I live), but you're super fucked if for some reason your IP changes. I usually set it to the /24 I'm assigned from.
