CoreOS is basically a trimmed down, customized Gentoo. It's not hard installing custom stuff on CoreOS, just be aware that CoreOS blows away a lot of the system directories on update, so you don't want to rely on putting things in /bin, /sbin etc.
/etc/ survives, though, so you can easily find examples of systemd service files to apply iptables rules, sysctl hardening etc, and I'm sure you can find fail2ban setups too.. It is also fairly trivial to put together a service file that'll run some arbitrary script you can put wherever you like to apply additional changes on boot.
Systemd will let you apply capability based restrictions etc. to fail2ban too.
CoreOS service files do get replaced, but you can override values in them using the systemd dropin mechanism to e.g. override settings for the ssh server.
Your main limitation is that CoreOS itself is intentionally very sparse. You have two options there: Deploy whatever dependencies you like or package up the code you need in a Docker or Rocket container, and run it with sufficient privileges if you want it to be able to apply changes to the host (this will work for /etc/hosts.deny based restrictions but not for applying changes to iptables, though it'd be easy enough to output changes to a file on the host and use a script on the host to apply those to iptables).
To be clear: CoreOS is not effort free. It makes very explicit value judgements, such as aiming for as much as possible to run in containers, that requires extra effort initially. If you are not willing to take extra effort to keep the "outside" host as clean as possible, CoreOS may not be the right choice for you. Most of the return on that investment first becomes noticeable when you're deploying larger numbers of servers/instances, and want to update them.
With respect to keeping it secure, note that the default behaviour for CoreOS is that all installed servers will check in with the CoreOS update servers regularly, and will apply the updates and will reboot once the updates have been applied.
This may or may not be what you want (it's easy to change - check the docs), but it means that if you're not paying attention, security issues like in this article will get patched for you. Doesn't mean you shouldn't pay attention, but the window for anything bad to happen is smaller.
If you are paying attention you can change the reboot policies so you can control when they happen, but if possible I'd recommend not to, as you can also make the system take a lock in etcd so only a limited number of your machines will reboot at any one point in time (or you can reduce the number of available locks to explicitly prevent reboots at times when it might interfere with something else, or set specific machines to manual updates only).
At least for anything running the alpha I'd very much suggest leaving it set to automatic reboot/updates to ensure stuff gets patched quickly when they find anything serious (not just security stuff).
Since you mentioned DigitalOcean elsewhere, DO's CoreOS images do retain the default of automatic reboots.
/etc/ survives, though, so you can easily find examples of systemd service files to apply iptables rules, sysctl hardening etc, and I'm sure you can find fail2ban setups too.. It is also fairly trivial to put together a service file that'll run some arbitrary script you can put wherever you like to apply additional changes on boot.
Systemd will let you apply capability based restrictions etc. to fail2ban too.
CoreOS service files do get replaced, but you can override values in them using the systemd dropin mechanism to e.g. override settings for the ssh server.
Your main limitation is that CoreOS itself is intentionally very sparse. You have two options there: Deploy whatever dependencies you like or package up the code you need in a Docker or Rocket container, and run it with sufficient privileges if you want it to be able to apply changes to the host (this will work for /etc/hosts.deny based restrictions but not for applying changes to iptables, though it'd be easy enough to output changes to a file on the host and use a script on the host to apply those to iptables).
To be clear: CoreOS is not effort free. It makes very explicit value judgements, such as aiming for as much as possible to run in containers, that requires extra effort initially. If you are not willing to take extra effort to keep the "outside" host as clean as possible, CoreOS may not be the right choice for you. Most of the return on that investment first becomes noticeable when you're deploying larger numbers of servers/instances, and want to update them.
With respect to keeping it secure, note that the default behaviour for CoreOS is that all installed servers will check in with the CoreOS update servers regularly, and will apply the updates and will reboot once the updates have been applied.
This may or may not be what you want (it's easy to change - check the docs), but it means that if you're not paying attention, security issues like in this article will get patched for you. Doesn't mean you shouldn't pay attention, but the window for anything bad to happen is smaller.
If you are paying attention you can change the reboot policies so you can control when they happen, but if possible I'd recommend not to, as you can also make the system take a lock in etcd so only a limited number of your machines will reboot at any one point in time (or you can reduce the number of available locks to explicitly prevent reboots at times when it might interfere with something else, or set specific machines to manual updates only).
At least for anything running the alpha I'd very much suggest leaving it set to automatic reboot/updates to ensure stuff gets patched quickly when they find anything serious (not just security stuff).
Since you mentioned DigitalOcean elsewhere, DO's CoreOS images do retain the default of automatic reboots.