

Ask HN: What are the most paranoid security measures for running a web app? - robmclarty

I have a few different web app ideas and some of them I feel would require maximum security measures to be put in place (e.g., storing someone's medical history, or a system for tallying votes without corruption).<p>What kind of tools would you use for something that requires high security? Is SSL enough? Are there alternative secure transport protocols? Should the entire database contents be encrypted? How would that affect data integrity and backup strategies? Are there certain databases that are more secure than others?<p>Ideally I would want the web app to be designed so that anyone can connect to it with a regular browser without the need for a special client-side installation, but I want to ensure that the information being transmitted and stored is as safe as it can be.
======
mschuster91
The first thing is to use full-disk encryption. Then, use ZFS because of its
unique snapshot features. Use a battery-backed RAID to prevent corruption on
power loss as good as possible.

Regularly back up your whole server disk off-site, and if possible, on tape as
well as on disks. This way you can, if a hacker compromises your system, at
least follow his tracks and minimize data loss.

Strictly firewall your servers - nothing except ports 80/443 public-facing;
use a "bastion host" for SSH logins and only allow public-key based
authentication (if you can, use two-factor auth, e.g. with SMS-PIN or Google's
authenticator service).

For databases, use only database engines with atomic read/write and disable
the write-back cache (i.e. force the database to only give a OK to the program
when the DB change was provenly committed to disk!).

Always protect SSH and other keys with strong passphrases so that in the event
your server gets compromised, at least a hacker can't easily decrypt your
data. Encrypt sensitive stuff in the databases (credit card data, personal
identity information).

Strictly limit management access to the servers to few employees. Protect the
hardware with features like auto-powerdown if the rack/the server case gets
opened. Disable the remote management interfaces present in many servers.
Disable the firewire and USB ports in the BIOS of the server and protect the
BIOS config with a strong password to prevent stuff like Firewire DMA memory
access.

Have a disaster management plan written on paper, which details who is
responsible for which operations part and how he can be contacted in an
emergency. Use a second datacenter for hosting a fully-redundant hot spare (or
production, if the code allows). Do not exclusively rely on Cloudflare for
protecting your servers from DDoS attacks; CF is a prime target for failures.

Keep the ops documentation current. Document _everything_ , because you can
always be hit by a bus, arrested by cops or injured or whatever.

~~~
robmclarty
Great points. I've never used a bastion host before. Totally agree that
documentation is critical.

~~~
mschuster91
Depending on your security needs, you can even go further - two linked bastion
hosts. The public-facing bastion host has two NICs, one for Internet, the
other one as a 1:1 link to the second bastion host (having two NICs again, the
first is the endpoint to the 1st bastion host, the 2nd connects with the
internal network).

Then make the public bastion host a VPN gateway so that the SSH bastion host
can only be reached by SSH. This way, even if the first bastion host/VPN
gateway appliance gets compromised, an attacker still needs a security flaw in
the SSH bastion host.

Oh, and I forgot: regularly update your systems! Subscribe to various security
mailing lists - this is the fastest way of learning about vulnerabilities.

And if you're running web apps: clearly separate the individual virtual hosts
using either a full chroot solution or stuff like suexec/suphp from each
other; same goes for the database users. Only grant necessary DB privileges
(e.g. no need to give a webapp db user the GRANT capability in MySQL).

If you're dead paranoid like me, set up the application servers as VMWare (or
other virtualization) servers, and for each webapp, create an own virtual
machine. Then, link the VMs with an internal network and create a "bridge"
server running nginx/apache/squid as reverse proxy (which can also serve as
SSL terminator to avoid spreading HTTPS certificates over dozens of
machines!). Maybe use special IDS software to detect/prevent common attacks
like SQL injections. This way, if your web-app gets hacked, at least the
attack won't easily spread to the other servers.

~~~
robmclarty
Thanks for all the advice! You've definitely introduced me to some things I've
never considered before. Now to cram more learnings into my brain... :)

~~~
mschuster91
:D Yet another thing that just came to my mind: use different operating
systems and software stacks on the two bastion hosts (or, at least, OS A on
the bastion host and OS B on the "normal" hosts). You may, for example, choose
a *BSD variant for the bastion host, and a Linux variant for the web hosts.

Simple reason: if the software stack of one of the hosts gets compromised
(e.g. a 0day is found in OpenSSH), then the other bastion host cannot be
compromised with the same exploit.

------
mschuster91
Use servers with built-in batteries (they're rare!) to allow erasing of the
RAM contents in case of power loss or intruder alerts (e.g. if malicious
entities break into the data centers and cut the power, the disk encryption
keys in the RAM can not be extracted by e.g. spraying the RAM with cool-spray
to preserve the content.

Set up issue-alerts for events (like: email all admins when someone opens a
rack or a server case to prevent rogue employees from unnoticedly manipulating
hardware). If you host your own data center: strict multi-level biometric
access controls, require the presence of minimum two or three people to enter
the secure data center.

Give the "doorman" of the data center a remote "kill switch" for the servers
(this also can be a PIN code for the doors), so that if the doorman is forced
to give entry e.g. by being hold at gunpoint or by a judicial order, the
servers can be disabled without anyone being able to prove it.

Record and store video and every other logs regarding datacenter access to
assist in post-mortem analysis. Take care to have at least two, better three
or four, dedicated power supplies for the DC, not including the diesel backup
engine and the UPS systems covering the time between power outage and diesel
kick-in. Same goes for the data uplinks (fiber interconnections to other DCs).
Guide them through different paths in the building and make the
fibers/powerlines exit the building on different sides to prevent accidental
dig-up by construction workers.

Regularly have your datacenter inspected by firefighting department, building
safety bodies and every other regulatory body. This provides certification for
possible other customers of your data center, as well as preventing closure of
the datacenter for violating obscure legal requirements.

------
adrianwaj
Have obscure passwords with keywords within them that are memorized in case
your password file is stolen or cracked. Record and lockout failed login
attempts (those records could be altered anyway.)

------
krz
Check out PCI DSS regulations for storing credit card numbers, they're pretty
strict.

~~~
robmclarty
Right. Will do.

