

How Cotap Is Building a HIPAA-Compliant Messaging Service on AWS - sergiotapia
http://stackshare.io/cotap/how-cotap-is-building-a-hipaa-compliant-messaging-service-on-aws?utm=email

======
martincozzi
Hi! OP here! Just wanted to clear up on a few things!

1\. Yes it's VPC we are talking about, and yes there is an added cost of $2
per region per day. Instances are also slightly more expensive when running in
a dedicated VPC. Overall this increased our bill by roughly %20-25. Having a
support account also increases the overall cost greatly since it's percentage
based of the total bill. However we've been very mindful of the money we spend
on AWS and our bill as of Today is still roughly equivalent if not cheaper
than the salary of a senior engineer. 2\. Regarding efficiency of messages
sent vs instances, you can't really divive num of messages processed by
instances and get an accurate picture. Every service that we roll out is
deployed across 3 AZs. So that means at a minimum 3 instances. Some services
don't need the capacity of 3 servers, but redundancy is important to us, and
we build everything with that in mind. We also invest a lot in monitoring,
logging and metric tracking. Think Riemann, Sensu, Statsd, Logstash (Indexers
+ Elasticsearch Cluster). I'd say about 1/5 of the infrastructure is dedicated
to monitoring/alerting/tracking. Most days our API backend runs off or 3
instances, our workers out of 3 instances. So if you must calculate it that
way, it's closer to 2M / 6\. But even that math feels weird. Then we have
various services like click tracking, photo resizing, video encoding etc. Also
our frontends are decoupled from the backend so that's another 4 to 5 services
on top of my mind ( so another 12-15 instances). And then there is our data
pipeline, crons, our distribution service (apt/gems) and a few other things
that you need to run in a VPC like bastion hosts, NAT instances etc. Those all
add up and contribute to the service but don't process messages.

What's nice about running most of those behind AutoScalingGroups is that it's
very easy to adjust and "scale down" on the hardware to keep the costs down
when not necessary. We found that c3.larges work well for most Rails related
stuff. Frontends can run just fine on m1.smalls and scale horizontally when
needed etc.

Hope this was helpful! Let me know if you have more questions I'd be happy to
answer them!

~~~
rubiquity
Nice work and thank you for sharing this. I'm currently working on a migration
plan to AWS for an app that needs to be HIPAA compliant and stories like yours
are useful.

------
comrade1
They have to use Amazon's VPC service - AWS has a 'secure' vpc service where
you don't share the server with other customers which is a requirement for
HIPAA compliance. That service gets really pricey.

I run a similar service but for SMS messaging that is 21cfr11 compliant. We
rent a 1/4 rack with our own door/lock and throw in pretty much disposable
servers as needed. It's much cheaper than doing the equivalent on AWS, and
it's not that difficult to set up your own secure servers once you have a few
recipes to follow.

We started it about 9 years ago before a secure service with amazon existed.
Every year or two I price alternative solutions like AWS's secure service,
VPCs, etc and it's always cheaper even taking account my labor to stick with
our 1/4 closet. In fact, it seems they base their costs on the going rate for
rack space, plus a premium for using the convenience of using their servers.

Oh, and it's funny - we have almost the same number of users and we're
processing more messages per month than them, all on a old-fashioned
webserver-appserver-dataserver arrangement - three servers (with fallover
redunandancy not counted) and it's barely breaking a sweat.

~~~
count
You dramatically misunderstand VPC. Everybody on AWS today uses VPC, unless
you happened to be grandfathered in to EC2 'classic'. There is NO COST to
using VPC (aside from admin over head maybe?). It's a huge security upgrade,
as it gives you the ability to isolate things much better.

The specifics of HIPAA/HITECH compliance on AWS are available under their BAA
(which is confidential), and do require some things that cost more money. VPC
is not one of them.

If you have only a quarter rack of servers and want the minimum s for physical
security, yeah, you can probably make that work cheaper than AWS. When you
have a whole datacenter worth of gear and now need a more sophisticated
security / entry system, etc. the argument becomes much closer.

With that said, you misunderstanding how VPC works and what it is makes me
wonder if your pricing is coming out right: it's really complicated to price
out a full AWS cost, just as much as to price out your own costs.

~~~
rubiquity
GP is refeerring to the dedicated instance requirement of having a BAA with
AWS. It costs $2 per hour per region on top of your existing server costs.

~~~
cuu508
It is $2/hour. So about $1500/mo plus regular AWS charges.

~~~
rubiquity
In my head I meant $2/hour but $2/day came out instead. Thank you for
correcting me!

------
fasteo
>>> process about 2 million messages each month... >>> ... We run about 60
instances in production

Seems like a very low efficiency in either the hardware or the software. We
run a comparable service with roughly 2 million messages per _day_ in a 6
server dedicated cluster.

~~~
dantiberian
Agreed, that stuck out to me too

2,000,000 / 30 / 24 / 60 / 60 = 0.7 requests/second/cluster.

However it's probably worth extending them the benefit of the doubt, there may
be other factors that they didn't cover.

~~~
count
Is your message handling a linear deal over the month? They may process 1.5M
of those in the last 8 hours of the day, which would be 3750 requests a second
instead...

~~~
thwarted
I'm not sure how you arrived at this 3750 per second number.

Processing 1.5 million messages for ⅓ of the time (1500000 messages / 30d / 8h
/ 60m / 60s) is an average of 1.7361 messages per second during those 8 hours
per day. If the remaining 500k messages were spread evenly over the remaining
16 hours a day for a month, that's an average of 0.963 messages per second
((((500000 / 30 / 8 / 60 / 60 )×2)+1.7361)/3) over the entire month.

Even if you were to process 1.5million messages during the _last 8 hours of
the month_ , that is only 52.08 ( 1500000 messages / 8h / 60m / 60s ) messages
per second during that 8 hour period.

~~~
count
I, uh, don't know how I got that either (must have fat fingered in calc). I
was talking about all in one 8 hour deal though. Thanks for the clarification.

