Could probably save a good bit with committed use discounts.
Spot / Preemptible instances would not work, in fact before Emma did this calculation a lot of people thought this kind of thing wasn't possible on public cloud because of perceived instabilities in a multi-tenant system.
I'm trying to do some estimates using the AWS Pricing calculator.
You can't go with a single instance and a ton of EBS storage, because it caps out at 16TB of disk, and 19Gbit[0] of EBS bandwidth, even on an instance with 100Gbit networking.
So, depending on how you can allocate storage, you're probably going to need some kind of clustered filesystem like GlusterFS
It's also not clear how well the application can spread it's writes - if it's all focussed on writing one file at a time, we need the most throughput to a single node at a time.
Storage:
Option 1: "GlusterFS: Hope it spreads writes"
20x c5n.9xlarge (36x vcpu/96GB RAM/50Gbit NIC / 9.5Gbit EBS) + 16TB st1 HDD storage each
= $43k
Option 2: "GlusterFS: more EBS IO"
20x c5n.9xlarge (72x vcpu/192GB RAM/100Gbit NIC / 19Gbit EBS) + 16TB gp2 SSD storage each
= $89k
Option 3: "GlusterFS: hey local storage is faster/cheaper"
6x i3en.24xlarge (96x vcpu/768GB RAM/100Gbit NIC/ no EBS)
= $47k
I was wondering about insane ideas like using mdadm in RAID0 over NFS Mounts presented by (say) 281 t3.2xlarge instances each with 1x 1TB EBS volume. That comes out at around 62k for the storage instances.
Compute:
I don't know how important CPU vs Disk IO bandwidth is.
The instance the author is using has 4x 15 cores (60 cores total).
The most I can get with standard EC2 instances is 64 cores, but that has 25Gbit network, and the next down from that is 48 cores.
1x i3en.24xlarge (96x vcpu/768GB RAM/100Gbit NIC/ no EBS)
= $7.9k
There are bare metal instances with more cores/memory and up to 100Gbit networking[1], but I can't find any pricing on them.
All up, I think $51-55k/month using standard instances would probaably do the job.
[0] There are the bare metal instances mentioned in the link below that get up to 28Gbit EBS per instance, but again no details on pricing.
> You can't go with a single instance and a ton of EBS storage, because it caps out at 16TB of disk
I just used the calculator to price out a single instance without issue. Just type in nineteen 16TB EBS volumes (you’d create an LVM volume group for them if launched). I used to have EC2 instances (albeit not by choice, I inherited the bad architecture) with 42TB total of EBS volumes using LVM without issue.
I didn't realise they'd upped the maximum volume size from 1TB to 16TB, so thought the calculator was telling me it was capped at 16x EBS volumes per instance. The new calculator isn't helping things here[1] telling me that I can only assign 16TB to an instance.
So, given that - then the issue becomes is 25Gbit NIC / 19Gbit EBS bandwidth enough IO to at least equal the needs of the task.
On paper the total bandwidth of the author's disk controller was 24Gbit, but that will depend on how the output is spread and whether the EBS limit includes any overheads that aren't present in DAS.
Interestingly, if the requirements are mostly sequential, you can get better performance/$ going with throughput-optimised HDDs rather than gp2 SSD.
Applying striping in either case will ensure you saturate the per-instance EBS bandwidth limits.
