
An Open Source Tool to analyze wasted EBS capacity in your AWS environment - prakashmanden
https://www.fittedcloud.com/blog/open-source-tool-analyze-wasted-ebs-capacity-aws-environment/
======
ian_d
Nice tool, thank you for releasing it.

Unfortunately, in my experience it's a pretty normal strategy to over
provision EBS gp2 volumes to get the IOPS since they scale linearly up to
~10k. I think that's the break-even point where it's actually cheaper to
switch over to provisioned IOPS volumes.

So we'd have a number of little-utilized (in terms of data actually stored)
3TB drives just to avoid paying for provisioned IOPS.

~~~
RantyDave
Have you considered getting a single large volume and sharing it over NFS? If
this is a bad idea, why?

~~~
jimktrains2
Nfs can be slow and have consistency and locking issues under heavy write
loads. Also security isn't often top notch.

It really depends on the use case.

~~~
jimktrains2
Instead of downvotes, please explain why I'm wrong. Not only do I want to
know, but it's helpful to other.

These have been my experiences with nfs. It's great for certain things,
especially read heavy ones. It's very not great at write heavy or ones that
require locks.

~~~
subway
None of these statements are inherently true about NFS. They may be applicable
to some deployments or applications of NFS, but they don't hold generally
speaking.

NFS is used heavily throughout HPC for workloads that don't require a parallel
filesystem (and even then, pNFS is plugging along, even if it's largely
ignored in favor of Lustre et al for accessing parallel filesystems.)

~~~
jimktrains2
I used to work at a supercomputing center and nfs wasn't the choice for much
of anything. Lustre or AFS were the normal options depending on use case. That
was a decade ago.

Also, nfs is bad at applications requiring locks. Perhaps some implementations
aren't, but I haven't seen one.

~~~
toomuchtodo
We used pnfs at FNAL across 5000+ physical servers doing analysis of CMS
detector data. Worked well.

~~~
jimktrains2
Interesting. I hadn't used that. Thanks!

------
vasco
More than purely looking at disk space, I think it's more useful to look at
overall disk write usage which you can get for free by just loading up Trusted
Advisor.

~~~
antoncohen
Why disk write usage? Assuming the reason for analysis is cost, storage space
utilization is more important. GP2 and other non-provisioned IOPS types are
priced per GB. I've seen EBS costs amount to 50% of the AWS bill at a couple
companies, so reducing disk size can be a good cost saver[1]. You need an
agent/script running in the OS to get disk space utilization.

[1] As another poster pointed out, sometimes you want to over provision size
to get IOPS.

~~~
prakashmanden
Agree, disk write usage wouldn't show capacity utilization. One needs to look
at the file system utilization and make necessary changes (such as resize) to
eliminate wasted space. Our experience is that most customers are way over
provisioned from a capacity (and even performance point of view). Various
opportunities such as right-sizing (manually doing it is a pain), right-typing
(selecting gp2, st1 or sc1) io1 IOPS adjustments are available to
significantly the reduce storage cost. Automating those tasks is key so that
the solution can scale without user action.

