Hacker News new | past | comments | ask | show | jobs | submit login
How can it be that unlimited high costs can arise with AWS?
26 points by depod 2 days ago | hide | past | favorite | 20 comments
Hello all,

I would like your opinion on unlimited costs at AWS.

One of my customers, had, the perfect use case for the Step Functions I read about. I worked with it for a few days and was able to see the first results quite quickly. Shortly after that, I started the service that used the API gateway to accept data and process it through multiple processes using Step Functions. The cost was about 50$-100$ per day. Due to another feature request, from my customer, I extended the Step Function with a waiting loop, which looked ideal in my mind. Unfortunately, my idea turned out to be a big mistake. I built some kind of loop in one of the steps. At the beginning everything looked fine and my workflow worked exactly as it should. I watched the Step Machine for another day. Everything looked good and I took a few days off after a stressful time. However, in my absence the queue kept growing and suddenly I had 1000 tasks running in parallel and the costs shot up to $2000 a day. On the last day of my little break, I pulled up the AWS iOS app and frightened when I saw the "Estimated Costs" for this month. 33,000 $.

This project is my first contact with AWS and went completely wrong. I directly contacted the support and hoped that I can be helped. I hoped they could give me a refund.

The whole ticket was opened on 01.06.2021 and closed on 10.06.2021. 9 days I hoped that I could be helped. At the end they apologized for the waiting time, but they could not offer me a solution.

For me as a self-employed person it is a disaster, since it is my fault.

- What would have happened if the infinite loop, would cost $10,000 per day? When would AWS respond? For every credit, no matter how small, the credit rating is checked, why can I incur unlimited costs with AWS?

- What can I do better in the future and more importantly, how can I prevent that costs could rise unlimited?

- Is there a way to turn off all services at once?

I avoid AWS like the plague because it's not small business friendly. It just seems foolish to me why any org that wasn't mid-sized or above would use it. There are plenty of cheaper vendors in the world! Tons of them! They can literally be 10x cheaper with more resources. Wild.

If you actually have to use AWS though because you have RFP requirements to meet, then I would ask whether or not you can halt instances using CloudWatch. I don't know the answer to that, but Amazon makes it seem like you can. Although their documentation seems to suggest you can only do it for idle instances.[1]

[1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Billing alerts. That should be the second thing you set up with AWS after MFA on the root.

For managed services, never put them into production until you understand the cost structure. Unlimited scalability == unlimited cost.

Thank you for your opinion. The billing alerts I actually had set up, stupidly I set up a budget of 1000$ hoping the budget in that case means all services will be shut down when they reach that limit. I received an email after the first 500$. I never thought I could spend more than $100 a day, so I didn't check with the Budget Manager often enough.

It's my own fault for working on that premise and not checking what budget means on AWS.

I always had this doubt

Do billing alerts work immediately? Or do they let you know like an hour, a day, or whatever, after the limit has been reached?

Not immediately, especially with predicted costs. In some cases with new accounts and low usage it can take hours for anything to show up.

Reminds me of warning message from Linux sys admins.

``` #3) With great power comes great responsibility. ```

I would suggest to write to AWS support again, explaining your financial situation and requesting some waiver on the total amount. In am aware of few cases, where AWS was supportive and reduced the bill. But it totally depends on your luck and person handling your ticket.

Only way to close all services in one go is to close your account.

Only way to close all services in one go is to close your account.

AFAIK, this isn't completely true[1].

Closing your account might not automatically terminate all your active resources. You might continue to incur charges for some of your active resources even after you close your account.


Apt comparison, AWS is basically sudo in terms of resource provisioning.

It’s a really dark pattern that you can’t set an account maximum and get cut off.

What does cut off means? If you add a lot of files to your S3, does it mean S3 should stop allowing more uploads or should AWS start deleting your files?

> What does cut off mean?

AWS probably has a thousand product manager and tens of thousands of engineers, and can't come up with some reasonable options for default limitations?

I bet that these 10x overruns have common causes: infinite bandwidth, infinite storage, and autoscaling to infinite compute. Your 1 TB of storage and 3 EC2 instance isn't causing the problem, and doesn't need to be deleted. Limit growth and you'd probably satisfy nearly all problems.

I'm not saying there's a simple solution, but I'm sure there is a reasonable solution, and AWS is a big place with lots of resources, and should be able to figure out how to let you set budgets like everywhere else you'd buy something from.

Here's a question: how would preventing you from spending more money make Amazon more money? Hint: it won't. Features that generate more revenue are prioritized over those which will decrease or limit revenue growth.

Definitely stop more uploads.

Deleting files is nasty but I don’t see an alternative.

But there must be lots of alerts if they do this!

Limit set separately for backups would make sense so you can recover. Even if your production limit starts chaos monkeying.

That's why you can define billing actions, arbitrary lambdas executed when you hit the limit you set. It's impossible to have a sensible strategy for everyone.

"Your storage device is full." Why should this be any different?

But you still have to pay for long term storage, that means your bill is still going up.

Ever use Glacier?

Obviously, the good analysts at AWS have the talent to cook up complex billing models.

Did AWS make Glacier free?

At the end of the day, you are renting hardware, not buying it. I do think AWS should've done a better job at billing notification and documenting the potential disaster.

Should be configurable! For many use cases deleting is fine, i.e: side projects

You know what the next post is going to be, right? AWS auto delete my EC2 with all my source code and data on it because my billing alert was too low!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact