I once ran up a $15,000 bill on Azure completely by accident when trying to get one of their video processing services to work. Once I figured out the service wasn't going to do what I wanted at a price I could afford, I tried to detach it and shut it down and thought I had succeeded. I didn't.
The offending process costing me money didn't appear on the Azure console and I had no idea it was running, how to access it to stop it or even to know what was going on. When it turned up on my bill I nearly had a heart attack.
Thankfully they let me out of it once I pointed out I was getting billed for something I couldn't see. I appreciated that greatly but I've never gone back to Azure and the experience scarred me so much I don't think I ever will. This was about 4 years ago so more than likely they have sorted it out.
They keep reminding me I have $50 of test/dev credit on Azure through my Visual Studio subscription but it flat out frightens me to even try to use it.
AWS isn't perfect but at least you have half a chance of working out what is costing you money through the billing console.
> AWS isn't perfect but at least you have half a chance of working out what is costing you money through the billing console.
The first few times I used AWS for tutorials, something similar happened to me. I thought I shut everything down, but kept getting billed and wasn't able to find it without contacting them. It was just a few dollars, but I've been wary about any services where you can't cap the billing.
Cloud platforms generally don't let users cap the billing, because those overages are good income for them. I prefer using services like DigitalOcean or Linode where you can be sure that your new site crashes for 15 minutes instead of bankrupting you.
Oh me too. That horrible "mythical beasts" tutorial. I thought I removed it, and then each month for the next two months I'd get nailed for 50? dollars or so.
By definition, I don't understand AWS, so figuring out how to turn it off was nearly impossible. AWS "support" didn't exist. Stackoverflow AWS geeks were in high dudgeon that I'd ask the question, "how do i disable this?" and would kill my question. Finally some kind soul did give me the trick to finding the last service to disable.
BTW the tutorial is absolutely useless. Just a thousand different incantations to repeat. No real understanding communicated. Felt more like an ad for myriad services.
There are consultancies whose entire expertise and premise are advising on AWS billing to companies who already know AWS, so the idea that a newbie should be aware of whatever magical incantation to limit their spend is ludicrous.
That’s not entirely fair. While I’m not going to defend the complexity of billing in enterprise cloud services it’s also not really that hard to set an billing alert in CloudWatch and track spend in Cost Explorer. Sure it requires a little bit of AWS knowledge but you shouldn’t really be using enterprise services like AWS (over other more accessible services like Digital Oceans) if you’re not willing to spend the time learning it; and at an individual level it’s very manageable.
The reason those consultancy firms exist is because billing scales terribly. Once you’re a business using AWS you’d likely have a multitude of projects running across a multitude of departments which need to be billed to a multitude of different customers and internet cost centres. This all needs to be processed by an internal 3rd party financial system managed by non-technical people who wont even know what AWS stands for let alone what it does and how it works. In those situations the problem of billing becomes exponentially more difficult than a one person hobby project.
> it’s also not really that hard to set an billing alert in CloudWatch and track spend in Cost Explorer
Billing alerts aren't good enough.
Consider a scenario where you're consulting on someone else's small- or medium-sized project and your bug costs the client a huge amount of money in the middle of the night. Now who pays? Say goodbye to your paycheck or reputation, even though it should have been preventable.
Another scenario: you launch a startup, and a bug empties the bank account and kills the company. If the solution is to just not use things like AWS and GCP (including Firebase, which has no billing cap) when you're getting started, why are they advertised that way?
> Consider a scenario where you're consulting on someone else's small- or medium-sized project and your bug costs the client a huge amount of money in the middle of the night
You can also set alarms that warn you of projected usage.
> Now who pays? Say goodbye to your paycheck or reputation, even though it should have been preventable.
If it’s legitimate usage then I’m not really sure what you’re advocating; are you implying a service being suspended in the middle of the night because a hard spend limit has been hit is somehow better for your reputation?
Or maybe you’re suggesting that it’s not legitimate costs, in which case you’ve set AWS wrong to begin with and thus your reputation probably deserves to be queried.
> Another scenario: you launch a startup, and a bug empties the bank account and kills the company. If the solution is to just not use things like AWS and GCP (including Firebase, which has no billing cap) when you're getting started, why are they advertised that way?
No cloud service operates that way. In that situation you’ll almost always get charges refunded. Even in instances of gross negligence (which would be the case here since for the bank account to be emptied it means you’ve not been watching your spend for more than a month and no business should operate that way)
I do get the points you’re trying to make but I’ve been working with the cloud for some time and have seen plenty of horror stories, all of which were due to gross negligence and most of which were still refunded by AWS as a gesture of good will. They’d much rather have your repeat business than burn their users with bills that cannot be paid.
I think the issue with billing is the same issue you get with OOMs - which services should go down first if we’re out of money? In practice “OOM” (Out Of Money) is even worse than OOM because with Out Of Memory you are just above the available memory threshold, but with Out Of Money you are literally at 0 capacity which means every single service needs to be killed.
I’ve built solutions for customers who would prefer an unplanned outage over an unplanned $250k expenditure. Many customers think that I’m a dinosaur for saying it, but if there’s a 2-5 year expected lifetime for something, it’s almost always more cost efficient to use traditional colo or VPS.
Also, operationally it’s possible to have something more than all or nothing. Big companies usually have “Tier-0” services that must be up all costs. AWS is no stranger to complexity - this type of function doesn’t exist because it would cost them money. They probably make 9-figure money from obviously idle services.
If you want your spend to be capped then there’s nothing stopping you from setting a CloudWatch alarm at a budgeted threshold that then scales down your infra.
AWS is like Lego. You’re supposed to build on it to create the behaviour you want
Hum, I don’t know think you understand what I am saying:
If you are out of money, all of your services needs to be destroyed immediately: including all of your database hard disk drives, because every single piece of infrastructure yields some regular cost.
It means that it is NOT going to be “a simple unplanned outage”, it’s going to be more akin to formatting your hard drive with all of your family photos on it.
Pretty certain AWS just finds it easier to sometimes write off costs rather than implement something so radical that customers may be even less happy afterwards.
> > and at an individual level it’s very manageable
> And yet, we have horror stories of students and even experts being hit by surprise AWS bills.
They obviously didn’t bother to manage it. But that doesn’t mean it’s not manageable. It takes all of 5 minutes to set up a budget alarm on CloudWatch. It was one of the first things I looked into doing when I set up my own AWS account years ago specifically because I didn’t know what I was doing back then and thus didn’t want any surprises. If I managed it then, then I find it hard to believe others cannot too.
It's been 5 years since I was responsible for anything AWS, but back then at least support was surprisingly good even when spending less than $100 a month. Emails would get answered and I could often get someone on the phone within 24 hours if I needed.
I think it depends heavily on what area you want support for. EC 2 support is generally very good, billing support is reasonable if a bit slow. Support for a lot of their managed services, or media services is beyond useless. All the support engineer does is take your complaint and says they will check with the internal service team. Your ticket will just end up in "waiting for Amazon" state for weeks or months.
> BTW the tutorial is absolutely useless. Just a thousand different incantations to repeat. No real understanding communicated.
I share this sentiment after going through a lot of tech tutorials and onboardings. And it's not limited just to AWS. I find myself forgetting most of the information I was supposed to learn. I'm trying to be extra mindful and offer additional explanations when I'm writing procedural guides myself, but I still have a lot to improve.
I understand what you feel completely. I actually wasn't even able to complete the tutorial since the commands given would trigger a permission error one way or another. I got charged $50 as well since I wasn't aware that there's still services running. Such a terrible thing to front as a "Getting Started" guide.
Disclosure that I'm the Co-Founder and CEO of Vantage - and used to work at both AWS and DigitalOcean....but this is a large reason we offer a fairly generous free tier on https://vantage.sh/ to help folks figure out where their costs are coming from to take action.
It can be really frustrating and annoying when you can't hunt down where costs come from.
I'm nitpicking on unimportant details here but this seems like more of a general UI/UX problem than a free tier problem.
AWS and Azure both have pretty straightforward free tiers, but people end up accidentally racking up large bills anyway. A UI to see a list of running services, sorted by cost either doesn't exist or is not prominent enough.
It's slow, you can't open links in new tabs, one wrong click and you have navigate to the right page and wait for everything to refresh again, but you can get the list of billed items. Plus a "Other subscription charges" that contains some mystery costs not linked to anything.
The state of cloud billing seems like exactly the result you'd get if you didn't have a strong mandate that product teams implement a unified, centralized billing interface.
And it makes sense from a growth perspective: new products grow revenue, bad billing systems only annoy customers (but mostly invisibly).
We have services across a lot of cloud providers, Google and Azure for a variety off things, Digital Ocean for some and the main stuff on AWS. I'd love it if a service such as Vantage could track them all, I can only find solutions for AWS.
I feel like a lot of cloud platform tutorials out there should start with having you create a billing alarm that emails you when the bill crosses a reasonable but higher than expected amount. For basic tutorials, that might be something like $20.
> Cloud platforms generally don't let users cap the billing, because those overages are good income for them.
No. They don't have caps because it's really hard to implement and because the negative press of an app going down because the cloud didn't scale is far worse than any received by surprise bills. Scaling is a large chunk of what you're paying for, after all.
This is quite obvious when you look at how lenient AWS is with retracting surprise bills. And the money perspective doesn't make sense, either: AWS is living on customers that have bills in the 5 figure range and up. The occasional 10$ from someone playing around aren't even a drop in the bucket.
You sure? Say your app is reaching the cap. What do you do with ongoing costs? Do you block writes? Shut down VMs? Delete stored data?
So you might say, simply project the cost (with some magic) and prevent that from going over the limit. So, imagine, your app suddenly experiences a load peak and you need to scale up. However, adding a VM would increase that projection too much. Do you not scale, despite this possibily being a small load peak, and let the app go down? Or do you risk the situation described above? Doesn't matter, you'll get bad press either way.
And beyond that, you'd still need to project costs like traffic volume, which can vary extremely. Not to say anything about the technical difficulties of coordinating that billing information across hundreds of services in real time.
And even if you do all that, you still get bad press of the likes of "we forgot to remove our payment limit and it killed our app while being on the front page (and our alert did not trigger because we couldn't afford another mail)".
There's no way AWS (or any other cloud) is eating all these drawbacks just to have a limit. I bet it's orders of magnitude cheaper to just eat the occasional surprise bill.
Does it shut down at $150.00 or does it shut down “some amount of time and unknown dollars after you cross $150”?
I’m willing to bet it’s the latter. If in a normal account, you’d then incurred $152.78 or $166.39, did the limit work? Would customers agree?
My cloud bills continue to change for several days past the end of the month (for legitimate calculations that come in for usage incurred during the month).
We couldn't technically make it stop exactly at $150.00, but only at $167.89 or whatever, so we are letting it run to $15k.
For catastrophic cases it doesn't matter. If it saves a person from an unexpected $15k bill then it works. Even for many businesses it would be ok to drop everything - I know some which can withstand being offline for a day, but not a $250k bill.
Make the MVP opt-in, delay any irreversible stuff by a few months (ex. deleting s3) with a deposit to cover costs, figure out the rest from user feedback? Aka do it like any other new feature is developed in a modern shop
> You sure? Say your app is reaching the cap. What do you do with ongoing costs? Do you block writes? Shut down VMs? Delete stored data?
Two approaches:
A) Hard limits: freeze the services immediately if your cap is reached, ideally by giving a heads up some time beforehand with predictions, if possible; this is what many VPS providers out there do for unpaid bills and such, which makes sense
B) Courtesy: allow the services to keep working, but at a degraded performance level - that's what some of the other VPS providers out there do; for example, decrease disk performance, cap the CPU performance, limit the network speeds etc.; probably eventually also block writes, but don't delete data outright; any of the aforementioned should trigger monitoring alerts on the developers' side and Zabbix or another solution would alert them in minutes, as well as the vendor should also send e-mails about these measures either currently being put into place, or about to be put into place, so that the necessary actions can be taken
> So you might say, simply project the cost (with some magic) and prevent that from going over the limit. So, imagine, your app suddenly experiences a load peak and you need to scale up. However, adding a VM would increase that projection too much. Do you not scale, despite this possibility being a small load peak, and let the app go down? Or do you risk the situation described above?
There's a difference between having the current capacity with a degraded performance during the spike and killing the entire app. You don't always need to scale up, depending on your failure modes. Having consistent service response times is overrated, as is needing to serve every single request without ever telling a small portion of your users that your service is experiencing high load - there should be solutions in place to deal with the backpressure and prevent data loss even under these circumstances anyways.
Unless you work in a Governmental organization or another critical piece of software for society, degraded performance is probably okay and no one feasibly cares or remembers even small outages - regardless of whether it's large sites, or small non profits or even side projects. Whereas if you do, then you probably have enough money to throw around for billing caps to not be relevant.
If you subscribe to those beliefs about always needing to be up and serve requests, however, then there's another option:
C) Billing alerts: something that most of the providers out there already provide in some capacity, however in fairly bad ways; if AWS can bill you for Lambda functions on a 1ms basis, then there's no excuse for not receiving billing alerts the very instant when this spike first happens: https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-changes-duration-billing-granularity-from-100ms-to-1ms/
Better yet, allow your clients to choose which of those mechanisms they desire to use, in the order of the potentially least expensive (infrastructure wise) to the most: A, B or C. That way the little guys for whom a 10k bill would be life ruining could just use A, whereas startups could stick with B and huge corporations who have a large runway of cash to burn could use C.
> Doesn't matter, you'll get bad press either way.
Bad press? As opposed to what, going broke and not being able to pay your rent because of unpredictably large bills with no way to limit them, just because your side project got popular on Reddit or Hacker News?
There's a world of difference between what's needed by corporations and what's feasible for private individuals, so for as long as there's a chance of such bills, i will not use Azure, AWS, GCP or any other platform like that.
Remember: these surprise bills will only be "eaten" by the larger providers based on their own goodwill. There's not much preventing them from banning you outright.
Wow, it's really interesting to read how clueless folks are here on HN.
The reality is kortilla is the same person that if AWS deletes all their data when they hit their "cap" to stop the billing will be on here complaining.
Payment method not go through? Hit your cap? To stop charges AWS needs to delete forever almost everything in your account. All your S3 data gone. All your backups / databases and archives gone.
Oh - you actually DON'T want them to blow up your platform? Maybe they could provide instead a billing console
Some folks seem to have almost no clue about what AWS customers who pay the billions want. Is there ANY chance that AWS listens to its paying customers? Maybe has become successful by doing so (at the cost of total feature sprawl in my view?).
My quick trick is to login a day or two after I think things are shut down and look at projected bill and current month billing. I've left some very large instances running a time or two, easy to turn off.
And with billions of requests (PER SECOND) on the aws network, there is NO WAY they are doing real time billing. That is not happening. Look for daily aggregation and similar. Just the scale of permissioning on API calls must be insane per day. These are going to need to be doing local counters that aggregate periodically.
I do wish they'd maybe aggregate 4x per day (6 hours).
They should go talk to Azure then, because Azure for Students gives you $100 per year. The second you hit that $100, it kills everything.
I’m not saying it would be simple, but it wouldn’t be a bad idea to have a global monthly billing maximum where it’ll nuke the account at that number. There’s lots of new developers that are probably too scared of the free tier to use it without something like that (I have AWS experience, and I don’t use it for personal stuff specifically because that doesn’t exist. Which means I probably won’t tell my employer to use it either).
I do like this idea, but for their bigger customers a cap where billing (and services) all stop is not what customers are asking for. Instead they are asking for durability / resilience / object locks etc.
I'm serious, what large businesses wants to lose EVERYTHING (all static IP's, all glacier and S3 data, all database and compute) over a billing issue.
You have to see how that's both totally different and a terrible idea for a business, right? If anything the existence of hard limited student accounts combined with the fact that support always* refunds giant surprise bills underscores the point that caps for business accounts don't exist because of the damage they could do.
Surely it's the customer's role to decide whether or not it is a bad idea?
How 'bout you run a non-profit and have an allocation to run the services, would you prefer your nonprofit to lose the website for a couple days, or for it to go bankrupt?
What if you run a company for which online presence is a means of advertising and not the revenue-generating platform, would you prefer to run your advertising campaigns on a budget or without limits?
This seems like a strawman. You could almost definitely ask/warn the user when they set the cap, or you could make the cap not apply to anything that is not trivially recoverable.
They have this with their cost explorers, alarms, budgets and more.
Realize that most customers are more focused on will their data be preserved.
Blowing out your entire EC2 / RDS / S3 / Glacier backup stack over a billing issue (or someone setting a cap up in the accounting department) makes no sense.
Are major customers really asking for this? Why risk it, why even build a tool that can blow out a customers setup so completely.
This is why I don't understand HN sometime saying AWS is "BS" etc. Does HN not thing AWS talks to their big customers to find out what they want?
I'm convinced folks who complain about "unexpected recurring small bills on AWS" have never taken the five minutes it takes to learn to using the billing console.
It's like asking why the knife you're using keeps cutting you when you put your finger on the sharp end. Learn to use your tools, and they won't surprise you. Learn to track your costs, and you won't have unexplainable recurring bills.
> five minutes it takes to learn to using the billing console.
I spent hour or two trying to find anything running and didn't find a damn thing. Yet Amazon decided to charge me a buck or two every month for "storage" so I just canceled whole account. I mean, if I can't find what I am paying for when not using it, how am I supposed to understand the bill when I actually have a dozen of instances?
That's a good comparison. Have you ever cut yourself with a knife, any time in your life? I certainly have. I've also had a £150 AWS charge I had to pay.
In both cases I learnt my lesson. I'm careful not to put my fingers where a knife can cut it, and to not put my bank details where AWS can charge them.
I know Google burns through user trust like it’s incense candles, and so people fear of losing their account access or features being deprecated without notice, but, I really think GCP is underrated. In GCP, it’s easy and best practice to group things into granular projects, and if you want to, you can shut down an entire project all at once. The billing story isn’t perfect, but it’s not bad either, and it’s not the only thing going for it.
I know I’m inviting replies airing grievances with GCP, and there certainly are many, but I’ve come to really like what it offers. Especially, GKE is really cool and at least to me feels very whole-assed as far as Kubernetes offerings go. Obviously that would make sense, but still, it really is nice to use.
It's the same in Azure, really. It forces you to setup a resource group for everything (can't have any resources outside a resource group). If you're trying something out - just put them all in same resource group, and at the end delete the resource group.
That billing model makes it very easy to find out what the operational cost of a particular product/service/team is costing the business and I think it's one of the advantages that GCP has, personally.
It took me 2 years to get AWS to stop billing me a few cents a month. I didn't try that hard but every month or so I'd get another bill for like 23 cents. I'd login and try to figure out what was still turned on.
AWS has been billing me a few cents per month for over a decade now. Eventually, the credit card I was using expired, and so every month for the past 7 years, they've sent me an email:
> Dear Amazon Web Services Customer,
> Your AWS Account is about to be suspended. There is still an outstanding payment problem with your account, as our previous communications did not lead to successful payment of your past due AWS charges.
> We were unable to charge your credit card for the amount of $0.26 for your use of AWS services during the month of Aug-2021. We will attempt to collect this amount again. Unless we are successful in collecting the balance of $0.26 in full by 09/30/2021, your Amazon Web Services account may be suspended or terminated.
The balance never increases, even though it always says the charges are from the previous month. It's as though they forget the balance each month because it's not worth collecting, then my account runs up another small charge, which they promptly forget.
And of course they have yet to terminate the account. I would gladly close it myself, but I'm locked out, and it's not really worth my time to figure it out how to get back in (plus they keep telling me they're going to terminate it, which is exactly what I want them to do!).
I got this too, and they did eventually terminating my account. I missed all the emails tho, but when I went to use AWS for some project I found out I was terminated and the only way for me to use AWS again was to create a new account (I emailed them, they can't un terminate me)
i have this happening on gcp and aws. i get a text from my credit card each month for like 10-25 cents. i’ve gone in multiple times all the consoles show $0 it’s impossible to track down. it’s so cheap i haven’t cared enough to dig more
I never use AWS or Azure privately. Everything I test I do on company or customer credit. You also more or less have to believe what they say you used in CPU time, storage is a bit easier to check.
But even with a simple webservice you quickly get to 50 - 100$ alone from testing and deployment.
I heavily recommend to rent a server for private use. They are always the cheaper option at fixed costs. Of course you don't have those fancy services... I only use a virtual server right now and pay $45 quarterly with domain. It still does have quite a lot of power though.
I'm a huge fan of Oracle's (yeah, I know right) free tier for especially this -- you can't accidentally use paid stuff without manually toggling an upgrade. It's such a relief to know that you can mess around with things and not be charged for it.
In Azure I put all my stuff in a new resource group and then when I'm done I just delete the entire resource group. This has worked well for me so far and I haven't had any surprise charges like I did on AWS and Digital Ocean.
I wanted to clear out my account so I deleted the VM but it didn't delete the associated static IP, so I got charged for the unused IP address that month. I didn't know the IP was still around until I got the bill. If this were in Azure I would have deleted the entire resource group and the IP would have gone along with it.
Just one thing: credit card. Once they have all details, they can do whatever they want with you. They can let you go - but that's only their good will.
Personally I'm in love with Hetzner cloud. It has less "features" (=proprietary complexity) than AWS/Azure, but I can understand everything perfectly and, most importantly, I have a guarantee I won't be charged over a limit.
I had a $7k bill from AWS due to a bug in a SaaS product. They wouldn’t refund it. I wouldn’t use them again after that.
I then tried to close down another account based on AWS organisations, and it was super complicated to get it all removed to stop billing to the extent I would have needed to hire an experienced consultant to do it rather than just click “close account.”
I'm glad they have made progress in that area. As others have pointed out it can sometimes be tough to track down sources of costs in AWS as well (especially if you accidentally started something in a zone you don't normally play in). I'm still gunshy though.
AWS have a setting in their billing that can fire off if you have exceeded a certain threshold which would be pretty much invaluable if you're running something that auto-scales.
>They keep reminding me I have $50 of test/dev credit on Azure through my Visual Studio subscription but it flat out frightens me to even try to use it.
The VS credits are hard capped. At least they were on my account (didn't even have a credit card loaded)
You hear about all of these horror stories about services from IaaS/PaaS/SaaS providers essentially being black holes for money with no way to actually set hard limits for billing, even after all of these years and can't help but to think about more reasonable alternatives.
Providers that just give you VPSes that you can run whatever containers on (or just host things the old fashioned way) and basically use whatever open source software that your project needs. More importantly, providers that give you predictable billing at a flat figure per month (or less, if your VPS isn't active all of the month) and simply slow down your network connection if the set limits are exceeded.
If you're just doing something to practice and haven't sold out to SaaSS (https://www.gnu.org/philosophy/who-does-that-server-really-s...), you probably really don't need to worry about scaling just yet - you can migrate over to AWS, GCP, Azure, or any other of the large providers at any time, by just running your containers on their scalable infrastructure, if there's even any point in doing that, since the smaller providers also can scale similarly in most cases.
Lastly, it just feels dangerous to give AWS, GCP, Azure or any other entity that's known to give people insane bills your personal details and personal credit/debit card details - what if they decide to block you because you can't pay and your complaint doesn't become popular on Reddit or HackerNews? It almost feels like setting up a shell company and using limited virtual credit cards would be more reasonable, same as people always say that you should have a separate Google account for your personal needs and anything in any professional capacity, so nothing gets blanket banned.
Sadly, there aren't many tutorials that start with: "Here's how you set up a company that's detached from your personal details, and here's how to easily make one credit card per vendor." Honestly, even the internal workings of companies like https://privacy.com/ are unclear to me.
For running personal projects that don’t need to scale instantly to demand you are 100% right that you should not go with cloud providers. Hell, most startups shouldn’t either.
> "AWS isn't perfect but at least you have half a chance of working out what is costing you money through the billing console."
Same thing happened to me on AWS. I was trying to get a Windows VM in the cloud to run some CAD software. Ended up blowing through $500 credit in a week and shut it down before it went through the credit.
Unexpected bills like that makes for a terrible experience to be honest...
I've had a $1.5/month charge on AWS that I tried repeatedly to track down but failed each time. Eventually I just let it run until I canceled my credit card.
The offending process costing me money didn't appear on the Azure console and I had no idea it was running, how to access it to stop it or even to know what was going on. When it turned up on my bill I nearly had a heart attack.
Thankfully they let me out of it once I pointed out I was getting billed for something I couldn't see. I appreciated that greatly but I've never gone back to Azure and the experience scarred me so much I don't think I ever will. This was about 4 years ago so more than likely they have sorted it out.
They keep reminding me I have $50 of test/dev credit on Azure through my Visual Studio subscription but it flat out frightens me to even try to use it.
AWS isn't perfect but at least you have half a chance of working out what is costing you money through the billing console.