I spotted Little Workshop when I saw https://equinox.space/ on Hacker News and noticed it was running on Netlify. Loved the fluidness, speed and art direction of a game running directly in the browser and working smoothly on my phone.
Immediately thought of them when we started thinking about a 5 million developer celebration and reached out. Love the result :)
We’re very grateful for the opportunity to create this experience! Huge thanks to you and the Netlify team for supporting innovative campaigns like this one.
I knew I recognized the feel of this project. It did quite well on the front page [0]. Hopefully you and others keep hiring them so we can keep enjoying their work!
Our support team has reached out to the user from the thread to let them know they're not getting charged for this.
It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn't come through in the initial support reply.
One additional feedback, for consideration: to me, your Pricing page[1] doesn’t make it sufficiently clear that the “Starter” plan may incur costs at all (let alone in this ballpark). It’s now more apparent when looking at it in hindsight, but you have to either read very carefully, or go to the separate “View Features” page to understand this.
“0$ to get started, then pay as you go” reads to me: “0$ to get started, and then you can order add-ons and extra features as you need them”, not “$0 to get started, but we may start charging you virtually unlimited amounts at any point without prior notice”.
When signing up for the “Starter” tier initially, I completely misunderstood this. I didn’t have to enter any credit card or invoice details, so I thought as long as you don’t have that info from me, you can’t and won’t bill anything.
How on earth could I, as a customer, be sure that netlify hadn't paid someone to DDOS me? If I were in charge of a business like that, I would have that thought constantly...
Why go through that effort when they could just lie about site usage and say you incurred a bunch of traffic? Or make fake site "hits" from localhost?
It's really the trade-off for using any cloud host. You are implicitly trusting the host, their monitoring tools, their billing system, and their customer support when things go wrong
If you charge them for the extra bandwidth usage, it is. Not saying it's morally right, but definitely something a shady business would do. There's nothing "conspiratorial" about that. You'd be surprised how many conflicts of interest Big Gov and Big Business find themselves in.
> 0$ to get started, then pay as you go” reads to me: “0$ to get started, and then you can order add-ons and extra features as you need them
I think I disagree with this, but maybe I'm misunderstanding you.
Pay as you go sounds strongly to me that you pay based on your actual usage, not that it's free except for add-ons. A pay as you go phone, for example, does not imply you need to buy a telephony add-on, an SMS add-on, etc.
PAYG phones, however, were always prepaid, so I think I would expect PAYG hosting to be similar. That said, if my site was publicly accessible without my prepayment, I think it would be clear that it works the way it apparently does.
It's potentially misleading, but I don't think it's intentionally dishonest.
The disagreement is on what "usage" means. I wouldn't assume that "usage" includes things that don't take any action on my part.
If I don't use my phone, for example, I wouldn't get any "usage". A phone pay-as-you-go plan would probably trigger similar outrage if they charged you potentially unlimited amounts for phone calls that hit your voicemail overnight.
Do you know how web hosting works? You pay for a service so other people can use it. Extending the phone analogy, it is like you set up a public phone that anyone can use, and you pay for every time someone uses it.
Your analogy break down because no part of buying and setting up a phone yourself is free. If some company offers to set up a public phone "for free" in your neighborhood, and charge you for "usage", you wouldn't expect to be charged if you don't place calls.
The "do you know how it works" is completely unnecessary and rude.
> It's potentially misleading, but I don't think it's intentionally dishonest.
That’s my interpretation as well.
The usage of the term “add-on” is not clear here in my opinion. On their main pricing page[1], Netlify currently lists “Additional bandwidth” as “Add-on”. To me, that sounds like “I can actively order additional bandwidth in case the included bandwidth isn’t enough.” Not: “Additional bandwidth is automatically allocated and charged for as it happens to occur.”
In addition to that, there is a big bold “$0” at the top of the “Starter” plan.
1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
Any cloud platform should have a spend-stop amount built in.
i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline".
Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you.
This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
This is something I really like about Nearly Free Speech.net. Their model is that you deposit funds up front, and they will deduct from those funds as you use services. It helps that they actually are nearly free so that a single $20 deposit can last for months or years in many cases.
It's bizarre to me that more services don't support billing this way, since there are tons of situations where I would much rather have a site or service go down than be hit with a surprise bill and have to depend on social media and magnanimous corporate PR.
Yes it’s nice like that. Specially for side projects on AWS that could go wrong on your personal credit card. Also I heard they forgave bills sometimes.
Amazon will, but they also gauge their discount in how many prevention and security measures from their 5 Pillars you follow in your environment.
You can do stuff like "disallow any of these instances to be used in your env", so if you never use graphics cards, disallow the whole class.
You can also set limits like "no more than 20x m5.4xlarge".
But again, AWS is the worst about no actual hard limits, cause each system generates bills. Ive also seen the hell of "hidden system AWS Billing doesnt have is still submitting billing and we dont know what it is". Again, AWS enables basically infinite liability.
Ive also discussed with C levels that "every engineer and dev with AWS logins have an unlimited credit card to of which you're on the hook for". Lets just say that 'heartburn' doesnt even begin to describe the terror on their faces.
When people have to read and implement "5 pillars" or "cloud adoption frameworks" before start using cloud, learning stuff like hosting with hetzner or self hosting start to seems like comparatively easy and simple.
Exactly. Cloud was sold as "simple", but in reality you MUST know about data center operations, security in an effective zero-trust environment, failover vs HA vs load balancing, and so many footguns.
The big advantage to "cloud" was that you can provision more resources in seconds. Existing data centers had terrible non-VM and non-good VM management software that provisioning more CPU, RAM, storage was a weeks or months long trial.
And you can still get a 100k bill for hosting a SINGLE text file, cause DDoS and shitty hosting providers who demand unlimited credit is a thing.
Frequently I'm looking up if there's a way to have a hard limit on AWS billing, and it seems like many other people have the same concern as well. I do understand that the massively distributed system hosting 100+ products each with ~10 things to bill for means you can't have each service going to the magic billing limiter service and be ask "can account X spend 0.000001 USD now?" * every request * every cloud tenant, etc, etc.
That said, I still think there should be an easy way to set a daily limit. Should I use the Budget service to do that? Cost Explorer? Billing Alarms? Is it possible to have them shed whatever's spending all the money?...
Again, I see the whole can of worms here: what if your service is jamming tons of data into S3 because of a bug? Or you actually started something that got popular and you have a gigantic Dynamo table? Stopping an EC2 instance is maybe an easy call, but deleting data is iffy.
AWS just feels like a minefield because I'm occasionally worried with all the products, I'll check a box when creating an instance or SG or whatever, and that'll (e.g.) trigger CloudWatch to read all my logs, but I have some crappy debug config for some app which will vomit out dozens of logs a second accidentally, and instead of just trashing `/var/log/` I get billed for millions of log events or something.
AWS doesn't have to check that for every request. They only have to eat the cost if you go over and use more before they shut you down. And the shut down should be in a way that they switch data to read only and give you a day to react before they delete.
They might even offer this as an insurance, so you pay a little more but can be sure you stay in budget.
A couple years ago I switched from a standard, contract-based US cell phone plan to a prepaid service it has felt so much more natural. Have a messed up my autopay and my phone just stop working in the middle of the day? Yep. But You just find some WIFI, pay, and its back on in minutes. I know exactly what my service costs: $35 dollars. No fees, taxes.
I stopped ~all autopays when (Boost Mobile?) deduced 200 instead of 20usd from my debit account one month in college. They refunded the difference relatively quickly, but I racked up 5 or 6 overcharge fees before realizing; ended up being a pita for an already broke 18 y/o to figure out.
I was a broke and stupid kid. Never use a debit as an autopay. But since then I like to track where each penny goes as it goes.
Why would one ever want to add a watch to a phone bill? ISPs are notoriously fleecing their costumers on every front, why would you ever go to them to get a loan?
Weird. The cheapest T-Mobile non-prepaid plan I see is $50 + taxes and fees, and that is without watch service. The cheapest T-Mobile prepaid I see is $30 + taxes and fees, and that also is without a watch.
I pay $20 for the phone plan, and $10 for the watch add-on. No fees or taxes Feels reasonable to me.
I think for this purpose you could use something like privacy.com, generate a virtual credit card, and set a monthly limit on it. This way you don't have to deposit any money in advance.
We did this at DigitalOcean for similar reasons, wasn't a feature that was commonly used. Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
What Netlify is doing here is really the best approach for both parties. And typically speaking a $104k bill would be hard to get paid up regardless if the customer's typical transaction balance was $5/mo and their credit card limit wouldn't be that high.
Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
> Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
So your suggestion is to issue a chargeback.. to get money back that should under the terms of whatever service you signed up for be owed?.
That seems like bordering on fraud tbh.
> Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
Legit concern and something I mentioned, I'm gonna guess there are broad two camps on that one - mine which is "I want a safety ripcord" and "whee, nice problem to have".
However since this entire conversation is around a guy who got a massive invoice because of a bill he wasn't expecting and couldn't have set such a limit I'm still gonna go with a "I want a way to constrain the financial downside - hell turn it off by default but give me the option".
Since broadly a lot of cloud stuff doesn't, I'll constrain it a different way.
So is the solution to have two thresholds?
Notify me urgently if the traffic exceeds 100$, giving me a chance to evaluate what's going on but shut it down at 1000$ if I don't act.
> Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
That has not been my experience. I've had to do a few chargebacks for services not rendered, and I've never won. I will submit my evidence, then the vendor will submit 100 pages of random emails, and then I will have my claim denied. Then I will appeal, will point out that they sent 100 random pages of email, and then they will reply with the same 100 pages of emails and I'll get denied again.
It seems that the vendors have found the hack for chargebacks -- just inundate the credit card company with so much data that they assume the vendor must be right.
It makes sense -- the vendors pay the credit card companies a lot more than I do. They'd rather keep them happy than me.
Plenty of morons just use the service and chargeback right before renewal so they got the service for free, some just chargeback instead of cancelling their plan or asking a refund.
I get hit with 15$ bill plus I lose all the money even if I provided the service.
Whatever email / data from my system I send is ignored and the scammer / moron gets his money back.
I'm sure that's how it works if the vendor is large enough.
If you are a small fish you don't stand a chance, which is why for my next project (where I'm supposed to charge a lot and spend a lot on behalf of the user - and I'll be royally screwed if I start getting chargebacks on 500$ of which I've already spent 450$) I'll just accept crypto payments.
That's strange. I think I've done about 10 chargebacks over 35 years. All but one I "won" with just an initial submission and waiting. One my card company came back to me for additional details before also siding with me.
Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
But that's on the user. The user shouldn't get upset in that scenario and has no right to. You're giving the control back to the user.
How about just fix the pricing formula to account for massive surges.
Instead of forcing user to set a low cost limit and missing a viral opportunity or the platform writing off the massive bill the customer can't afford... just put the billing mode into a reduced price mode or have some more nuanced configurations. Sometimes is just asking the question the right way. Instead of "max spend limit" or similar "If your site goes viral, how many requests do you want to serve before going offline? 1M=$20, 10M=$100, etc" at this point, I feel like bandwidth consumption is a bad metric for billing; just use requests/visits/actions and price for those.
This is not prescriptive just illustrative. The point is make a better pricing formula to account for these massively unexpected events. Couple it with an aggressive notification policy when this traffic event gets triggered. The user should know the traffic pattern has changed and a high traffic event is happening. They can login and change the configs and decide if they want to keep it going or not.
> But that's on the user. The user shouldn't get upset in that scenario and has no right to.
I agree. I also agree that when dealing with large numbers of people, there will be people who don't understand this and/or will actively try to social engineer their way out of their own decisions.
Setting customer expectations and meeting them successfully isn't easy.
The site user/admin saying "If this spend goes over $100, shut shit down" is called being a fiscally responsible adult.
The fact that most cloud operators don't have actual hard cutoffs to maintain financial responsibility is intentional. Azure does, but only for specific account types. If it's PAYG, you can't do it. The end result is if you do something "weird", or someone DDoS's you, you're liable.
With a hard limit, a DDoS just takes your site offline.
No, not really. Not really what attorneys do. There might be collections agencies interested in recovering the debt, but if it's some rando guy who doesn't have the money, even that is open to debate.
I mean I'm not familiar with every debt collection scenario under the sun but Internet randos seem to think this is a real thing where like a cloud/hosting company sends an army of lawyers to repo some guy's house and runs him into bankruptcy because of a traffic overage. I've never seen it work that way, what happens like with most business debts, is someone at the company negotiates with the debtor to try and get as much out of them as they can, and failing that, possibly refers it to a collections agency which does the same but plays a bit more hardball.
In the case here with Netlify even before it went viral they reduced the amount from $104K to $5K, no lawyers, collectors or repo men involved, and while I'd hate to be stuck with that $5K bill, I dunno, that does feel closer to the mark of something that maybe you should be on the hook for if you're responsible for 200 TB of bandwidth overage over 4 days? Is this so bad on the part of Netlify?
All that said I'll just add that I've never given my credit card to any sort of host/cloud who had terms where they could bill unlimited overage fees like this. Never will unless there's a cap. Not Netlify not AWS not nobody. That goes for my personal life as well as for the business I operate. The terms is the terms and the answer is to not use these services unless you can afford them imho.
> I'd hate to be stuck with that $5K bill, I dunno, that does feel closer to the mark of something that maybe you should be on the hook for if you're responsible for 200 TB of bandwidth overage over 4 days?
The responsibility part is the tricky part of the equation.
If someone hits your site with a DDoS attack, are you responsible? There's literally nothing[0] you can do as a customer of a cloud provider here because anything you can do is limited to the servers and services you're given access to. For example even if I had access to billions of requests and built an anti-DDoS tool it would still need to run within the cloud provider's provisioned server which means I'd be on the hook for all traffic costs because it's something running in my account.
That doesn't seem reasonable to me as a customer. It means a cloud hosting provider can put an extreme financial burden on a customer and make a killing in profits because of the markup they charge on bandwidth. The incentives are terribly misaligned.
[0]: I mean you can sign up for DDoS protection through a 3rd party company but in this case I'm talking about taking actions within your hosting provider.
All fair points but do they apply to the Netlify situation? As I understand it they generally won't hold you liable for resource usage generated by a DDoS, the guy on Reddit said this was a DDoS, the Netlify CEO said the traffic "didn't match attack patterns..." I think telling a free tier customer that they owe $104K was a pretty stupid PR move either way, but we don't really have enough info to say whether this was a DDoS or not
> As I understand it they generally won't hold you liable for resource usage generated by a DDoS
From personal experience as a customer of a cloud provider (not with Netlify btw), usually cloud providers who profit from bandwidth costs will write their TOS in such a way where almost nothing qualifies as a DDoS attack unless it's truly a distributed and targeted large scale attack specifically on your site.
A random person on the internet who spins up a few VPSs around the world and slams your site with looped curl requests won't count as a DDoS attack even though from your perspective that will result in a massive bill increase due to bandwidth costs.
In other words, I'm not surprised "didn't match attack patterns" was used. I'm guessing that will be the case most of the time.
Traffic doesn't cost money. Bandwidth costs money. Unused bandwidth doesn't cost less than used bandwidth. So, no, you shouldn't pay so much for something that doesn't cost them any money?
Mostly false. Either transit is billed on a 95th%ile basis (so...more money for more traffic), or if it is flat/netted, you're still paying for the capex for the switch ports (fatter connection to support more traffic means more $$$ for the gear to support it).
At Netlify scale, it'll be the latter. And the appropriate thing to do with free-tier traffic is set lower QoS so it doesn't interfere with paid traffic. This, it doesn't have cost.
You can say "the sum total of free-tier traffic requires X additional connections" but the sudden burst of DoS traffic did not raise marginal costs (besides an inconsequential usage of electricity).
They profit when their customers can't hard limit their spend and end up racking up large bills by accident, or for reasons outside of their control.
By the way, your comment was flagged which seemed odd, so I looked at your profile and it seems like all of your comments are being (automatically?) flagged and don't appear on HN by default. You might want to talk to the HN staff about that.
1. This doesn't seem like a rational thing to do. A trillion dollar business built on people making mistakes and actually running up a bill? Which exec is getting a bonus when little Johnny gets hit with a $1000 charge on his CS101 project? Doesn't seem likely.
2. To me, it's more likely they don't have one because there are edge cases to consider that make "hard limits" difficult to implement. What is AWS supposed to do when you hit the limit? Is it a hard limit? Ok, so when I hit my budget, all my s3 buckets get deleted and all my EBS drives get dropped? Do all my code deployments just get deleted? Do you "bless" certain services so that they continue to charge the user even after the hard limit? How is all of this communicated?
You can set an alarm in AWS today and the user can decide what to shut off. If you really need to, you can create a script that can hard nuke your account once a limit is reached; but I don't see why AWS should nuke your account for you.
1. Even if you assume perfectly good intentions, they're not incentivized to fix the problem because they're not on the hook for the bill, but stand to profit from it. Their margins are eye-watering, so sending out a $100k bill for a service which cost them no more than $5 to provide doesn't have a downside (other than bad PR, which they seem to be blind to).
If providing this bandwidth cost them $50k rather than $5, I would bet you my entire life savings that they would QUICKLY find a way to add hard limits to their service, no matter what technical challenges they're quoting now.
2. This is probably a 95/5 problem. The vast majority of these sorts of cases that I've seen and read about are caused by increased traffic, which hits the customer either with extortionate egress fees or unintended compute scaling behavior (FaaS/VMs).
Storage is trickier, but you could stop accepting writes or reads after passing some hard limit.
This leaves some edge cases, sure, but it should handle the vast majority of unexpected bills without any destructive actions.
> You can set an alarm in AWS today and the user can decide what to shut off.
That shifts responsibility back on the customer, which is exactly the problem to begin with. I need assurances that I won't be billed more than $X in any given day, or month and this solution doesn't provide that. Maybe their API down, maybe it's serving stale data, or maybe my automation fails for some reason.
> How is all of this communicated?
Hard monthly limits:
Egress: [ $ 10 ] - When exceeded, all outbound traffic is limited to [ 0 Mbps ].
Compute: [ $ 10 ] - When exceeded, scale all compute instances to [ 0 ]
Data Storage: [ 1 TB ] - When exceeded, all writes are rejected
Reads Ops: [ $ 1 ] - When exceeded, all reads ops are rejected
It's really not that hard. Offer these limits on a per-project, or even per-resource level, that would naturally allow you to limit the blast radius. A personal website that has gone viral would likely want to have different limits than an email server under the same account, for example.
>If providing this bandwidth cost them $50k rather than $5,
No, I believe you are assuming that proving a "hard limit" feature is cheaper than just refunding people from time to time. If you consider the all the products AWS has to offer, all the different ways they are billed, then figuring out what do to for each product once that limit is reached, and potentially doing a destructive action, and then all the code and testing on top of that - it seems far easier to add a human in the loop to just refund people on a case by case basis. It's a ton of complexity for likely a rare issue, and if someone really needs it they can build one themselves and choose how to handle what needs to be done once the limit is reached.
Building all this out just so that some college kid isn't charged a bill they could obviously never pay likely isn't a good use of resources. It's not likely AWS has a reputation of being greedy either, if you explain the situation they refund you. If they are truly doing this to make money they have remarkably poor execution.
>It's really not that hard.
Storage pricing isn't done up front. You pay per gigabyte-month. If your "hard limit" kicks in, then what does Amazon do with the data you are no longer paying to store? Does Amazon drop your entire database once the credit limit is reached? There are plenty of Amazon services like this. Consider secret manager. You are charged 40 cents/month per secret. You have 100 secrets, so $40/mo, but you set your hard limit to $20. The middle of the month rolls around, what does Amazon do? Do they just drop all your encryption keys?
There are 100s of AWS services and many where you can't just apply a sensible rejection policy once you hit that hard limit without doing something the user cannot recover from. It's only not hard because you aren't actually thinking about the matrix of AWS products that exist and how they are billed.
> There are 100s of AWS services and many where you can't just apply a sensible rejection policy once you hit that hard limit without doing something the user cannot recover from. It's only not hard because you aren't actually thinking about the matrix of AWS products that exist and how they are billed.
I don't think you understand what a hard limit is. When I set a hard limit, that means that under no circumstance should I be billed more than this amount. From this you should be able to deduce that the platform needs to stop you from ever entering into a situation where they could end up having to decide between nuking your data, or overcharging you.
This means that for all data storage scenarios, they should apply limits to projected monthly usage rather than actual monthly usage.
If 1 GB of storage costs $1 per month, and my monthly limit is set to $10, then they should not allow me store more than 10GB of data, rejecting writes as needed. This design guarantees that my hard limit won't be exceeded and the data is never at risk.
Any service which doesn't persist data can apply limits based on actual usage, and safely shut them down when the limit is exceeded.
> Does Amazon drop your entire database once the credit limit is reached?
No, they should refuse further writes when your database reaches a size that would exceed $10 a month. That's a safe behavior.
> Consider secret manager. You are charged 40 cents/month per secret. You have 100 secrets, so $40/mo, but you set your hard limit to $20.
If you set your hard limit to $20, then Amazon should refuse to create more than 50 secrets. That's a safe behavior which limits your spend to at most $20.
> Building all this out just so that some college kid isn't charged a bill they could obviously never pay likely isn't a good use of resources. It's not likely AWS has a reputation of being greedy either, if you explain the situation they refund you. If they are truly doing this to make money they have remarkably poor execution.
This is just a bad faith argument, everyone from broke college students to medium sized businesses has complained about this at one point or another. Relying on the good will of a megacorporation to forgive a surprise $100k bill is nuts and if you scroll through these comments you'll find some that claim that AWS is no longer forgiving these bills like they used to.
I'm not claiming that this solution is perfect, or that it would cover every failure mode, but it sure beats the status quo and would solve the vast majority of surprise bills that result from unexpected traffic.
>This means that for all data storage scenarios, they should apply limits to projected monthly usage rather than actual monthly usage.
>No, they should refuse further writes when your database reaches a size that would exceed $10 a month. That's a safe behavior.
I think we have gotten to the point where we've lost the plot here. You previously stated "It's not that hard". We are doing projection forecasting! And the storage layer for these range of services (S3, MySQL, Postgres, MSSQL, Kafka, Keyspaces, DynamoDB, probably 20 more im missing.) must now either call out to some financial projections service and reject writes for billing reasons. On top of that if you have a hard limit you are locked out of dynamic scaling. Need to 100 nodes for an hour? Nope, you now must disable billing limits because Amazon assumes you will use every resource for the entire month. Surely someone wont disable billing limits for an hour and forget to re-enable them, right?
>but it sure beats the status quo
I'm not sure handicapping every service for some barely functional billing limit beats the status quo. It's certainly more expensive for amazon in terms of engineering load but with so many footguns I don't see how it ends up anyway other than Amazon having this feature and still having to do service refunds because billing limits were disabled due to awkward limitations.
>This is just a bad faith argument, everyone from broke college students to medium sized businesses has complained about this at one point or another. Relying on the good will of a megacorporation to forgive a surprise $100k bill is nuts and if you scroll through these comments you'll find some that claim that AWS is no longer forgiving these bills like they used to.
Medium sized businesses have had their bills forgiven for things like cryptomining for example. My point is billing limits don't really work for AWS, alerts is the best you will get because AWS doing destructive actions for you or assuming your usage patterns is worse. It's not an easy feature and to imply they keep the status quo solely because Bezos needs another yacht is reductive of the problem.
> I think we have gotten to the point where we've lost the plot here. You previously stated "It's not that hard". We are doing projection forecasting!
It's not a complex "forecast". It's simply the monthly rate as I've illustrated with numerous examples.
If my limit is $10, then don't let me occupy more storage than that. It's a 1:1 conversation between gigabytes and how much they cost on a monthly basis.
Do you have ties to cloud providers you'd like to disclose?
It's a very simple concept that you're refusing to understand, which usually happens when your job depends on it.
> If you set your hard limit to $20, then Amazon should refuse to create more than 50 secrets. That's a safe behavior which limits your spend to at most $20.
I have 100 secrets and no hard limit. Now I set the hard limit to $20. What does AWS do?
That sort of adversarial weaponization of credit cards is really a US thing. In most of the world, payment is not relevant to the question of liability: you create a debt, whether or not you can debit a card. So, looks like DigitalOcean is another name to avoid.
Also: what's common? At the scale these companies so desire, small percentages are still thousands of users. Supporting them with what's ultimately a fairly trivial option shouldn't be seen as such a hassle.
Partially agree, yes the liability is independent of the payment. But in practice the one who has the money has the benefit of the status quo. The other party without money has to actively take legal action which is costly enough that it's often not pursued.
Yes and no, collection costs significant proportions of the outstanding debt, and if you object to the collection of the debt saying it's not justified the business ultimately has to sue. Collector can't do everything.
2) I understand your opinion that you prefer chargebacks but I disagree with it.
The very reason I stay with Hetzner is that I know in advance what my bills will be for the whole year. Heck, I even charge my account in advance so that I don't worry about any charges!
> usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down
Of course. And that’s why any limit against a dynamic variable should also have alerts linked to it
Send an alert to the user when traffic starts spiking, especially if a simple projection shows it’s going to go over their limit
Then the user is aware, hopefully with enough time to lift the limit if needed
That's a level of responsiveness that doesn't exist for the vast majority of organizations.
If your customer is aware enough to notice they are being hit with a DOS or legit traffic while it is happening, then great! They can respond, and if needed, engage proserve to get support for scaling or defense depending on needs.
If your customer is not alert enough, then their site is offline, and they won't hear about it until their customers are screaming at them, which will result in a P1 ticket to look at a vendor who won't turn them off during an unexpected peak.
It's a catch 22, and if you have to choose between:
a) a PR hit because you have to go on a forum and post about waiving the fee, or
b) a PR hit because someone posted a blog post about how you killed their site during a moment of critical growth
any reasonable business will choose A every time because A is far more supportive of customer growth and has drastically better optics. Anyone who thinks A is worse is probably too inexperienced to have an opinion.
> Anyone who thinks A is worse is probably too inexperienced to have an opinion.
I'd say the same about those who only think in absolutes.
That's a false dilemma. You can give your customers a choice in these matters with an optional hard limit, which I realize seems like a rather extreme idea these days.
Someone running a personal project will likely opt to have a low hard limit, say $10, while businesses will more likely opt in for alerts without, or with very high hard limits.
> What Netlify is doing here is really the best approach for both parties.
Sort of, but the approach to the approach isn't great. If you're going to void charges from DDOS traffic anyway, you might as well make that explicit policy, rather than doing it after the fact in a way that seems discretionary.
The Reddit thread is full of people who are saying that they intend to pull their static content off of Netlify and move it to Cloudflare Pages, which has no overages on the free tier in the first place. I can tell you that I personally chose Cloudflare Pages to set up a new static site a couple of weeks ago, and Netlify wasn't even in the running.
If Netlify's free tier is important to their customer acquisition strategy, then they really need to retool how they're offering it, or Cloudflare is going to eat their lunch.
> I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
I still prefer this too. Kinda funny how server resource limitations became a feature and not a bug when it was one of the problems the cloud sought to overcome
Cloud is somewhat managed though so charging a premium makes some sense. The blank check factor of how they price their services is a major risk IMO. Also a turn off how every single bit of functionality becomes productized with its own pricing model.
Yes, any price that's not hard capped is unacceptable. One reason why I quit Amazon cloud after the trial year. No because they were too expensive, but there was no way to guarantee they wouldn't charge me more than planned...
If a cloud platform offers such a limit, but the user fails to set it up, then uses $100,000 of bandwidth, is the platform then justified in NOT forgiving the bill?
If forgiving bills for this kind of a thing is a standard practice, how come this was the customer support's first reaction:
>We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225.
I will put there the other obvious offender: Vercel . Not sure about bandwidth, but dark patterns, keeping serious RBAC procedures we'll hidden until asking a fortune even for startups, to provide not things like SSO, just reasonable RBAC.
With all that money they then can finance the free tier until they get too far and become platform locked-in.
Surge.sh Im not sure. But shows all the sign of some greedy acquisition, regular long outages , as if I have been sitting as a free tier for too long, quick nudge to pay. For barely accessed sites even behind CDNs, steep. I even worry they one day just wipe all my buckets (they did for a few already) and support would recommend me to be a "normal" paying user .
Nothing is free. And nothing too good to be true is true .
His claims are directly contradicted by his employee's actions. When asked about this, he provides no clarification. I just don't understand why you'd consider him even remotely believable.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
This isn't what you said in your first post, you said:
> It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
So forgiving "lots and lots" doesn't move the needle. Do you or do you not forgive _all_ such cases where your DDOS protection doesn't take down the site? What was your employee referring to when saying that the usual discount is 20%? Are you saying that you _never_ discount 20% and instead always discount 100% i.e. "forgive"?
1. Forgiven many, is Netlify forgiving all obvious anomalies? Is the question, which if so but you said many so it is a no, it would make you reconsider the next point
2. Favoring keeping people site up ? Would you go as far as keeping them up if they stopped paying for the meter? If not you simply should not let that meter go overboard.
Hey I'm a taxi driver. Hailer fell asleep on the back, so I kept driving all night, once he woke up I dropped him to his place and asked for my monthly wage. I "forgive" many, but just a few are juicy income so I adopted the policy to never wake any customer up. If people ask I say it would be impolite, principles prime.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
Sequence of events doesn't support this answer:
1. User gets charged 100k
2. User complains to support
3. User receives discount to 20k, then 5k. Support states policy is normally 20k
4. User discloses to the world. Goes viral.
5. Invoice is forgiven
While you might forgive "lots and lots", fact is that you still presented the invoice to a free tier customer, and when they complained you gave them a discount, but still charge. Only when it went viral did you forgive it.
Quite... It does seem that either the story we're getting isn't completely accurate or the support people who handled this need a little reminder of what's supposed to happen.
I'm a paranoid person by nature so "It's free... just... give us your card details" is always suspicious.
Do the changes you are working on that will cause "the default behavior to never let free sites incur overages" involve providing users with spending limit controls?
Solving this only for the free site use case doesn't address the core problem that people are bringing up about a lack of spending limit controls.
I wouldn't think it's a binding policy at all, because the billing procedure (automatic full bill, manually discounted bill, etc.) would follow it if it were. More of a procedure.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
No offence, but this sounds like "trust me bro" billing and it is not good enough. Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially.
> 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
I hope you understand that chance someone who used to pay you $20 / month unlikely want to ever get $10,000 bill. Yeah people might dislike that their website went down due high traffic, but it's not gonna bring this much negative PR as incidents like this. There should be some sanity check at least.
2. is obviously what should have always been the case, but it's good news to hear you've now gotten there. Every single hobbyist website would always choose downtime over a hundred thousand dollar charge.
With a properly configured nginx, you can easily serve 10's of thousands of requests a second on vserver type hardware. Netlify just offers these build pipeline kind of static site with cms UI.
But this is a good reminder why my gut feeling always made me avoid these overengineered solutions.
It wasn't the engineering that did it, it was the egress charges. But I think the IaaS tend to charge more for this than the clouds, who in turn charge more than the "servers for rent" people.
They aren't engineered, they subsidize (more) enginnering effort , and (are meant to) cost less as a result.
They do. But of course maximizing profit is the sole true prerogative of capitalist enterprises. And the market is not totally competitive. So yes your intuition was correct, to be cautious against over enginnered pricing to get y'a.
I mean those companies cater to hobbyist. Then ...
Render seems more fair-play. Until a change of mgt occurs of course.
You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries
This seems like a really good idea to me. Or at least cap overages at a specified amount, like 2x the free level (a $55 surprise bill is a totally different universe from a $100,000 surprise bill, obviously).
Honestly, this terrifies me---I run a bunch of different sites off netlify, and I would have never imagined that a site could jump from 0 to six figures of bills a month without something hitting a tripwire somewhere and cutting it off or at least communicating with the account owner. At least users should have the capacity to self-impose bandwidth caps to prevent this sort of thing.
Thank God for social media that the user was able to get attention about this on Reddit which he was then advised there to post this on HN. It must have been stressful to see a six-figure bill and then get told that that, no worries, you’d ‘only’ be charged $5k instead for a static site. It’s just ridiculous to me to be sent a 6-figure bill in the first place.
I hope this is not one of the cases that get simply forgotten and in a week or two their beginner unfriendly platform gets recommended again without a second thought.
With models like this and AWS people will get afraid of success
I mean, social media is pretty much an inevitability once mobile phones/internet became mainstream. Just like the invention of the gun and gunpowder, I think we are still debating if it was good for society right to this day.
You don't see VPS providers like Vultr forgiving bills like this, nor do they make the news. Granted they are not the same scope as Netlify, but still.
if only i had $1 for every time for every time someone asked this exact question on HN. yes, we all get it: easy question is askable and not answerable. you want a gold star?
I’ve been a netlify user since 2017 and I just deleted all my sites. I can’t risk receiving a $100k bill for toy projects. Your “current policy” is not good enough.
Same, as it stands you the user are legally liable for the full bill unless netlify graciously forgive it.
Even in op's case, they didn't (still charging 5k!).
If there was an option to cap billing, or at least some legally binding limit on liability, then I can countenance using netlify.
Until then, it's just not feasible nor worth the risk.
the fact that once it arrives to the limits does not display an error page.
At this point I honestly do not care about they changing their policy, they should have thought that a normal person receiving a 100000$ bill on a free tier shall not been at all on the table in any circumstance, even if they forgive the bill, nobody needs to stress out like that.
Same. I will (almost certainly) never incur a $104k bill, but switching to Cloudfare Pages looks free and I don't want to depend on unwritten policies of goodwill to mitigate the potential risk.
Same here. Will I ever get a level of traffic that would cause this problem? Extremely doubtful. Is it worth the risk when Cloudflare Pages is a similarly easy offering, and took 5 minutes to switch to? Hell no.
> What’s a good, simple alternative for a VueJS app?
I'm not sure about VueJS specifically, but I run everything I can off a $6/m digital ocean droplet (static sites, web apps, git repos, RDBMS, some other custom apps I've written) and it hasn't broken a sweat yet[1].
My understanding used to be that requests will be dropped if my virtual server can't handle it, and I'll have to transfer 10,000TB to get to a $100,000 bill.
In practice, my server will not physically handle the load to serve more than maybe $1000 of data a month; it will fall over before that.
In summary, using a VPS is sorta like an instant hard cap.
[1] Until I tried using Jenkins. Which crashed constantly because apparently 512GB of RAM is too little for what it does. I'm now in the process of writing my own little CD tool that isn't going to go over 30MB of RAM just to run my deployment scripts.
Having a personal VPS is the way. I run a SvelteJS app as well as my personal website, blog and a couple other services on a $6 droplet and it runs great.
I agree with both of your philosophies and also run a VPS. However, lots of people would have no idea how to manage a server from scratch or to install a web server, even a static one. Netlify really is pretty amazingly simple for what it offers, which is a lot. And even among those who think they can run a server, many probably have wide open security holes they are oblivious to.
> And even among those who think they can run a server, many probably have wide open security holes they are oblivious to.
This is the big thing, but I also think that modern out-the-box OSes are pretty damn secure these days.
IOW, the amount of knowledge and time needed to maintain my single VPS is a lot less than the knowledge and time you will need to manage your costs using multiple hosted SaaS suppliers for static hosting, web-app hosting, database hosting, repo hosting, etc.
Cloudflare pages is pretty much drop in for netlify. And it has unlimited bandwidth for free (at least in theory. Guess they might call you if your site does 1 petabyte per hour)
The only "fix" here is to act like Hetzner and null route upon DDoS, price cap the thing, or offer unlimited bandwidth on the free tier like e.g. Cloudflare Pages.
Uncapped but paid is a recipe for disaster and you'll always be subject to the will of the support staff when something happens. If they can grasp to a straw leading to suspicions that it's not in fact a DDoS attack, you can for example be sure they'll do just that. Just no.
With a 48 core Epyc or 80 core arm server, one really shouldnt need much more for a middling project. There are enterprises who run entire services on such hardware.
How does price caps work on Hetzner? I never managed to figure that out from reading their price lists. It looks to me like they charge for each TB, and the only thing I can see is that you can set an email alert to go off when close to some threshold?
Traffic limits change depending on our products:
See https://docs.hetzner.com/robot/general/traffic/
1) dedicated servers: unlimited
2) cloud servers: changes based on package
3) colocation: changes based on product
4) managed servers: unlimited
5) managed vServers: 20 TB
6) Storage Boxes: unlimited
We only calculate outgoing traffic. We do not count incoming and internal traffic. --Katie
Thanks, I have 3 sites I need to get off Netlify and I’ve spent the morning reading about Cloudflare Pages.
My only hesitation is the 20k file limit; one of our sites is a large Gatsby site that generates certain content programmatically, sometimes using a series of YAML files as the source. Pretty sure we’re over 20k pages by themselves. Been meaning to move this one to Next.js sometime but that’s not on our roadmap anytime soon.
I've run all of my hobby projects, including personal web pages, a Wordpress site that serves a local club, a small single-JS web app, and E-mail hosting for my family and a few other domains, on a single $5/mo VPS, and have never received a bill higher than $5 for the past, I don't know... 15 years.
If your web site makes you money in proportion to the amount of views or bandwidth you use, by all means, go with a provider that increases your costs when your traffic rises. But if your web site does not make you money, why not host it somewhere for a flat rate?
I don't get that either. I earn all my money from my websites and serve > 1mio requests daily all from a single Vultr instance for less than $150 including backups and more traffic than I ever need. With monitoring, CDN, DNS, ddos protection, and whatever I pay about $200 a month, no surprised and a lot of room to grow or get HN hugged, or whatever.
I've been on a $25 instance for years for all my sites, which worked as well.
Did exactly the same, moving everything over to Cloudflare took me less than 15 minutes. “We’ll forgive those cases, pinky swear” is not a valid excuse when putting (even opt-in) hard limits in place is technically viable.
"Current policy?" So, you will retain a right to change such fees when you feel like it.
This is a serious matter. We are building a new site for our company with Netlify, but we can't open ourselves to this predatory practice. And even if you do not mean to be predatory, even the option of such is enough.
If not resolved with a clean, legally binding promise, our company (and probably quite a few others) must move our business to Cloudflare, Amazon, or some other competitor of yours.
The paid tier (like $19/mo/user) has the same vulnerability. Overages are charged at the same price per GB as the free tier and they could still be charged $100k for the exact same thing.
Why are you asking this question here? Any actual company would have reviewed all the legal documents prior to choosing a provider. The promises you seek are the exact reason "enterprise-grade" providers can (and do) charge so much.
« Apologies that this didn't come through in the initial support reply. »
"Didn't come through" doesn't actually match the user's report of having support explicitly offering 20% and then 5% payment. It sounds like maybe you have a training problem? That seems like one of the important points to speak to.
> It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
That doesn't square with the 5% fee on the original $104k that your company told the OP to then pay.
> It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Well, giving the option to users to plan ahead would be best, no? Like a setting to choose whether they want a potentially unlimited bill versus downtime.
Instead of that, you are choosing to stress and make people scared/anxious/homeless even (if they don't think of raising the issue on HN).
Seriously, this is not rocket science. This must have been discussed before in your company, and someone actually made this decision to stress people about such bills.
Frankly the only reason I can even come up with that Netlify wouldn't have such controls in place is exactly if they do _not_ simply forgive these sorts of jumps in costs (as the CEO here seems to be claiming). I'm pretty sure if they'd be left holding the bag, they'd manage to find some way to cut off these kinds of jumps in usage.
That would potentially make this situation much, much worse (in the US tax system...YMMV). If Netfly forgives a business debt and reports it to the IRS as uncollectable so they can write it off, the IRS can consider all or part of the forgiven debt as income to the person who is forgiven (there are lots of details, IANATA/IANYTA, YMMV). I don't want a blindside 100k bill from my hobby site, but I sure as phuck don't want the IRS thinking I made an extra 100k of taxable income. I might be able to shame Netlify into forgetting about it, but the IRS is not usually so easy to deal with.
Not a dodge, (tax) accounting doesn't work like that. You can create accounts receivable of 100k and then write them off, that leaves you in exactly the same state as if you had just not entered the whole thing into the books at all. No benefit to "writing off".
Well, note that they're only talking about giving refunds if there's an attack and they miss it. Doesn't mean they'll give a refund if you get $100K worth of real user traffic.
> It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
The legitimate mistake sounds to be on _your_ side if anything. You failed to match the attack pattern after all.
> Apologies that this didn't come through in the initial support reply.
The support email said you normally discount the attacks to 20%, but in this case it would be discounted to 5%. Are you here publicly claiming that your policy is to in fact to forgive (i.e. discount 100%) these bills? Was the support reply totally incorrect in claiming that you normally discount the attacks to 20% or are you lying when saying that your policy is to forgive the bills? You might want to clarify your position here.
To be fair, these days, things can become viral literally overnight.
That said, instead of depending on unreliable heuristics, they should just allow an option to change the behavior. The "current policy" to charge small sites on the free tier thousands of dollars instead of just throttling/shutting down the traffic is really predatory.
I'm not trying to justify netlify, because it's pretty obvious all cloud providers have vile intentions here (even though they pretend otherwise): it's obvious that total majority of use cases for such platforms are toy or small-scale projects, and literally everyone in this market would prefer their website being shut down, rather than getting $100K (or even $5K) bill, BUT…
1 million downloads (= visits) is nothing for "going viral overnight"
Anyone exceeding their plan with a factor of 10 or hell, let's make it a 100, almost certainly didn't anticipate it and thus isn't prepared for the kind of bill that apparently comes with it (or even knows that there would be a bill). On top of that, there currently is no way to state such rules up front! Moverover, according to their own explanation, it was almost certainly not organic traffic!
I wager the vast majority of people in the free tier would gladly cap their traffic at the (generous!) bandwidth offered by Netlify. Even to the majority in paid tiers, 100k bills where there previously was none must be unwanted and unintended.
I understand that you need to pay bills, but auto-billing over the bandwidth budget just isn't OK, or at least not unless the user specifically configures that that's OK. I for sure didn't understand your bandwidth tiers that way.
You can avoid this sort of bad press and disgruntled users and your support cost by just giving users the option to shut down the site once the bandwidth budget is up.
Lol this deescalated pretty quickly, went from $104K to $20K to $5K to $0
Which basically means you almost scammed the customer for $5K or $20K. Super negative practices. I for one could never trust a company operating in that manner. It would be much more honest to say "unlimited bandwidth" and set a hard-limit for maximum budget, then people know they won't be charged, than to go through all this crap and then pretend you're doing a favor to the customer (you're not). If I'm normally spending $10/month any idiot out there would know for sure that I'm not going to spend $104K instantly. That's a very basic filter to have. But you don't place such filters because obviously you're working on the principle to scam people many thousands of $ if they fall for that. Heck, for all we know you might send that amount of traffic to your customer and the try to scam them and if it doesn't work then pretend you're doing them a favor.
This is in the FAQ: https://news.ycombinator.com/newsfaq.html. See "How are stories ranked?" and "Why is A ranked below B even though A has more points and is newer?"
About this specific case: it set off the flamewar detector (a.k.a. the overheated discussion detector) and also got moderation downweights. We sometimes turn off that penalty, but I don't think we'd do so in a case like this, because HN gets so many posts of this nature. They flare up with Big Drama that is sensational for a while but not particularly interesting, and therefore not really what the site is for.
In fact HN gets so many posts of this type that it has become a joke, and not only that but a cliché, so much so that the top comment of the Reddit thread repeats it [1]. That's about as repetitive as anything gets. The basic idea of HN is to gratify intellectual curiosity [2] and avoid repetition [3].
I read this whole thread before the CEO posted and after, and neither time thought any of the comments were out of line or even that the general mood was any more heated than any other random HN thread. People are politely asking pertinent questions.
And I think once the CEO makes a statement which contradicts the company's support response, that becomes very interesting. Particularly to anybody that uses their service. I'm certainly not finding the conversation very repetetive or cliche.
You're welcome to disagree, of course. My main concern is to explain what the principles are. I'm not saying we apply them perfectly—sometimes we make bad calls.
I can tell you pretty much for certain though, that we'd hear many more complaints if a Reddit thread about a customer support shitstorm stayed on HN's front page for very long.
Btw, the Customer Support Fuckup category is one of several $X where HN has become known as the place for $X, but only because HN is not actually for $X. Another example is the Site Is Down category—people often come to HN to find out what's going on when some $Site or other is having an outage. But just as HN itself isn't a site monitoring platform, it's also not a customer-support-of-last-resort platform.
If the community feels like this customer support fuckup is altogether more interesting, I'd consider reversing the call, but again, my gut feeling is that we'd get even more complaints that way.
I agree completely with the underlying principles, I just think once the CEO has commented and stirred up some interesting discussion that's relevant to a large segment of your userbase, the thread doesn't really belong to some generic "customer service shitstorms" category anymore.
I learnt more about Netlify, Vercel etc and how they operate from this thread from the last 100 "customer service" threads combined. I learned about Cloudflare's offerings, and a bit about Hetzner. And it was all very interesting.
You said you sometimes turn off those penalties, I think this thread would be a good candidate.
Thanks dang. This was one of the most important posts on HN for me since inception, and prompted my own immediate migration away from Netlify hosting[1]. I would hazard a guess that there are many Netlify users on HN, and becoming aware of the unlimited billing exposure (as well as Netlify's official policy (such as it is)) makes for important reading.
Something that makes me feel uneasy about the fact that the post gets hidden is that this strongly benefits Netlify. It seemed like lots of people moved off Netlify after reading the post.
I'm not suggesting that HN actively took an action in Netlify's favor, but the potential is there. Is the algorithm for flame war detection open source? Or do we essentially need to trust you that there was no interference from Netlify? (I do trust you but others might not).
We didn't have any private contact with anyone about this post other than a couple users emailing to ask why it wasn't on the front page anymore. Nor were we thinking about who would or wouldn't benefit—that never crossed my mind. I was just thinking about standard HN moderation practice.
I don't know how to get every user to trust us. All we can do is answer questions when asked. That seems to be enough to satisfy most of the community most of the time, and it's probably not possible to do a lot better than that, much as I would like to.
I'm not a fan of opaque ranking algorithms either. Just use this, the only thing it has is a 500 point threshhold and surprise, that works perfectly fine to determine what is currently trending in the community
Heck, at that point, why not "send some traffic" to your customer? It's not like they have any way of verifying its source. Hmm... why even send traffic at all? Just add a multiplier to their metrics!
This is very weird take. I'm struggling to understand why this is incident as a reflection of "super negative practices" or is somehow a "scam". The CEO came here and publicly apologized for the mistake and mis-communication, and the issue is resolved for the user with no charges. What am I missing?
What price would the dude have to pay if he didn't publish it? How often does this happen and why is there no protection against charging free customers 100k out of the blue. Why charge it and shock the customer if practice is to waive it? The CEOs response kinda just made the situation worse.
Yeah, I don't buy this conspiracy theory. The reason why they charge it could be as simple as they calculated the bandwidth usage following a ddos attack. It amounted to 104k worth of bandwidth usage. There system is not sophisticated enough to recognize it was a mistake due to attack on their site. Thus a manual intervention was needed, and now it's resolved.
Right, but there's still a huge contradiction here. The support email says it's standard practice to charge 20%. Now the CEO's comment says it's standard practice to waive it entirely. So which one is true?
It's only a weird take if you don't have any common sense. It's super simple:
either offer unlimited bandwidth(since you're not charging these anyways), like Cloudflare Pages does,
or put in place controls that will allow customer to set a top limit for their budget. You can't just all of a sudden send them a $104K bill and expect them to pay when the've never spent more than a few bucks. And then even worse, you can't pretend to expect them to pay 20%, then 5% then pretend you're doing them a favor by completely liftig it off. That's just arbitrary billing and preying for any victim that would fall and agree to pay 20% or 5% etc. I'm just asking for common sense and practices that build trust, not arbitraty billing rules.
In New Jersey I have to let an attendant pump my gas. If I have a heart attack while he’s pumping gas, but I never explicitly say “please stop once it’s full” and he, innocently enough, takes the still-flowing gas hose and pops it into a sewer grate once my tank is full, you’d be hard-pressed to find a reasonable person agree that the attendant was throwing me a lifeline when he refunds me after I come back complaining about my $2k gas receipt.
This is a dumb analogy, but the point is there is very obviously a pattern in this payment process that is ripe for abuse. The question of whether or not you aim to be an abusive business, plucking every shady profit where you can put the onus on the customer to try to come get their money back is one that many companies are deciding, and many are erring in the direction of the dark pattern.
By not working to avoid this problem from the get go, there is an implication about how a company wants to make their profits.
Pay for what I use works for airline seats and reserved compute/storage resources.
I have no control over how much traffic my public sites get. There is zero value in me signing up for a service which charges me based on traffic if I can’t control the maximum they’ll charge me. Would you sign up for an infinite bill?
the CEO said they're "forgiving any bills from legitimate mistakes" which effectively means "just make everything zero dollars bro".
And no, he didn't use all that bandwidth, he was victim of a DDoS which the hosting provider should have measures in place to prevent or limit the service if it happens.
Perhaps a bit ignorant, but to be fair that Netlify attempted to charge him is absolutely ridiculous. With my hosting provider, I would pay a whopping 50 EUR for the same bandwidth that he was asked to pay 104.5K USD for. That just shouldn't be possible to happen, especially on a free tier.
Any person seeing a user that normally has a $0/$10 per month bill suddenly spike to $104K would see that this is obviously a DDoS.
If it has always been a "policy" to forgive bills, shouldn't it have been 100% forgiven immediately after OP contacted support in the first place? Why go through the trouble of playing the hero by offering "discounts".
The user was asked to pay 20% then 5k on a service that's called "free" but has some extras which actually cost money.
After this the CEO comes along and says that the policy is actually not to bill for this kind of event... But the company actually tried to bill this user 3 times... soo it all stinks really.
You can't rely on such a policy if it is not part of the actual contract. This doesn't address the enormous uncertainty and risk that is present here when using Netlify.
This is what sticks out to me about the situation. I would much rather a site go offline due to service overage triggering at some limit that I set - simply relying on the good faith of a host to subjectively waive fees is not reliable nor does it instill confidence that I won't be financially ruined by malicious third parties (like nearly happened here). I would imagine that the good faith of Netlify in this case would mean very little to a court when there is a contract that stipulates costs for services, and the worst case scenario for a user is that Netlify could take the issue to court with the contract the user agreed to and demand full payment. Even the possibility for this situation to occur without any tools existing to prevent it is terrifying and is a terrible value proposition for a service.
By the time you forgive the bill you may have caused significant psychological distress, maybe even irreparable. This doesn't feel like a responsible approach.
This is the way most companies work unfortunately. Paypal limits your account and makes you wait 6 month to (maybe) give you a way to get the money back.
This is why I stopped using PayPal. My credit card company allows me to issue chargebacks with typically very little friction because of my credit history. PayPal once put me through the wringer for an order I cancelled and never received a refund for. After that I deleted my PayPal account and decided to never use it again. In this case, I would take my site off Netlify and never use it again.
I’ve already migrated my two sites off Netlify after reading about this incident, and seeing other replies where folks said they were stuck with large bills.
This large bill doesn’t look like a legitimate mistake, it looks like everything worked as intended until things got escalated via Hacker News.
This leaves all your other small business users potentially on the hook and at the mercy of your mercy.
Not only should this stuff be capped rather than the dam allowed to flow, but your systems should have picked this up immediately and known it for its nature.
Thus must have been a nice little earner for you over the years.
> traffic spikes that doesn't match attack patterns
I interpret this as "we always charge for traffic served, but we attempt to block illegitimate traffic" which means of course that the worse their traffic discriminator performs, the more money they make!
I assume you'll be offering this user a good amount of credit on their account for having to deal with this BS and the stress of being told they owe you $100k?
So the original support worker just pulled 20% (and then 5%) out of thin air? Given your internal knowledge, can you maybe explain why a support worker would ever do that if policy is simply to forgive the debt?
Anything Netlify deems them to be, of course. That's why these sorts of T&Cs use weasel words like "legitimate", "reasonable", "expected", etc., instead of giving specifics you can action against. That way they can claim every thing they've done is legitimate and reasonable no matter how fallacious that claim is, and double-dog dare you to spend the time/money to take them to court (or worse, imposed arbitration with an arbiter of their choice) and prove them wrong.
So this one got attention due to some good Samaritan on Reddit who told OP to post here. Now, to the real question here: have others not received as good advice and just paid up?
I'm moving my domain name and personal site off Netlify (already deleted the sites, DNS transfer requested), probably moving to Cloudflare pages.
It may only move a few MB a month, but I just can't risk if I put anything more substantial there that I might get hit with a bill for $100k and you maybe will forgive it. And that this has apparently been policy for nearly a decade makes it even worse.
Sorry, but there is a lot more going on here than you addressed, these charges were incurred on your "starter tier", which has no mention of additional costs.. I've noticed a lot of "sponsored content" by netlify, and again no mention of this possibility.. Also, no comment on not having ddos protection, or at least a spend limit?
Sure, this instance was resolved, but it's also the top post of the last month. Who honestly things it would be the same outcome if not for going viral...
I’ve been a Netlify users on the Pro plan for a few years now. Moving from Netlify to CloudFlare after this; “this didn’t come through in the initial support reply” doesn’t cut it for a $100k bill.
But you do see how _not_ addressing this in the initial support reply is going to cost you all in the long term, right? The real lesson here seems to be for small projects, it may well be worth the investment to handle my own hosting. All I see here is that getting you to do the right thing required publicly shaming you, which means you can be trusted about as far as I can throw a piano.
I’d rather be shut down than have a heart attack from a $100k bill. That could literally kill me from stress, even if you pinky swear to refund any oopsies.
That is an outrageous and inhumane policy. People get panic attacks when they get told they owe 100k they don’t have. People will be terrified your internal process wrongly determines the bill is legitimate. Imagine you have to study for an important exam or that you have a paper due. How can you possibly focus with this nightmare at your doorstep?
The survey is a reflection of what our community responded.
Next is the largest framework in usage and is really will liked by it's users. This is clearly visible in the charts of the survey.
But for the first time we’ve run this survey, Next decreased in usage year over year (from 47% to 46%).
Astro jumped all the way from 11% to 18% with 87% responding they want to use it more.
Eleventy dropped in usage from our respondents from 19% last year to 16% this year.
None of this is an attack on anyone, it’s just data from our survey. The rise of Astro is one of the most newsworthy bits of data from the survey and reflects genuine excitement in the community we’re part of.
We changed the title to better reflect that the survey asks plenty of questions not directly related to jamstack (like usage of AI tools for web development etc).
Nothing changed in the methodology since last year and as always the survey was run by our data team.
If you scroll past the editorial part of what our team found newsworthy in the data and down to the actual survey data, you'll see clear charts of absolute framework usage and detailed breakdowns on satisfaction for each framework.
But the percentage changes are correct, aren't they? I don't think readers will think that the graphs represents absolute popularity, also (IMO) any survey taken by a company will be used for marketing purposes and should be consumed as such. (It's just some text on the internet, nobody is going to take it as gospel.)
It would be nice if the visualization reflected the information you've shared here (including both the absolutes and the amount of change). Then it wouldn't feel misleading and you would not need to write explainer comments on HN.
The absolutes are in a bar chart in the framework part of the survey that makes the absolute usage from the responses very clear. The chart Zack took issue with is from the editorial part of the survey where we found the biggest news in the framework section to be the growth and satisfaction for Astro.
This comes off as a disingenuous response, by typing out a large block of text while neglecting to address any of the individual concerns written in the article. "it’s just data from our survey" is a cop out -- the presentation of data is just as important as the data itself, and the concerns of underlying bias via communication model presented by Zach all seem reasonable to myself.
It's fine to not be an expert in Data Analysis, and if it's true that the company no longer employs someone with those skills, there should be a greater willingness to make adjustments and corrections when problems are raised with publicized analysis.
I would actually argue that Jamstack has won to the point of basically just being "Modern Web Development" by now.
In the 7 years since I first presented the term at Smashing Conference in San Francisco, I can't think of a single new successful commercial CMS that hasn't been headless or API first. I can't think of a single new successful commerce platform that's launched in that period, that hasn't been headless or API driven.
On the contrary, big existing players have mostly changed their strategy. Shopify is more and more embracing a decoupled approach of building globally distributed shop UI's with Remix (running either on their own platform or places like Netlify or Cloudflare) pulling in products, pricing and check out flows from their API layer. Most existing CMS companies have started integrating API first approaches into their strategy and in the data space we've seen an explosion of "Database as API" from Neon, to Supabase, to Xata or Converx, etc...
Part of the confusion has always been the conflation of Jamstack with "static" when even my first presentation on Jamstack back in 2016 had a big slide with the word static crossed out to underline that I personally didn't intend to conflate the two. The real game changer for treating the web UI as it's own decoupled application, and the backend as a set of API and services where some are your own running your core business logic, but a lot of them are provided by external providers.
At Netlify we're now focusing less on evangelizing the Jamstack approach of treating the web UI as it's own decoupled, independent layer, running on top of API's rather than stuffed into your backend systems - and more on helping really large companies adopt this at scale for their core web architectures (Composable Architectures). But not because we're any less bullish on the first part, on the contrary - because we don't really have to anymore!
And the article's conclusion that we somehow failed is absurd. Sites and apps on Netlify are visited by more than a billion unique visitors a month, we delivered more than 10 Petabyte of
data out of our network in December alone, have onboarded more than 4 million developers to our platform, and continue to prove that we can scale this architecture to some of the largest and most complex companies in the world, running big complex projects with faster time to market, higher productivity and higher conversions and revenue.
JAMStack isn't modern web development. 80% of the internet still runs on PHP on traditional servers. Netlify is needless complexity (nevermind the vendor lock-in) 99% of developers will never need.
You also don't address the OP's points where Netlify has suffered the same fate of "enshitification" where features slowly get stripped out and moved into pay to use buckets, likely at the behest of needing to payback 100+ million dollars in VC funding.
Your comment doesn’t refute the idea that JAMStack isn’t modern web development. All you did was pull up the over-used statistic of “80% of the web is PHP” which I’ve heard for well over a decade. It may have been true at one point, but I highly doubt it is now. (Citation needed)
Netlify has done nothing but innovate and push the needle forward for front-end devs. I’ll be there until there’s a VERY strong reason not to be.
PHP is know for being very leaky about being PHP (and since PHP + ecosystem have a bad history of CVEs, being leaky about being PHP is not cool).
Java/Kotlin/Go/Rust/Ruby/Python/JS/TS are a lot less leaky about what language the server-side is written in. Usually the webserver used advertises itself name and in a server string, but it is considered bad practice and thus often switched off.
Reading "php" extensions in paths is a clear giveaway, so are "htm" extensions for microsoft products. Tools usually guess the language/framework based on some of these giveaways and the better the tools the less this is evident.
I jut checked some web apps I worked on, and only the one I last touched 10+ years ago is detected with buildwith.com; it's a Rails site.
All the Java/Kotlin/Rust/Hasura+Elm apps I worked on since are now shown as "Nginx" (the rev proxy in front of it).
I just checked the day gig site... builtwith claims it's using Webflow and Apollo GraphQL (Neither of which it is) and doesn't mention at all the language it's actually implemented in (Python), although that's not surprising since it's an in house framework.
> How is being “leaky” in any way bad or even good?
Information-gathering is a common early step in any attack against a system; knowing the language & libraries involved (especially their versions) allows you to search for any existing CVEs that apply.
> Are you advocating “security by obscurity”?
I don't think OP was implying that security by obscurity alone is sufficient, just that it's unwise to advertise information that's not relevant to end users, that could help would-be attackers.
While it kind of is security by obscurity, it's a very basic piece of server hardening to stop telling potential attackers what software you're using (within reason).
Back in the day (!), server software used to honestly respond with things like the software name and exact version number it was running.
Naturally, that meant scanning for vulnerabilities was a lot easier than it needed to be.
Not true. There are some real cryptographic realities that are based in "open" math principles.
There's also the way of most using runtimes/libraries that (constantly) have CVEs in them; and understanding why it is that these languages have CVEs in the first place (see my comment on "eval()").
If you languages has "eval()" or something similar, it is a lot easier to attack. Same for when it allow you "upload a file in some place where it gets executed".
These things are not so easy, say, with a C++/Rust/Go app. Or even in most JVM configurations. JS has similar issues, that Deno is trying to mitigate to some extend.
obviously being properly secure is better. but if you leave your unlocked, it's better to not also hang a sign above it saying "this door is unlocked".
obscurity is absolutely part of good security practice, as long as it's not all you're relying on.
If people are picking up non-JAMstack solutions for greenfield web development, then that means JAMstack is just one of many options for "modern web development". (Along with Laravel, Rails, Django, and even/especially Wordpress, depending on how we gatekeep what we mean by "web development")
Objectively speaking our free tier today have far more features, higher limits and more capabilities than it's ever had before.
We are also building a real, longterm sustainable enterprise business. We're not a non-profit and we're here to create a big lasting company that can keep investing into the future of the web.
Almost every feature you charge for is something you can achieve for free inside of a basic VPS. I understand you have the classic SV "Hotel California" model where you can check in but you can never leave. But frankly this makes the internet worse in every way possible and part of the point of the original article.
I gotta say, this comment really comes across as being written by someone who has quite literally never tried Netlify, or doesn’t understand the value prop at all.
Seriously, it is orders of magnitude faster to deploy a static website on Netlify - simply drag and drop a folder from desktop - than it is to spin up just a single VPS on Vultr… and by the time you’ve configured that VPS, I could have done a dozen revisions to the website, and it would still be more difficult to deploy updates to the VPS than to Vultr. Don’t even get me started on the complexity of a global CDN.
Do I wish all of Netlify was free? Sure, yes I do. Does this mean it’s not valuable? Of course not.
The irony of this is that you seem to be saying that (a) Netlify locks the customer in with useless features, and (b) it can be trivially reimplemented in an entirely custom but otherwise “free” VPS.
The real question is, which “free” VPS are you going to use that will serve the same capacity as Netlify’s paid plans? AWS? Do you think you can avoid lock-in using AWS - of all things?!
No, you asked for a free comparison. If you want to go to paid features netlify loses by a mile. You can search all of this yourself with all of one google query.
I didn't ask for anything. I am just responding to your comment. You said:
> Almost every feature you charge for is something you can achieve for free inside of a basic VPS
I have plenty of experience with both Netlify and VMs, and I don't think they are even remotely comparable.
> You can search all of this yourself with all of one google query.
You're the one making the claims - I'm not even sure what I would be searching for since the parameters of a VM are vastly different to those for Netlify. How could I even find TTFB for a free Oracle VM? In fact, my personal experience with Oracle VMs (granted it's from before 2018) is that the jitter would be very high, such that any TTFB would probably be meaningless. And it would depend on the deployment region, too. Not to mention the HTTP server being used.
Words are cheap. Unless you're willing to substantiate your claims with some actual facts, nobody learns anything.
You are definitely getting much much more with a free VPS than with netlify. Netlify's value is in being easy, not cheap. If your argument is for being cheap then you will lose that argument all day, every day.
You really need to explain this. Honestly. Actual numbers.
Netlify’s free plan is awesome. You might get more bandwidth with a VPS. But you won’t get a CDN and you won’t get atomic deployments without a bunch of work. You’ll use half your VPS storage on the operating system. You’ll need to keep updating your software and maintaining backups. You’ll spend hours or weeks a year maintaining this thing, versus zero for Netlify.
It’s not just about the cost of bandwidth or storage. You gotta compare like with like.
You can't compare like with like. How will netlify compare with the below?
1. Full server control (install any package)
2. Host any kind of server - Django, Rails, etc. etc.
3. Run database servers: MySQL, PostgreSQL, MongoDB, etc.
4. Background processes (cron or systemd)
5. Run VPNs and proxy servers for personal use at the same time.
6. Host email servers
7. Multi-purpose usage (Use for remote dev environments, cloud storage, AI/ML platforms, etc.)
You know you have to secure, patch, and monitor a VPS right? Why pay for a VPS at all? You can get more compute, memory, and storage with a dedicated server for the same price and even setup multiple VMs on that server, each with all the features of a VPS “for free”.
The difference is I'm not out touting using a VPS as "The next generation of web development" and having payola articles written that if you don't use a VPS then your career is going to get left behind.
Setting up a VPS is pretty easy. Ensuring your VPS is configured to restart your app when the server restarts, maintaining OS and library updates, maintaining security, updating the app itself with simple-enough conventions, and configuring monitoring is not so necessarily easy. That’s not including documenting (even if only for your own future reference) how things are configured.
Digital Ocean in particular has great guides to get you from the starting line to something that in most cases will work okay, but as a long-term solution to “I have an app I want to run on some infrastructure”, I agree that there’s a non-zero cost to managing a server that, like you, I’d much rather not deal with.
>80% of the internet still runs on PHP on traditional servers.
Do they? Or do they just use Wordpress? This is important to differentiate because people's idea of PHP on traditional servers ITT is a far cry from setting up wordpress.
Netlify worked for me when all we did was basic static hosting for about $500 a year. Then they wanted to up us to about 10k a year and suddenly the cost was significantly worse. The only reason we didn't dump it is because we didn't have enough engineers to swap it to something else because we had projects that needed to be done, but it was our top optional priority.
Unfortunately netlify went from loved to hated overnight.
what vendor lock in? Netlify at its core is still quite simple: pull stuff in from GitHub/Lab, run the build commands in an Ubuntu VM and publish the results.
Sure, there are some plugins that you can use to transmute the result builds and there's Edge functions, but nothing is hard to move to another provider.
> I would actually argue that Jamstack has won to the point of basically just being "Modern Web Development" by now.
Don't you think there might be just a little bit of hubris creeping in here? JAMSTACK is such a niche/fringe it's not worth talking about. I'm a solo dev and network with many others. I don't think I've ever heard JAMSTACK mentioned let alone adopted. You're living in a self-generated bubble.
I've been doing web development since 1997, but posts like these let me know how far out of the loop I have become. I have no idea what Jamstack, Netlify, Headless CMS, etc. even are. I'm still in the: here is a webpage (html, css, js) that makes an ajax/fetch call to a webservice that SQL queries a Postgres database and returns the results as a JSON string.
Honestly, I've just finished 5 years as CTO with an e-com company with our CMS built on Elixir and I can't tell you how wrong you are. Jamstack, headless CMS and SPA frontends etc are just a massive waste of time. There's no joy in having separate code for your API and frontend.
I worked at an ecom that used Elixir instead of an spa, with the backend being the older legacy app, and elixir rendering pages based off API data from the backend. This was well before live view
It worked amazingly fast. Some things were painful, and would have been better served with some progressive enhancement via a dollop of frontend js, but they were rather small exceptions
When I left a few years back they'd undergone some leadership turnover and the new wanted to migrate to some messy jamstack thing. They had millions of SKUs. Last I heard they migration hadn't gone anywhere
Tale as old as time lol. Elixir is uniquely great at content sites due to the fast template rendering - but it’s really not about Ex at all. Just don’t drown in unnecessary moving parts and build tools. It’s just a website.
LiveView wasn’t ready yet when I started our startup (react on front, Elixir and Hasura on back).
But man, I wish it had been. React is nice enough but the idea of not having to redefine my types at several different layers and fight with npm sounds like a dream.
We were in a similar position. CMS ran on dead views with a bit of progressive enhancement first, and then we had a couple of react apps around for interactive parts. When liveview was released we tried it out and it basically “just worked” as much as a software lib can, so set about getting rid of the react.
Sure: key idea is pages are documents. Each page contains a JSON blob which specifies the blocks it's made up of and the content for each block. Each block is just a module with a schema specifying the data it takes and an associated template and a render function. Rendering a page is a URL resolution -> Controller -> DB lookup for the page -> pass the block list to a master render function and done.
The thing some people don't realise is EEx is compiled to a function call, which means no string interpolation, regex whatever. Getting a bunch of HTML is just a function call.
Jamstack is "Modern Web Development"? Are we back to that point in the webdev rollercoaster where everyone pretends like embedding business logic in a client using javascript is a good idea?
The author never actually articulated what his beef was with Netlify. So yeah I think you're just as confused as the rest of us. Apparently Netlify was supposed to become Heroku, but Heroku is bad, but Render (which is... Heroku with a fresh coat of paint) isn't? I dunno.
Render seems to have autoscaling with a reasonable round-robin load balancer [1], whereas Heroku only has autoscaling in (expensive) Private Spaces [2] with a dumb load balancer [3].
Yo. Thanks for Netlify. It’s legit and I, for one, welcome our new overlords.
All jesting aside it’s easily one of the most pleasant developer experiences you can have as a modern web dev in 2023. Everything just works and failures are loud, obvious, and usually dead simple to fix.
FWIW, I love you, Netlify CEO. I've got a bunch of different static-ish websites hosted with you---my personal site, the sites for three different books, etc., etc. Everything is build using the CI, with a different frameworks and/or build processes (essentially whatever I felt like playing with at the time), and it makes it incredibly easy and cheap. So don't listen to the haters.
I don’t really understand the concept of Jamstack or why it’s new. Isn’t it just HTML+JS (whether built with a framework such as Angular, React, Svelte, Gatsby, etc) that uses a backend api? What exactly are we discovering? What is the innovation here? Markdown?
So far Netlify Edge Functions runs before the cache layer, so you can actually use a minimal function to rewrite the URL to remove all unique params, etc, and then let it pass through our system to a Netlify Function which runs behind our caching layer.
For anything you can do at build time as a static HTML pages we already strip query parameters from cache keys.
Interesting, thanks... do you have any docs on how we might achieve this with Next.js? Am I right in thinking we would have essentially a custom Edge Function first that handles query params, and then a second Edge Function that renders the Next app?
I work at Netlify on framework integrations. Next has beta support for running the whole app at the edge, and Netlify supports that. If you create you own custom edge functions they will run first, so you can do just that. You can also run Next "normally" (i.e. in the node-based Netlify Functions) and run your own edge functions in front of that. In those you can modify the Request in any way you'd like, before passing it on to the origin.
One of the big reasons for going with Deno is that it's an open runtime closely based on web standards. You can download the open source Deno Cli and all code written for our edge layer will run exactly the same there.
As more and more front-end frameworks starts leaning in on running part of their code at the edge, we felt it was important to champion and open, portable runtime for this layer vs a proprietary runtime tied to a specific platform.
Netlify CEO here. I'll try to answer the questions from the thread so far:
Some of our customers are affected by an outage of Googles Load Balancer.
These customers are not taking advantage of our DNS management, or they are not using a DNS provider that supports CNAME flattening and are using their root domain name for their website (ie, no www prefix).
While we don't recommend the setup, we do provide a single IP address to bind an A records for customers that want it.
In general we run our edge infrastructure as a large multicloud setup spanning several different network providers, and offer two separate networks, one for free/self-serve customers that will get newer features faster and one for enterprise customers running mission critical projects where we guarantee very high uptime and reliability through formal SLAs.
The single IP mentioned above however corresponds to a Google Load Balancer, and they are unfortunately currently having an outage for all load balancers in the relevant region. Read more on https://status.cloud.google.com/
Again, while we generally don't recommend using the A name setup for anything mission critical, we are currently doing everything we possible can helping enterprise customers that have chosen this setup to change their configuration.
Really sorry for all the trouble this are causing for our users, full RCA will be forthcoming.
> These customers are not taking advantage of our DNS management
I think I understand the point you are trying to make, that customers who are utilizing Netlify DNS Management are unaffected because reasons, but this is phrased in a way that implies that it is your users fault for this downtime because they didn't chose to use your related service.
Full RCA with the steps the team has taken to improve this setup will be coming soon. The main issue with AWS's DNS solution, in this context, is that they don't support ALIAS records or similar techniques (CNAME flattening, etc) for A records pointing to any external provider. That limits our options a lot in terms of what we can do, since anyone using this setup need to point all their traffic to one or more fixed IP addresses.
Our current solution for the free/self-serve tier of Netlify has been to rely on Google's load balancer product to give people a stable IP pointing to a highly available solution. In light of recent issues, our team has setup a new permanent IP for A records (75.2.60.5) backed by a different solution, but due to the way DNS providers with no ALIAS record support work, it does require our customers to manually change their A records.
I totally get that moving DNS providers is a big deal and we want to give the best experience we can regardless of what provider you're on, but we have to work within the technical limitations of those providers and it's the nature of things that we do have more options to deliver a completely seemless experience when we operate both the DNS and the edge layer for customers.
Route 53 General Manager here. Flattening of external provider CNAMEs has a number of availability and accuracy risks. Route 53 offers a 100% availability SLA, and we really mean it. We’ve heard over and over from customers that reliability is our most valuable feature. We can’t provide that same reliability when external queries are in the mix; if we query asynchronously then features such as geo-based routing don’t work as expected for customers. If we query synchronously, then latency and availability are impacted directly.
We do offer ALIAS records between Route 53 hosted zones, and this capability is open to providers such as Netlify. We’d be happy to have customers ALIAS to a hosted zone managed and updated by Netlify. It sounds like your IP addresses are relatively stable, keeping these in sync doesn’t sound like it would be a big deal, and would give you a lever you could pull to change your customer DNS quickly in an event such as this. You could also configure health checks on your own DNS records, which any customer ALIAS records that point to your DNS records in Route 53 would inherit.
If you’re interested in going this route, please contact me at alecpete <at> amazon <dot> com.
If each Route 53 POP is already close to the querying DNS client, then things like geo routing with cached answers might just work well enough in most cases? With each POP having its own cache.
Auto-refreshing the popular records in the background before the TTL expires to help smooth over any temporary issues?
Other big name DNS providers have ALIAS type records. I imagine according to the SLA, AWS Route 53 is still "available", even if it can't resolve a "target address record" (as the ANAME draft calls them) but Route 53 is still able to respond.
Phrasing can always be better but the point is that there's a way to map your DNS to Netlify which is risky and Netlify hasn't made the aggressive decision of blocking it. They outline in their docs all the reasons why you shouldn't do it, provide instructions for how to avoid it and also offer (but do not require) a hosted DNS setup which avoids this pitfall by design.
Some folks still choose to use this way, some have no other choice for various reasons and some don't care/comprehend the potential pitfalls. I do believe most users avoid using a root domain name for their website.
As someone who is a little clueless about network infrastructure: if I own "dwrodri.com", and I'm not running a bunch of other services which need to point to this domain, is there any reason why I wouldn't have my root domain pointed to my personal website?
I would personally imagine that any individual or SOHO business hosting their website on GitHub/GitLab would just buy "MomAndPopShop.com" and point it there. I guess I don't know off the top of my head how many of those sorts of places on the web still exist...
The problem is not that they're pointing their apex domain to a personal website; the problem is that they have a CNAME record in place for their apex domain, which is not actually allowed per the DNS standards
This should not be the case; if you'd like, Netlify's Support team will be happy to review your settings to help discover why it didn't help you out (start from https://netlify.com/support) and ensure that you are "futureproofed"!
I switched to using your DNS to resolve this issue, but https://js.la is still busted and because I'm using your DNS, I can't manually set the A record to go to the workaround IP address.
"These customers are not taking advantage of our DNS management"
You're right. I'm using Cloudflare's DNS. I trust them more than I trust Netlify and that's just a function of their size vs Netlify's size. This response needed better wording.
Depending on your config, another DNS related issue with Netlify is the way NS1.com (their vendor) handles domain names. A domain can only be added to one NS1 account. So if Netlify adds to their account internally, you can't use NS1 and vice versa.
Honestly, "not taking advantage of our DNS management" is a garbage response. We use AWS for our DNS management. If you offer a configuration, you should support it fully.
Our sites have been down for 3 hours now, and you're blaming someone else? We have 5 properties on Netlify now and will have 0 this time next week.
Yes, it is our fault for believing Netlify had contingency plans as hosting is their core business. We're fixing this mistake now so that our customers don't have the same experience.
Nobody is telling parent's customers how to feel. But the OP suggests that Netlify customers should be faulted for choosing the the wrong setup. Broken trust goes all the way down the chain, which is why the middle links have every reason to get ticked off.
The difference is that Netlify communicated the risks to its customers, something other parts of the chain apparently did not do, in addition to not evaluating the risks presented to them by Netlify.
Did you read the docs [1] before writing this? Putting a "(recommended)" on one branch of configuration instructions isn't the same as saying that the other option has a single point of failure. Also, people on both sides of a service don't have the same responsibilities - that's the whole point of the service.
Communicating about risks OR outages are both hard, and every company has both. I'm actually a happy (though impacted) Netlify customer. But it's completely bizarre to me to try to invalidate this customer's complaint.
Yes, I’ve visited that page before today. I admit my familiarity with these DNS setups may have made the tradeoff jump out at me. No problem invalidating the complaint.
I'm not sure your organization's setup with Netlify but isn't the whole point of Serverless to be... "serverless"? I could migrate twice the amount of properties you have to another provider in less than 3 hours...
I get your frustration but maybe cut some slack. If anything is mission critical, you should have had a backup plan if Netlify, Vercel, Cloudflare, or something else.
We use(d) Netlify for the frontend. I agree, our mistake was believing Netlify could be used for more than toy websites and took care of backup plans for us. Clearly they do not.
It's not just migrating the front-end if they're also using other functionalities like Netlify functions, forms, authentications etc. Netlify is not just static file hosting.
