I am talking internally with the customer support team about what happened to this customer's traffic. It is definitively not the case that doing "four requests per minute" gets you rate limited.
My application[1] with similar rpm as OP received 503 status last month for about an hour first time in 1.5 years, I assumed it to be an one-off issue at CF end. It happened while I was using it and so I'm now concerned if it's a regular affair for my users, most of whom are not from my geographical location(data-center).
I'm a free plan user because my application is not monetized and obviously would switch to paid plan once it starts generating money. CF free plan has been indispensable for many such free projects.
UptimeRobot(Free) doesn't show anything fishy, except for CF error 521 occasionally. Again UptimeRobot requires paid version to check for HTTP status message, guess I'll have to do it or setup a self hosted(recommendations?) solution to monitor HTTP status of CF Free application.
If CF does perform rate limiting for such low rpm, I would prefer to be intimated as even third rate shared hosting providers do it and I hope CF wouldn't put their reputation on the line for something like this.
Why don't you send an email to the customer if there is an error while serving content by cloudflare? It can be a daily digest of the errors of the day.
If the problem is on cloudflare's side then the cloudflare user doesn't even know about it if it's an intermittent error.
While we are on the subject of rate limiting and such, does Cloudflare allow using its service for serving video files for a media heavy social media app where we store the videos on Backblaze B2?
> The Services are offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as part of a Paid Service purchased by you, you agree to use the Services solely for the purpose of (i) serving web pages as viewed through a web browser or other functionally equivalent applications, including rendering Hypertext Markup Language (HTML) or other functional equivalents, and (ii) serving web APIs subject to the restrictions set forth in this Section 2.8. Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service. If we determine you have breached this Section 2.8, we may immediately suspend or restrict your use of the Services, or limit End User access to certain of your resources through the Services.
And I am wondering what kind of subscription would be appropriate for my use-case if any, that would allow me to host the files on Backblaze B2, and serve these files through Cloudflare.
(Cloudflare would also be hosting our API, so it’s not just video files that we want to serve via Cloudflare.)
For what is is worth, I've been using their Free Tier to serve over 10k requests per hour with ~60% html / ~40% media (mainly pictures, very few videos) and I've never had a problem.
However, for a CDN for video content YMMV. Perhaps you should try contacting them directly to have a written estimate.
I basically have something similar to your use-case. While I'm on the free tier for all my domains, the supplemental agreement, at least in the US version, removes the specific limitation to serve primarily html/text content if you enable Cloudflare Workers, which also has a free tier, applies on an account and not domain level, and has no requirement for you to serve your content through their KV system. I ended up paying for my CF Workers subscription for a different project on my account but the system worked fine prior to that. I served 111GB of a combination of m4a and png files in data from my B2 bucket via CF in the past month.
Is it possible that this person's free-plan traffic was routed through a POP that was under attack? Then, upon upgrading to Pro the routing rules for the account were modified to avoid under-attack data centers.
I understand you can't share a customer's account details.
But can we at least get an update if this is the intended behaviour of the CF free plan accounts (so people know to upgrade) or is it an one-off incident?
I think it might also be worth considering whether returning 503 is the right response when rate limited. 5xx indicates server failure, a "429 Rate Limited" is a more appropriate response.
4xx is a client error and 429 indicates that particular client is making too many requests. 5xx is a server error and 503 says the service itself is unavailable irrespective of the client’s actions. In this particular case it’d be unavailable for contractual rather than technical reasons, but it’s still the right status
A great first step would be to take responsibility for what the customer has proven to happen. "It is definitively not the case that doing" is directly at odds with what the author provides, you might want to reframe your message to "it shouldn't be like this but unfortunately it is like this right now, sorry" rather than dismissing the author with that they are not correct.
you can (and should) put monitors on the outside of your infrastructure as well as inside. I've multiples hitting the login page and robots file where I work, and never got an unaccounted 503
Like I wrote, for some integrations it is not possible to gather all the logs. Also how will you know that a client accessing your website in a browser gets 503 instead of your web page?
You don't, but your monitors will show the 503 happening, how often, on which endpoint operation and in which regions; that will give you a pretty good picture of whether is actually your CDN layer or something else triggering the 503
I’d look into idle connection handling, I’ve had problems in the past with cloudflare and it’s long open sockets to origin being quietly closed at origin, breaking requests to cloudflare.
Please keep us posted. From a naive perspective, what happened has a flavor of "protection fees." A detailed explanation would help quickly dispel this perspective.
As someone who has run a wide range of sites on Cloudflare for many years, I am certain this is bupkis.
1. Cloudflare doesn’t send 503 when they’re deliberately rate limiting; they send it when there’s an unexpected problem. It’s fairly rare, and it usually results in something being posted on cloudflarestatus.com if it persists.
2. Sometimes error rates are low enough or the conditions are obscure enough that you may have to contact support to point out that it’s happening. This is even rarer.
3. When you switch between plans, the IP addresses assigned to you may change, and users may hit different edges. If the issue is regional (as is usually the case), this may resolve it.
4. I have been sending tens of requests per second to a free plan over the course of several years and have never measured an increase in 503 responses on Cloudflare’s end compared to a paid plan.
5. Cloudflare’s response mentions specific 503 errors that are fairly rare; they’re notable because they look like a standard Nginx 503 page, rather than a nice Cloudflare error page. The only difference is that they will say “cloudflare” at the bottom where the Nginx version would normally be. You probably frequent sites that use Cloudflare free plans; how often do you see this error message?
6. What the hell is this n=1 correlation? It reads like a conspiracy theory. It is a conspiracy theory.
There are plenty of reasons to criticize Cloudflare, but this isn’t one of them.
While this blog post has very little in the way of proof, we actually got screwed over by CF a few weeks back. Basically, one morning all requests started being slow: small responses would take 7 seconds (within a few milliseconds, if you accounted for the RTT to the edge) with 7 seconds TTFB, larger ones (not sure exactly what the threshold was, but it was hundreds of KiB, not MiB) would take 1 minute (again, almost exactly 1 minute), but these would just stream very slowly. All while having cf-cache-status: HIT.
It turned out we got throttled because we were serving some (a minority, something like 10% of the traffic) video files from the domain and they wanted us to upgrade to enterprise (from business, I think). We just stopped serving video through them.
The annoying part was that we weren't notified and, obviously, had some downtime (no way you can call a site loading in 5 minutes up). They said they have released an update that does notify customers when this happens, though.
Hmm, we've been experiencing vastly reduced throughput as well (something like 100 kB/s on images >100 kB in size, cache hit). We're on the free plan, though. Makes me wonder if there's actually some throttling going on behind the scenes.
I can only comment on our exact situation, where the response times were as I mentioned them, exactly. I've not experienced that, but I guess it's worth shooting a question to support (though, expect several days on the free plan)
Might this have something to do with people using Cloudflare to share child pornography a couple of years ago? They've been talking about it again recently.
Today there was a topic on HN about customers who complain about limiting free tiers. That’s a great example of it, especially from someone who provides paid SaaS product.
Free tier is awesome for small websites, keeping domains and many other things. But if you run business who earns, then just pay for that god damn thing, and stop ranting. :-)
Someone came on HN to report that a free-tier product is failing in a way that’s undetectable and doesn’t make sense as described by the products’ marketing materials, and the resolution is to pay (a small amount) for the upgraded version. One of the Cloudflare engineers* immediately responds to the article saying he’s looking into things because this shouldn’t be happening. Everything seemed to be going so well in this story. I only continued reading the comments to find the first one where someone turns this into a story of blame for the poster being cheap, and your post was it.
It’s not the case with everyone, OP could wait until supports get back to him and investigates everything, and then push a pice on that. Bug -> Contact Support -> Wait for answer & solutions -> Happy or not ending.
What if he accuses them without justification? Which is probably a case here.
From where I sit he is entirely justified, and the only reason problem is getting fixed is precisely because he finally decided to warn others about this problem.
They did get back to him. What evidence do you have that they were still investigating anything? They answered his questions and the answers were no good.
Could this be explained by something like “there was some bug causing this person’s config to be in a weird state (on the cloudflare side), and changing the payment tier merely gave the system a kick that got it out of the bad state? After all, the person payed $20pm not because they thought it would give a better reliability but because they wanted better support for a weird issue. I’m not really very appalled by the idea that free users don’t get great support.
OP here. This article generated some attention and traffic. The claims contained in it are incorrect. After another contact with CF support, it turned out that the Bot Fight Mode behaves differently for Free and PRO plans. That's what caused the instant improvement after upgrading. Another way to resolve the issues I was experiencing would have been to disable the bot fight mode altogether or add a custom page rule disabling it. My website never experienced any traffic throttling. I've decided to remove the article, to stop spreading the misinformation about CF services.
Chatting with the folks internally I'll make sure that our documentation and support are clearer here. This should have been sorted out the moment you contacted support (or even before with clearer documentation). Sorry you had so much trouble.
I'm starting to lose faith in cloudflare lately - we're on the business plan and there's been some really strange UI bugs in the dashboard where it looks like state just isn't reflected properly (Access and Load Balancing being ones we see quite a bit)
Summary of things that we've had issues with lately:
- Caching of a specific route just stopped working.
I log a support ticket (nightmare to find in the dashboard) and miraculously it starts working again after the ticket gets responsded to.
- Some of our staff got locked out of part of our app because we exceeded the 5 free users of the "Access" plan. Upgraded to the 50 user plan in the new Teams part and it still didn't work. Contacted support, fixed again but no explanation. (multiple day turnaround on tickets)
We're invested in their tech a lot and I love workers - they really take some big tasks off us. If Azure come up with something compelling we'll probably switch though (assuming I can make sense of Azure's billing and product naming strategy)
How do you know for sure that you didn't? One of the main takeaways from the original article is that these errors are invisible from server side (your own stack) because it's the traffic is being throttled before it gets to your own servers.
Unless you have very good and consistent client side availability metrics you probably wouldn't notice the 2%-3% drop.
There is a pretty big collection of Hungarian historical photos online, a bit more than a terabyte, well above 100K of them and the photos are on B2 with Cloudflare fronting it. There's many terabytes of traffic each month and it's on the free tier. Since the hosting costs are covered by yours truly, I can assure you the project couldn't possibly afford cloud storage traffic prices. We would need to store it on a VPS and all the problems that come with it -- disk size, availability, reliability etc. I am so grateful for B2-CF to do this.
Special kudos for allowing the -- very cheap, only $5 -- worker addon without forcing a paid plan.
I just upgraded back to Pro, i will post the results in 1-2 days time. I used to use pro but there's really no benefit or difference.
Anyway i did a quick check, my 503 error is currently at 3.23k the past 7 days , that's out of 1 million request the past 7 days.
I was about to upgrade to ARGO and pay for it since i was optimising the bandwidth the past month until I saw this article which really gave me a shock.... because I heavily invested in Cloudflare stock market...... Which so far has good returns and i still believe in it but this article is critical ... If what this person say is true, I might exit Cloudflare earlier, from the stock market i mean.
Sounds weird. I'm not saying to trust Cloudflare blindly, but I used a social media site (very "reddit-like") my friend made and kept for over a year on free plan with no issues. Around 20-80 people online were actively creating content, private messages, browsing, upvoting, with live notifications with websockets and so on, it was a quite busy little site for a year or so). There never was issues with couldflare and it basically allowed the site to run on two smallest VPS machines mentioned friend could find.
Also why is author giving trust in a free service for his paid service? CF is not that expensive.
This article seems fairly scant on details to be honest. I understand how they came to the conclusion, but there's still quite a lot of conjecture here.
I feel like the main problem is the inability to properly debug your problem without paying while the plan essentially necessitates a degree of self-diagnostic when things go wrong. I also serve an xml feed via CF and also use it as a caching service in front of my Backblaze B2 bucket, and on the free tier by virtue of the files I'm serving far fewer unique users (less than half of the author's last 30 days) but a far greater amount of data (111GB). I'm not certain if the author should jump from what's clearly a vague and uncommitted conjecture from the Cloudflare tech about where the 503 error comes from straight to "they are definitely throttling me", since I've had the opposite experience and have seen 503s in all sorts of situations unrelated to throttling, but with a tiny sample size and no adequate diagnostic tooling it's also impossible for me or anyone to say that he's not being throttled, except CF themselves. I realize that it's their business model but it seems counter-intuitive that the service they sell seems partly "better ability for you to fix your problems yourself", sort of the opposite of say, managed/unmanaged servers. It'd make more sense, from a customer's point of view, that if I'm paying for a service, there should be service in both senses of the word - the product and the servicing of issues if possible, but not making diagnostics more transparent makes little sense because it prevents people who want to use the service from continuing to use the service, presumably part of CF's goal.
That does seem strange, I never experienced this myself. Also, I don't understand the use case for Cloudflare here, can you elaborate on "I’ve been using Cloudflare free plan for Abot"
My cost benefit analysis is that the Cloudflare free plan is a bargain. At any time I can instantly turn on the protection I might need because I have already integrated Cloudflare into my infrastructure.
I am careful about what technologies I add to my stack because I am a sole developer, but Digital Ocean and Cloudflare have been big wins for me.
Unpopular opinion - there's a lot of entitled people here and jumping to conspiracy.
There is no such thing as a free lunch [0] for people to depend a commercial service on. If your business model depends on another company providing you a free service perhaps you should reconsider.
This seems to be a person being cheap and jumping to conclusions, claiming broad assumptions and conjecture as fact.
Yes, cloudflare's free tier is deliberately dropping your requests to foil your freeloading commercial company - raise pitchforks!
Assuming this is all true (which I don't), I don't feel sympathy for the author for not purchasing a paid plan.
I have nothing but good things to say about Cloudflare's free services. However my usage is limited to small static sites, which are free of surprises.
I was doing a couple of POSTs a minute on a paid plan, and I saw a random 2-3% of request to be failing they never hit my backend, speaking to support wasn't helpful at all, that's when we moved away from Cloudflare, I wouldn't base my business on it.
For what it's worth, Cloudflare actually provides error code stats for the free plan in Account Analytics. The 5xx errors bucket is at 0.18% for me, but I don't have that many requests.
One gotcha for CloudFlare is that I find they will serve CAPTCHA pages for asset requests (like JavaScript or CSS) which will break your site (obviously the user won't see the captcha when the browser was expecting JS or an image).
To avoid this you need to turn the firewall and security feats to "Essentially Off" at least for asset requests (you can do this partial blocking via a page rule).
That being said this doesn't seem to be an issue with Free vs Paid, just a general problem with their blocking.
I would have to check the timeline but I was seeing this frequently with their IPFS gateway (as you can't control the security settings). It was a couple of months ago now so maybe it has been fixed recently.
I'm curious, do you know how it was fixed? Is it not serving a captcha for the second request on a connection or something? Or does it somehow figure out that asset requests are "safe"?
My experience was with using Tor Browser to access websites using cloudflare.
I'd expect it to be based on the Accept request header (and possibly the Content-Type response header to prevent bypassing it). Or perhaps even the Content-Type of past requests to that url.
But I don't really understand what the purpose of these captcha checks is in the first place. Handling a captcha challenge is more expensive for the server than most GET requests. Perhaps it's done in anticipation of later POST requests (which can't be blocked transparently without breaking the functionality of the website).
I've been using Cloudflare for a site that has around 400k requests per day on a Free plan (although I did buy a dedicated certificate and a domain from them) and Cloudflare has been very reliable to me except few hiccups in the past. I've had an instance of Cloudflare failing only on a single region, but after talking to support it turns out to be some route caching issue on their side, which subsequently fixed. I would not assume anything until there's some clarification on Cloudflare side. It might just be a bug.
I know from experience that they might take some actions if you take too much of a DDoS on their free plan but I never heard of that on some usual traffic, especially not when it's as low as the blog suggests (sub 10 rpm).
The attacks my blog received were in the thousand requests per second area when it got suspended.
If you're such a high volume site then why not pay for CDN/DDOS Protection? I'm amazed so many people feel this is something they deserve for free, and then complain when there's some kind of limitation on the free thing they're receiving.
If someone gave me a free car and I suddenly discovered that it had stopped working at 8:30am on Monday morning because the free car plan didn't support peak hour, just when I needed it most, I'd be annoyed as hell. This would be the case no matter how much I got paid at my job. Getting a nasty surprise because you didn't realise a product was inferior is upsetting for reasons that should be pretty obvious.
Admittedly the cloudflare free plan does say it's for things "that aren’t business-critical", but what does that actually mean in terms of resource quota and expected uptime?
I don't care as much for DDOS protection as for free easy SSL which gets semi annoying to set up for me personally. I assumed cloudflare was an option from their advertisment and never worried about it. Now I suppose I will.
I thought about responding to some comments here but I feel like this point deserves a comment on its own: not to defend Cloudflare, but charging at least $19 / month, having 100 customers (as he said on the post) and not using a proper fully featured WAF to protect an enterprise product is... amateurish?
Also, if the infrastructure is in AWS, CloudFront would cost cents if he really does have "4 requests per minute" with full integration and logs.
I have the feeling that with the advent of cloud solutions, very big companies depend on these really small and very useful tools that disregard almost entirely a number of best practices and standards. The Solarwinds incident is going to happen a lot more, that's for sure.
Regarding Cloudflare: I'm fine with the free product not being great, but hiding the logs is not understandable. I bet a lot more people would upgrade if they knew.
Cloudflare doesn't offer a WAF in the free plan, only a DDoS attack mitigation service. This means it doesn't go into the application level (i.e. no SQL Injection protection or XSS). They do, however, offer a filtering capability.
You can see more in the Cloudflare page [1], they explain well what it is. None of the features they talk about come in the free version.
I've always seen free plans as a way to get started. A great way for a startup to get things up before they get money rolling in. Then as you grow, you outgrow the free plan but by then you have the money to upgrade.
2. Data collection (after all if you are not paying you are the product) There is monetary value in collection, observation, and tracking of a vast amount of internet traffic.
3. Education. If you get people just starting out in their careers using your product or service for they hobby, personal project, etc then when they have a business need for some like your product or service they will just naturally choose you. Adobe and MS has been doing this for decades with low or no cost education licensing
This is highly unlikely to be as simple as the author conjects, we're talking about a service that processes an enormous amount of traffic and if what the author suggests is true, would someone else not have noticed by now? It's certainly possible that different infrastructure is used by free/paid plans, and perhaps this specific site was hosted on an unhealthy instance. But we don't have any external data points here to analyse - only those reported by Cloudflare, along with anecdotal reports.
Edit: I see the title has now been changed, at least we're reducing clickbait.
You can blame Cloudflare for not meeting their promises on free tier. But you can't blame them for "hiding" the errors. You are responsible to monitor your service, and it's good practice to have that happen externally from your environment. If you have a public API, monitor it publicly. Things like Pingdom and synthetic monitoring exist for this reason.
> There's no scarier company online than Cloudflare.
Really? I don't think you're aware of what's actually out there.
Let's put it this way: Cloudflare has a straightforward business model. The fact that you can pay them is comforting. The big shady companies that you cannot even pay, yet still make money "somehow", are more scary.
what a load of horseshit, this is my free plan 30 day stats, 2 order magnitude more request, a order magnitude less cache hits, no request limit in sight
edit: changed bull crap to the more appropriate horseshit in response to this unsubstantiated-bordering-conspiracy blog post, because if the former is already enough to triggers the community downvoting brigades, no point in showing any restraint
Very few, we're a regional level startup, the CF map doesn't show it well because it's per state and not per pop on the free account, but basically all hits in logs are marked MXP or FCO
