Cloudflare served private HTTP traffic in response bodies, meaning that website results contained cookies, session data, encrypted traffic, all personally identifiable, and because it was served as response bodies, it was *indexed by search engines*, not to mention anyone else who was scraping websites during the time of the incident. It included credit card information, frames from videos, PII, the works, all linked to individual users.
This was ongoing for *months.*
Anyone savvy could use this information to hijack accounts, scrape personal information, view private browsing habits. Even when Cloudflare publicly announced it (and tried to blame others) when they thought they had cleaned up most of the data, you could still easily use search engines to find people's personal information by searching for the Cloudflare header strings that started the leaked session information.
Many countries have legal policies around data breaches, including required disclosure policies and penalties. In the greatest blind eye turn of the history of the internet, Cloudflare managed to get away with a single blog post, and no other penalties. https://blog.cloudflare.com/incident-report-on-memory-leak-c...
THAT is Cloudflare's disruption.
Eh, somewhat. AWS is already modular in a lot of ways. You want S3? You got it, no matter where you are. (We're talking after them doing some sort of fee drop here.) You want to run exactly one EC2 instance? No problem. You want a message queue? You don't need anything else. You can integrate it with the notification service but it's optional.
Sure, some of their services are integrated, but a lot of that integration is just "this service pulls from S3 and writes to S3", not massive integration at every level.
There is some stuff that is deeply tied in, yeah. But it's not like every single AWS service is deeply tied into half the other ones and the moment you open an EC2 instance you also are buying into a dozen other services. (It may feel like it if you put together a network and override the default block storage, but that's really just giving you knobs that are simply preset elsewhere, not really "lockin".) A lot of it is already pretty modular.
With the huge margins they must have on egress bandwidth, I'm not holding my breath though.
If your data is in S3 or DynamoDB, egress fees will encourage you to process the data within AWS. You’re not going to use BigQuery for that. If you want to add a search index on that, you won’t go shopping around for search products on other clouds, you’ll likely use AWS elasticsearch. That’s where AWS makes money - through this soft lock-in.
They’ll survive and maybe even thrive if they lose the revenue from egress. But who knows what’ll happen if the moat is destroyed? Every AWS service would need to compete on merit with every service on other clouds. They won’t be chosen simply by default.
Euh, 4-80x , the 80x is for EU and the US.
I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.
They very much have massive margin on egress. Given some of the cost comparisons floated today comparing R2 to S3 egress AWS is likely hitting 1000s of times (likely more) their return on the actual bandwidth cost they pay for month over month.
Usually DoS attack doesn't exhaust the bandwidth, come on..
Come on, Amazon is not serving traffic through a Comcast business connection, they're peering directly with other large operators for free or for next to nothing.
5Gbps*1 month is ~1.5 PB. AWS is about 0.02/gb or ~30k/1.5pb.
Approximately 30x the cost.
These are old numbers for specific use cases so I'm not sure how much that has changed.
but regardless, the EGRESS charges are what are absurd. The only logical reason for charging so much for data exiting is to make sure that you can't practically leave the AWS ecosystem.
- A 42U cabinet
- A 15A 120V circuit
- An unmetered 1Gbps IP transit link
All at a proper datacenter, namely Hurricane Electric's fmt2 facility. Includes a /29 of IPv4 and a /48 of IPv6, allows me to announce the /24 and /36 I own, and there's a free internet exchange onsite which I have a 1Gbps port at.
You need to provision enough power for simultaneous startup after a power outage, unless you have some really smart PDUs and automation. We have a 10 year old DC with 30A@208V per rack and we have to leave racks half full because modern servers are so power-dense.
In any case, even at your optimistic 250W per rack unit, a full rack of servers with two switches is more than 10kW!
You could possibly run this fictional daemon on your management switch :)
I think this is the most common “enterprise” datacenter server type in 2021, mostly due to licensing constraints from VMware/Microsoft/Red Hat/Oracle/etc. Such servers give the most “bang” per dollar when licensing costs are included.
https://i.imgur.com/Zr5rq4X.png a graph if you're curious :)
Just one example is Hetzner, who include 20TB of bandwidth, with anything over that charged at just €1/TB.
Meanwhile, AWS is gouging at €80/TB.
The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe.
If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps.
(Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)
Remember those ISPs need to make margin, that's why they charge what they do.
AWS _could_ make it's margin on services alone. They just choose not to.
Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) against the disk storage at another cloud provider (Amazon).
Even if egress were $0, I still wouldn't want to do that. I want queries to run as fast as they can and the WAN link bandwidth is opposed to that.
Is there anything about BigQuery that would compel anyone to do that instead of just using AWS RedShift?
Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects.
A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."
So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.
You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices).
So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that.
In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) . These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy.
I had a huge corpus of data and ran some instances that were basically downloading+parsing+summarizing data for a couple days. Again, I didn't think of bandwidth as they were EC2 instances querying S3 objects, what could go wrong, it's Amazon to Amazon, right? Wrong.
Bill came up in the 10,000s USD range ... fortunately, it happened during a trial period where I could spend a lot of money for ~three months.
I moved my stuff into a small-ish cluster of unmetered VPS and the whole thing is 100x cheaper.
To do anything more than a demo, Polly requires S3. To get a notification of a completed TTS synthesis, you must use AWS SNS. To get logs you have to use their logging service.
I suspect the primitives, like these required ancillary services, are the ones not tied in to each other. But how could they? (AWS probably has found ways)
For my project using Polly, I didn’t care. It was kind of interesting to explore AWS some.
But I’d guess a lot of folks might not want cloud service primitives. They want cloud products.
If they must use primitives to enjoy cloud products, they should want flexibility at least to not be stuck with pricing and feature set on primitives if they don’t compete.
They almost compete with everyone now.
DNS: They eat simpledns lunch
Pages: They eat Netlify lunch
Worker: They eat serverless/lambda as in AWS/GCP lunch
R2: They eat AWS Lunch
Email Forwarding: They eat ... my own lunch (I'm founder of hanami.run an email forwarding service)
That's being said, from a user perspective, if my domain is already on CloudFlare, I can just host everything on it.
Right now, cloudflare workers is pretty great to add some dynamic stuff. And pages is great for static site.
Will also check on NS delegation. Again, my hunch is that we only charge for it because it's something that less sophisticated users we worry would get themselves in trouble messing with.
It does. IIRC GitLab Pages used to have docs saying to set a CNAME for your domain without warning about doing it on the apex domain. I'm not sure if GitHub Pages was any better either.
It's only confusing the first time you mask your MX records. LMAO.
To me, my biggest pain points is I cannot set NS record for a sub domain.
My use case is this: I had a certain subdomains where I want to use LetsEncrypt DNS with DNS validation. I don't want to give the whole domain to the auto renewal script. With another DNS provider, AWS Route53 for example, I can easily create another zone for that subdomain say blog.domain.com and set NS record on blog to that zone. Then create an API key with only privileges to manage that sub zone. I cannot do that with CloudFlare though.
Agree on Cname flattening. It cause some issue for my email forward service in the past.
So we have customer use githb page and set CNAME on apex domain. Then they add a MX record for apex domain. CloudFlare UI allow them to do that. But upon resolving won't return MX records for the apex. So my customer aren't able to finish setup. Eventually we have to set the A Record on Apex to an IP.
Are you sure? I have a free site where `in.example.com` is delegated to Namecheap's FreeDNS so I can use it for DynDNS on some devices that don't support Cloudflare's newer (fine grained) API tokens.
(I know there's an import but it erases some of the other settings)
$0.99 domain providers mostly get this right
Aren't Cloudflare Workers a very specialized kind of lambda that's severely resource constrained and whose runtime is capped at 15ms?
If anything Cloudflare Workers compete with Lambda@edge, but it's very disingenuous to compare them with AWS Lambdas and it's completely absurd to claim they eat anyone's lunch.
Cloudflare Workers's usecase is extremely limited and specialized: run a script comprised of a couple lines of code that do not do much at all right at the edge. We're talking about things like adding a response header. Even then they are immediately killed if pretty much they don't exit immediately.
The all-inclusive Lambda workers are limited to 50ms of actual CPU runtime and can execute forever (i.e. hours) for IO bound workloads, as long as you stay below the 50 network requests per execution. And for that they cost $0.50/million, have unlimited in/egress bandwidth and free in-DC caching.
But they also have a more AWS-like pricing option that’s about 20% cheaper and charges per request, per GB-hour (for runtime, not actual CPU usage) and for bandwidth with a maximum runtime of 15 minutes.
They also have Durable Workers which provide you global singleton persistent functions for stateful architecture.
If you haven’t had a look at CF’s serverless stuff for a while, it’s worth a look again.
That's especially true if your workload makes sense for the all-inclusive Workers. I evaluated only the pricing a while back and Cloudflare Workers are far more attractive than anything else in the market IMO.
With AWS and Azure, it's really hard to calculate just how expensive things are going to be. I'd say it borders on impossible without just running your workload for a bit and waiting for the bill.
With Cloudflare Workers, it's dead simple. As long as your Worker runs in <50ms, it costs $0.0000005 per run. I can tie that directly to (page) hit counts and calculate costs with very little effort.
For my own reference point, I ignored the fixed cost per run, which is actually more expensive for Lambda@Edge, ignored the variable cost per run, which actually has minimums for Azure Functions, and calculated the egress cost per byte.
Assuming they use GB and not GiB for egress, it's $.09 / 1,000,000,000 = $0.00000000009 per byte. Now take your Worker cost and divide it by the per-byte egress cost and that's 0.0000005 / 0.00000000009 = 5,555.
That's <6KB of egress for the same price as a Worker run on Cloudflare. Even if AWS and Azure started offering free Lambdas / Functions, it would still be a bad deal if Cloudflare Workers meet your technical needs.
I'm not sure you're looking at the problem right.
I mean, if you write your AWS Lambda code to do the same thing Cloudflare does to their workers in their Bundled Requests pricing model and automatically kill them if they reach 50ms, you also get a dead simple way to know what you pay per request.
However, Cloudflare also charges per request and per execution time their Unbound requests pricing model, which leads us pretty much to AWS Lambda's pricing model.
Cost for log processing is $0.50 per GB and the minimum size of logs (just for the START/END/REPORT lines output by Lambda itself, before you start logging any of your data) is about 260 characters (or 260MB/million == $0.13/million).
Honestly, unless you have gone out of your way to implement something that isn't CloudWatch for logs (and there's almost no documentation on how to do that), its not hard to get an extra 5-10KB ($2.50-$5.00/million to process by CloudWatch) of logs per request.
Like AWS egress charging, CloudWatch Logs can quickly dwarf the cost of using the services themselves.
The really are. By Cloudflare's ow docs, Cloudflare workers are just scripts that are designed to be executed before a request hits the cache.
Quite the far cry from what AWS Lambda offers, which is a generic compute platform that handles both long-running batch jobs and handles events, and can be invoked any way that suits your fancy (HTTP request, events from other AWS services, AWS SDK).
At most, Cloudflare workers are comparable with Lambda@edge.
> The all-inclusive Lambda workers are limited to 50ms of actual CPU runtime and can execute forever (i.e. hours) for IO bound workloads, as long as you stay below the 50 network requests per execution.
You are right. According to Cloudflare's docs, Cloudflare Workers are capped at 10ms CPU time on their free tier, but their Bundled Usage Model plan bumps the CPU time limit to 50ms. There's also Cloudflare's Unbound plan which not only charges per request but also adds charges for duration instead of CPU time (i.e., also charges for idling time when waiting for responses) and that's bumped up to 30s.
> But they also have a more AWS-like pricing option that’s about 20% cheaper and charges per request, per GB-hour (for runtime, not actual CPU usage) and for bandwidth with a maximum runtime of 15 minutes.
No, not really. Cloudflare announced a private beta for their Cloudflare Workers Unbound Cron Triggers a couple of months ago, but that's about it.
I'm not sure how familiar you are with AWS Lambdas, but if you check their docs you'll notice that, unlike Cloudflare Workers offering, they are general purpose and can be even invoked from all kinds of events, including directly from HTTP requests. So, neither they are available nor are they comparable to AWS Lambdas. Thus I'm not really sure why you brought up something only made available through a private beta and is very limited in it's capabilities to compare with AWS Lambda, which is production ready for years.
I run datacenter(s) with thousands of autonomous machines. These machines run a small binary daemon. That daemon needs to check for a new version of itself, which is built/released as a CI push job on github (after all the tests pass).
A super simple CF worker serves as a reverse proxy to the GH API + the download of the binary. For $5/month, I've worked around the GH API limitations, in a massively scalable way.
Why not use package publishing tool like packagecloud.io (or setting your own private reprepro, dak variation) with unattended-upgrades configured for the repos and frequency you need ? Was the tooling at OS layer not adequate for this kind of setup ?
As developers we constantly discount the value of our time. Your time in developing/maintaining these scripts is not free.
From an org perspective, to maintain a home grown solution is not free either, inevitably someone would take over this part of your role, they would need skills they otherwise probably won't need to have (making hiring harder/costlier) and they will need more training and also will have to spend time maintaining it.
This is true even if you are the founder of the organization. I thought being a founder where I am going to go so early on built solutions like yours. It turns out that doesn't make any difference The role always changes . As an employee we become older acquire more skills/knowledge/experience and role changes or simply leave the org. As a founder we grow the company and have to hire more people to do what once we did.
FYI, running it locally would involve buying hardware and maintaining that hardware, at each datacenter (there are multiple). We don't have any server hardware at the data centers.
Yes that's my point. Unlike AWS Lambda, the usefulness of Cloudflare Workers is very specialized and narrow, like adding response headers or update a response document.
AWS Lambdas on the other hand can run freely for over 15min, have virtually no limit in how much RAM they can use, and can be pushed as a Docker image with a max size of 10GB.
If that is not enough, AWS Lambdas can be tied together into workflows with AWS step function.
Therefore, for anyone to claim that Cloudflare Workers win over AWS Lambdas, either they have no idea what AWS Lambdas are or have no idea what Cloudflare Workers are.
And no, it's fully act as a standalone app. I define the route to to route a part of traffic to the worker, other parts to our pages app.
For me, it works great and replace my aws lambda usage.
Also used them to return images from different providers, and resize them, on a single endpoint.
Plus, if you decide to pay a premium (but still often cheaper than lambda), they can run for just as much time.
All that’s actually missing here, for me, is the triggering mechanisms and integrations with services like S3, SNS, DynamoDB, etc.
Right now I have to use Amazon SES + SNS for this but seeing how cloudflare already has workers, this would be a killer feature for a lot of companies including mine.
> Never lost your emails.
Should be "Never lose your emails."
I think that big disadvantage in this approach is that they are not getting "mindshare", in contrast to that people are able to use Cloudflare serivces even for themselves and as they grow professionally CF's constantly increasing amount of solutions is there as something familiar, approachable and ready to use.
They were (are?) even more addicted to their egress charges.
Oh I never thought of that. So the next one is Q1 and final one would be P0.
> ...about once a week some character spots the fact that HAL is one letter ahead of IBM, and promptly assumes that Stanley and I were taking a crack at the estimable institution ... As it happened, IBM had given us a good deal of help, so we were quite embarrassed by this, and would have changed the name had we spotted the coincidence.
Love the company but the touch that the CEO gives a ____ really matters to me.
Oh wait, pretty sure those are EC2 instance types. :p
P0 would get disrupted by O-1
Indeed, it looks like "your margin is my opportunity" motto can work both ways for Amazon :)
Cloudflare has a lot to gain by fixing this.
I, for one, welcome our new Cloudflare overlords.
Without egress charges, assets in an object store can be backed at other providers for disaster/contingency/fault tolerance.
Add the excellent rclone tool which, I assume, will work immediately with (just another S3 compatible store) and there's a nice and easy workflow that adds some diversity to your infrastructure.
Definitely, appreciate Cloudflare bringing gun to a gun fight. GCP take notes.
That said, if CIO and CFO's are (truly) pissed off, then that is going to be a huge revenue swing for AWS shortly.
I personally doubt it. Azure and GCP are not that much cheaper here.
And AWS is offering some great pricing actually on things like ECS Anywhere, so now if you want it is (a bit) easier to bring a workload local to your data lake. I think that is not a great short term move by AWS, but long term helps with goodwill.
At the end of the day, you can either sit on the side lines and criticize the prices, or you can jump in and make money.
I think there is a decent reason why Cloudflare, at least initially, went the route that it did. V8 Isolates allow them to run code from many different people without many of the cold-start, memory, and performance issues of offering a more full environment. V8 Isolates allow them to be a lot more efficient than something like Lambda. It does come with the cost of being more limited for things like language support.
I definitely understand it being a turn-off. If you haven't read their post introducing them, I'd read it: https://blog.cloudflare.com/cloud-computing-without-containe.... It doesn't solve your problem, but I think it's a good read on why they made that trade-off.
All of Lumen’s value is in the nationwide backbone fiber although they are trying to make a play in edge computing. IMO those fiber assets in the hands of a company like CF combined with a service like Fly would be pretty incredible.
Huh, that's too bad! Meaning they've caused trouble for your customers? Definitely disappointing to hear...
For example, Cloudflare workers can't even talk to the outside world with anything other than fetch(). There's websockets, but only as a pair to talk to a browser that connected to you.
I'm a fan of Workers, but they do have limitations.
Wait, it's the opposite, at least on the infrastructure side. The Internet is increasingly centralized, due to Cloudflare and other big players.
Back then, using traditional IT providers or internal IT services, you needed weeks, paperwork, etc, to get any storage or compute.
Then AWS arrives, and you could have infinite storage, or tens of EC2 instances, within a few minutes. And pay with a freaking credit card!
It didn't matter that AWS' performance was abysmal at the time. Or that AWS was expensive. AWS solved a huge pain point for millions of people, and that's why it became a market leader.
Price is not the only thing that matters.
Price matters when all of your other advantages have been eroded away.
> I know why; I was there! (first AWS employee in EU, 2008, stayed there until 2014).
Btw, another of your claim-to-fame is that the current Amazon CEO follows you on twitter, one among the 300 odd accounts :)
S3 was orders of magnitude cheaper than anything comparable at the time.
Cheaper for cold storage. For anything that requires high IOPS or egress trafic S3 has always been very expensive.
If your load wasn't high, cause you're a startup or whatever, then paying the extra premium on the storage to save the engineering effort of building your own storage cluster worked out. Then when you were big, those egress costs had you locked out.
Plus add in thoughts about having to maintain infra vs aws do it for you and you had a lot of developer blogs/marketing sites of tech companies/whatever just serve on s3 since it was easy to use and the absolute cost for such a product is low enough that they didn't care about the relative cost of s3 vs other services.
Cross Region replication is typical setup for many DR use cases and S3 will charge you Inter-region transfer cost for replication as well.
Second tier Cloud providers would eat that up, since it would democratize things more. Even if a competitor gets the customer, at least it's not the guy who is waging a war of attrition against you.
For many organizations, the 24x7 staffing needed to provide an equivalent service alone would pay for their entire storage cost multiple times over. Even if your scale is sufficient to allow beating that, you are likely to have more compelling problems for that time to be spent on.
(This is not saying that the egress charges are great, only that I completely understand why many, many people decided it was an acceptable tradeoff)
Even the startups that grew, they might start on a VPS provider, but outgrow them. S3 managed to scale with them and retain them as customers.
That coupled with storage costs that were always very competitive and the fact you had unlimited scale and PAYG pricing got a lot of people hooked.
It’s going to take a long time for S3 customers who have experienced pretty amazing uptime and reliability for the entire life of the service to put the same level of trust in something else.
CF did a really smart thing by making R2 be able to operate as a transparent caching-replica of S3.
But doing durable storage yourself, especially once the amounts get a bit unwieldy is scary. For low amounts of data you can get away with just making plenty of copies in different places, but that gets much more difficult once it's a serious amount of data. Object storage is the most appealing service even if you want to do most of the stuff yourself. And this is an area where an established track record is important, you don't want to store your data if you're not sure the service is reliable.
With standard S3, that egress traffic would cost 45$ -50$.
Sounds like AWS is competing with itself.
Have bought since the IPO and am super bullish. It seems most Wall St analysts really don't understand the company or business.
I had a similar experience with Shopify. I had interactions with them in a company I worked at back in 2015 and regarded them very highly among the e-commerce platforms but didn’t buy the stock…
Wish I worked for them and was getting those sweet, sweet RSUs.
It's not undervalued in that quarterly revenue are $152.4 million, on a market cap of $35 billion dollars, for a staggering multiple of 57 times revenue.
Edit: Had quarterly earnings instead of revenue.
Cloudflare of course has no earnings. Their operating loss last quarter was $28.8 million. Their operating loss for the past four quarters was $106 million.
So whether or not Amazon intended it that way, it functions as something that’s anti-competitive because it forces you to go all-in with AWS.
Anytime a majority of developer job postings mention a specific product/company certifications, (think PMP, or Microsoft developer certs) , its time to pivot your skill sets.
I have had to write weird proxies just because of some missing AWS features, and it made sense because, unlike the last twenty years of my computing career, I don't have the source code to all of the code running in the environment.
So the author thinks that shared hosting or servers-for-rent did not exist before AWS' popularity?
I bought "The Innovator's Dilemma", read it, and said to myself "OK, we'll do that then".
Since you're here, would love for Cloudflare to disrupt the DBaaS marketplace.
I already run my entire business on Cloudflare (for services you have) but there is a significant portion of my infrastructure (>50%) I haven't moved over that is dependent upon needing a DBaaS offering. With a DBaaS offering, I could run near 100% of my infrastructure on Cloudflare.
(Workers KV is great btw, but there are so many times where just a traditional RDMS is needed that a key-value store doesn't fill).
A lot of ML startups end up buying hardware for training because they can get a GPU for what it would cost them to rent it for 2 months on GCP/AWS.
That being said, I certainly wouldn't mind seeing AWS and Cloudflare get into a price war that lowered egress prices.
Ugh - a clone of S3's functionality - that's not competing.
There's been zero innovation in cloud storage beyond S3's primitive capabilities. None of the competing services have gone beyond S3's stunted functionality.
Online storage should provide:
* An SFTP interface (and no, Amazon's "charge by the hour SFTP interface to S3" doesn't count)
* The ability to query and apply filters to queries PLEASE! For goodness sake its 2021.
* A webDAV interface
* The ability to incorporate object metadata into filtering queries
Why is there zero competitive drive in this space?
This is big and interesting and useful even if you ignore the bandwidth savings. If it were available already, we'd be trying to use it for Fly.io users.
I suppose it's now corporations stealing market share from each other...
Cloudfare is an exciting company with a lot of great products. But they're less than 1% the size of AWS or Azure. Let's see what happens.
Uhm, what did you think a market is?