So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.
But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number?
We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons...
On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.
> Fastly has been trumpeting for years that Compute@Edge
This is the main problem with Compute@Edge, "for years".
They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.
After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime.
Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed.
Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
> Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
I greatly respect Fastly and their technology, but they (and Varnish) are biased against small companies and make it impossible for me to sign up customers who would benefit from their services and one day might grow into big spending customers.
AWS has shown that the days of "min $5000/year" (as both these services have quoted me for individual products) can be over and you can be profitable. Companies who don't catch up with that are cutting off their future.
> Companies who don't catch up with that are cutting off their future.
I wish someone would smack Okta over the head with this and get them to figure that part out.
Yes, they're expensive, but the per-use pricing was acceptable for what we wanted. I didn't realise there were (non-public) minimum spend amounts until we hit the end of our trial period and couldn't find any way to pay them other than to contact sales.
Sales wouldn't take the order for just a handful of users.
I totally agree with this, Fastly makes it pretty obvious from their website they do not want small customers. There is no price list and the prices on domains and such is 10x Cloudfare. It is also difficult to se what type of services they provide, everything ends up in a contact us. They seem to have great product with Compute Edge but no free thier and fuzzy pricing.
If you read the discussions in the financial forums Fastly might be in a tricky place since their biggest customer is assumed to be Amazon and if one goes to their website https://www.fastly.com/partners/featured/ they show off all the biggest cloud providers and customers. Seems like Fastly are afraid to steal customers from their partners.
As someone who chose Fastly over Cloudflare for a small-medium business after a long, long deliberation I disagree.
Cloudflare's pricing is sneaky. They are good for small and personal projects with their free tier, but as soon as you need anything non trivial you get hit with their $xxxx/mo minimum commitment, "contact us for pricing" Enterprise plan. On Fastly you pay $50/mo (peanuts for a business) and that's it for basically anything you might want from a CDN, VCL is endlessly flexible and free. That $50 buys you access to a featureset that is arguably beyond Cloudflare's Enterprise plan. You will need quite a lot of traffic to go over $50, so overall it works out significantly cheaper for an SME that wants to use anything but a very basic CDN.
It's also important to remember that a lot of cases are perfectly served by VCL, I believe most businesses won't even need C@E. Sure, most people would need to learn a bit about it, but it's amazing to have it on your disposal for free compared to a usual rigid configuration.
Besides, something they don't advertise but is now the main selling point for me is support. Fastly was the only company I had pleasure dealing with where after shooting a technical question to their free tier support I got not just a technical answer, but a piece of code that solved the problem I had. In an hour. With a link to their Fastly Fiddle demonstrating how it works. When I encountered what looked like a bug, a few hours later their support person filed a PR to their TF provider fixing the docs. It was and is a breath of fresh air compared to my experience with any other SaaS, Cloudflare included.
I think the only thing that could've swayed me is Cloudflare's Chinese network. It's gated behind their Enterprise plan and they won't hold your hands much, but if you need it it's there.
> They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.
This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on.
(I'll note that this is separate from the problem of AI-based fraud detection run amok.)
I strongly believe that there will be new laws that adapt to the SaaS like business model.
Most SaaS are structured in an effectively "non competitive market" kind of way. Lots of information hiding (don't post prices), high costs of switching (vendor lockin), and so on. These type of things stifle competition and lead to a more pseudo Monopoly kind of state rather than a truly competitive market.
Anyway, transparent pricing should be required for all businesses IMO. The problem is particularly bad in healthcare, and big contributor into high costs.
Of course, it will take a few decades for laws to catch up
In their annual reports Fastly have previously said C@E was available. They regularly released PR pieces of new functionality on C@E.
The reality was it was in private beta and deep in active development. Their official annual reports and PR pieces used by investors for investment decisions where misleading insinuating a product was available that was not. Fastly annual reports where painting a misleading picture. What's worse is the honest answer, we are deep in development and now have several large existing customers onboard for private beta is nothing to be ashamed of putting out to your share holders. Fastly where probably trying to hide the amount of time it's taken them to roll C@E out since first announced early 2019.
I'm sure every piece of public information went through their legal department and vetted but at the same time every piece of mandatory training I've done at publicly traded companies have me believe I could suffer huge legal consequences with the SEC for doing similar.
It has been two weeks since Fastly's VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly's claims.
Are you going to remove the ToS clause or issue a correction on the blog post?
Sure, but you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks. I think the community here calling out the double standards and asking for an update/response is entirely fair. If you didn't want the flak then leave off the "I'm the tech lead on CF workers" intro. Seems to me like the ball is in your court to at least try and make your advice actionable.
I do not believe I suggested doing anything against the ToS. I think you're misinterpreting the clause. But not being a lawyer, I don't really want to get into that discussion.
If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
> I think you're misinterpreting the clause. But not being a lawyer, I don't really want to get into that discussion.
The clause says "Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"[1]
IANAL, but this seems to very unambiguously prohibit benchmarking Cloudflare's services unless you have written permission. I know you don't want to get into an argument on HN, but could you like... bring it up to someone inside of CloudFlare who would be capable of changing it? You can point to this thread about how this clause is generating negative publicity.
I am a lawyer and also the CEO of Cloudflare. I have no idea why that clause is in our ToS. It was a surprise to me when it was pointed out recently. Not sure when or why it got included. Best guess is it was when some “stress tester” services decided to “benchmark” us by performing DDoS attacks and we thought we needed another justification to shut them off. Has been a loooong time since we worried about such things. Regardless, we decided weeks ago we’re removing the clause during the next ToS refresh. That’s scheduled for the coming weeks. And, in the meantime, have no issue with anyone benchmarking our performance. And seems we should do a more thorough and unimpeachable set of comparisons ourselves. Stay tuned.
I appreciate this reply, and hope the initial engineer gets no flak for his personal opinions attempting to defend your company. It's nice to see a tech company with employees defending it and leadership making such public statements as this.
I upvoted Kenton’s post. He’s the reason Workers exists. Surprised anyone worried about him commenting. I’d only be worried if I were a competitor who pissed him off by publishing BS stats. I’d imagine there’ll be an incredibly thorough and totally unimpeachable benchmarking study that comes out of this. And, anywhere we’re not the fastest, we soon will be. Game on.
Truuuue. I suppose I meant cookies are normally an encrypted ID with a user salt. So what would be the harm? But you're right; my actual question was something else.
Yep. When you're the tech lead for a significant feature in a very significant company, indeed they are one and the same. Whether you want them to be or not.
Not quite at the level of a public figure (Pres. Biden can't go around making flippant comments) but more than being just a private citizen or "I just work here". No amount of disclaimer can remove that, and for better or worse it's part and parcel with the job.
It's not the same. The lead dev is not responsible for all marketing around his work. That responsibility lies with the author and any editors of the blog post. Ultimately the CEO can be held responsible for both dev and marketing and they already did own the TOS issue here.
At the same time can you change 2.2(a) and remove the "or sign up on behalf of a third party"? My clients are not technical, and when they do manage to sign up they then email me their login details in plain text...
Without this in your T&Cs I could create the account for them in a couple of minutes. And avoid doing a screenshare to walk those who fail through the sign up process.
If you're employed by your clients and do it in their name, doesn't that make you the first party? I'm no lawyer but I can imagine it's to legally be able to close off bots and other shady services.
> If you're employed by your clients and do it in their name, doesn't that make you the first party?
If you're an employee, yes. If you're a consultant, contractor, freelancer or similar then you are a third party doing as you do it on behalf of your client (the first party). This is for UK law, and the distinction of first/thrid party is important when it comes to tax (see IR35 for the mess created).
Perhaps you can have them provide an access key instead? I vaguely recall seeing a button on a 3rd party platform that let me configure my DNS in Cloudflare to route to the 3rd party. Not sure how that flow worked to be honest, but I believe there is some programmatic way to delegate.
I’m just reading, but I appreciate coming to the comment section of a post here and seeing I’m this interaction, the CEO of a company like CloudFlare posting and, to top it off, posting something like this.
As for "stand-up" with Fastly, I believe the whole situation brings only negative consequences on both parties. I always become wary towards any service that posts comparisons with its competitors (or simply with services of similar nature).
> Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent
This - I mean even the conversation prior to this message - surely constitutes "expressly permitted in writing by Cloudflare"?
I would not be surprised if the person on the other end of that conversation would take poorly to the existing contributions made to it by people who have publicly identified themselves as Cloudflare employees.
First off, just want to say thanks for your posts, I found they give useful context and I really appreciate them.
I don't want the following to come off as unnecessarily argumentative, but regarding the ToS, I'm not a lawyer either, but my "ability to read English" interpretation of the section on "perform or publish benchmarks..." certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn't publish it.
Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it's fair to call out CloudFlare for this.
Plus it was there to prevent DoS attacks which is understandable.
Even without the ToS language, if you're really going to stress test a service, it's probably a good idea to give them a heads up, lest you get marked a bad actor.
He still should take some responsibility for it being there in the first place. "Our legal department insists on adding user-hostile clauses everywhere they get the chance" is an OK excuse for a Cloudflare sales rep or engineer, but it's disingenuous coming from the guy who is in a position to tell them to take a friendlier approach by default.
I'm assuming this isn't the only overly restrictive clause in the contract. Maybe it's an anomaly in an otherwise respectful ToS.
He’s the CEO. He acknowledged it and is fixing it. There is no more responsibility left unclaimed.
We aren’t owed a historical explanation, and yet we’ll likely receive one with what I presume will be a TOS update blog post in a few weeks.
I feel like this is that moment where someone lays on the car horn because they want to be sure the other driver understands that they’re a bad person, and should feel bad about themselves. It’s not about making you right by their actions, it’s about making sure they know the depth of your anger at them.
That has little value here. It’s socially valuable in interpersonal interactions, but it’s a tire fire when left uncurbed at Internet scale, and becomes vitriolic and harmful to discourse.
I may have misunderstood your specific intentions and desires from the CEO, and if so, I apologize; but I stand by my point in the general sense for all of us.
You suggest performing one's own tests to evaluate performance of the services, which fits the definition of benchmarking.
The ToS explicitly say that performing any benchmark analyses or test relating to the services is not allowed without written permission from CloudFlare.
99% of the people reading and attempting to abide by the CloudFlare ToS are not lawyers, but as a rule contracts mean what they say when they say it clearly and unambiguously as this seems to.
> If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
The way I usually see people handle this, and what I do myself, is to both state your relation to the company and clarify whether you're speaking for the company. Ex: "I'm the tech lead for Cloudflare Workers, but speaking in my personal capacity..." or "I'm the tech lead for Cloudflare Workers (speaking only for myself) ..."
It's generally understood on HN that when someone says "I work at X" that they are not speaking on behalf of the company.
On top of that, Kenton is a frequent commenter here, and I've never gotten the "air of superiority" vibe from him where such verbosity would be necessary.
This is true, but it's better to just assume people aren't speaking in an official capacity unless they say they are. This would also shave 30% off most Twitter bios.
> If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
I think the nuance is that you are presenting yourself as someone with responsibility/authority/control over the subject of these benchmarks. As a comparison, consider wording that skirts taking up that mantle:
Full disclosure, I work for Cloudflare, but ...
Not trying to be argumentative and really don't have any hostile feelings or intent. I understand where you're coming from.. just providing my outside take on how the interaction appears. You're within your right to defend your product. Nobody seems to have a problem with that. But you also decided to throw mud on the pile, metaphorically. You admitted you aren't plugged into the back and forth, that's fine. But this isn't news for Fastly and it's hard to take your side in this discussion when your solution is "go do your own tests" which is exactly what Fastly is would like to do. They clearly call attention to your ToS preventing them from doing that in the piece we're discussing here.
I think a lot of us have run into this exact clause because of Oracle and then other database vendors. We are well aware of what happens when you violate one of these clauses.
i think it's good there was full disclosure about your employer, but it also set the post up for me as "i am wearing my cloudflare tech lead hat", as nowhere in your post you state you are speaking for yourself and not as an employee... very confusing.
In general, the situation is the exact opposite: companies want to claim authority, but when an employee makes a statement they don’t like they want to have the deniability of “oh they weren’t representing us, if you want our real opinion please talk to our spokesperson”. Corporate PR is a strange mix of wanting engagement but also being incredibly risk-averse, and it’s very different from how people typically communicate.
I’m not sure how much authority being tech lead confers? It’s the lowest possible line management position. I wouldn’t expect a tech lead to have any influence whatsoever on legal, contractual or communication issues
If you know a little about how corporations work, than a tech lead is not responsible for a ToS, probably doesn't know anything about it and that's fine. Since it's not expected either.
It's the legal department... And the CEO already mentioned that they are removing it and he gave a valid response/reason.
It seems that they will actually benchmark Fastly in detail now ( could be after another improvement week), which probably isn't what Fastly wanted.
Something definitely seems to be happening if you read their response and i'm awaiting it with actual stats!
My experience working at tech companies is that the tech lead, or anybody at the company, can post in an internal message board or slack to ask "what's up with this weird clause in our ToS" and expect an explanation.
It's nice that eastdakota responded here, but he had two weeks since the original tweet thread from Fastly's VP of Eng calling out the problems with their benchmarking. They didn't respond or retract the blog post in those two weeks.
As a cloudflare shareholder (and a fastly shareholder), I want Cloudflare to act ethically and either retract the blog post or issue a correction.
Cloudflare's blog post still says, in bold, "Cloudflare Workers is 196% faster than Fastly’s Compute@Edge based on the time to first byte from the tests we ran on 50 nodes using Catchpoint’s data from across the world".
It is unethical to leave that up after Fastly pointed out core issues with the benchmarking, like using a free tier that was rate-limited.
Cloudflare's test compared the free tier of both services. The post was explicit about this. Workers free tier has limits too, and we would certainly have preferred to use the paid version of Workers in our test, but as the paid version of C@E is only available with an enterprise contract, the only fair test we could run was between free tiers.
Incidentally, this means Fastly's blog post is currently displaying test results that compare the enterprise version of Compute@Edge against the free version of Workers. Granted, our bad for the ToS clause, but still.
Despite the strong language in their post, Fastly has not actually demonstrated that anything was intentionally biased or unfair in Cloudflare's test. They've only laid out their opinions as to what would make a more representative benchmark. That's a debate you can have about any benchmark, but that doesn't somehow make the original benchmark "unethical".
It's not the benchmark that's unethical. The unethical part is leaving up the original claim without adding a correction or a note that addresses the problems Fastly found with the benchmark.
But it still compares both free tiers as he mentioned. It seems dubious that you demand to compare Fastly's paying tier with Cloudflare's free one ¯\_(ツ)_/¯
Cloudflare's free tier does not optimize for speed/performance, but for available ( = unused ) datacenter capacity based on location. Which makes it less fast than their paying tier.
Additionally, there will be an update soon as mentioned before, based on past comments.
There are other things that you ignore/are unaware of that are not even mentioned in Fastly's post. Eg. That cloudflare also optimizes their network for routing to denser cities instead of rural areas. That metric is not even mentioned by Fastly...
Note: i don't know any inner workings of them as I don't work there. It's based on what I remember from their blog about their SDN and performance weeks. I suppose it's applicable to this scenario, if i got the details right.
I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS.
Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different.
Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare.
But I agree everyone should run their own benchmarks.
The material commonly mentions "isolates" or V8 taking 5ms to cold start, which seems clearly aimed at Workers.
But the same point applies when comparing against Lambda. It's not a cold start if the app code is already resident in memory in advance. Workers and Lambda also proactively load some apps in advance, and we don't call it a "cold start" in those cases.
They don't forbid it. They request you get permission first. No one seems to have bothered asking to find out if CF say no.
Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future.
This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.
It would be nice for the first person to actually own it and not just say "I have no idea who put it into our TOS". Besides, it's either disingenuous or something is seriously broken in Cloudflare if their CEO is more attentive to HN comments than their own SMM people, as Fastly called that clause out on Twitter weeks ago.
Anti benchmark clauses usually prohibit publication. This wouldn’t exclude running two systems with your workload and documenting the results for internal use.
I was about to disagree, but I just re-read Cloudflare's language, and you're right. Cloudflare forbids performing any tests or analyses, whether or not you publish the results. Even running something like `curl http://my-workers-site | time`, for your own personal curiosity, is in violation of their ToS. That's even more egregious than Oracle DB's infamous DeWitt clause.
Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)
Interesting, AWS terms of service seem more reasonable, allowing you to perform benchmarks at will, but requiring that you disclose to them the methodology you used to perform the benchmark. Seems like a more reasonable clause while still preserving the ability to respond to hostile or inaccurate benchmarks.
I think the way AWS does it are great. I understand why some companies doesn't like benchmarks, mainly because it is subject to marketing spin and picking specifics without context. But giving out exact methodology so others could repeat the test seems fair. Basically the whole benchmarks should be open source if you do intend to publish it.
I any civilised jurisdiction such ToS are unenforceable.
Why do firms continue to make unenforceable claims? (My least favourite are restriction of trade clauses in employment contracts that are simply illegal, unless paid for, where I come from. But they appear all the time in contracts)
I just skimmed the ToS, the only remedy they offer to a violation of the terms is a denial of access to their service. Which is something they definitely can enforce.
Because it requires lawyers to prove that the claims are unenforceable, and most smaller companies aren't going to have the pockets needed to sustain the legal back-and-forth, which forces them to settle even if the big company is in the wrong.
What do you mean unenforceable? In many jurisdictions it is a criminal offense to violate the terms of service of a web site. In Illinois the first violation is a misdemeanour punishable by a year in the county jail. Further violations carry 2-5 years in prison.
A ToS is limited in what it can legally contain by law.
The EU e.g. has 93/13/EEC aka "Unfair contract terms directive" which bans a lot of shenanigans in ToS (e.g. mandatory location for law disputes). Or just writing in your ToS that "you agree that contract parties are not bound by the EU GDPR" for another example, like some companies do write, doesn't mean this is a enforceable clause. Another example, Germany recently made a law that mandates automatic subscription renewals - such as mobile phone contracts, gym memberships or online services - can be cancelled every month (there already was a law before that limited the initial subscription contract to a duration of max 2 years).
If there are some laws specifically that disallow general "benchmarking" I don't know, but I wouldn't be surprised if at least in the EU such a clause would be unenforcable. That sounds like a clear unfair one-sided advantage (the customer cannot check the services actually provided match what was promised). Publication of such results is another matter.
Also, I am curious about what you said about Illinois? Sounds bonkers. I mean I could imagine criminal prosecution for a set of defined, deemed specially bad violations that constitute crimes, like "hacking" and "sabotage", but not for things like e.g. "failure to pay membership dues in time" and other civil matters. Then again, I remember that case of some guy in Florida who couldn't keep his lawn nice and green, and didn't have the money to replace the lawn again and again, so the HOA took him to court, he was ordered to replace the lawn (which he still couldn't afford) and ended up in prison for a couple of days for "contempt of the court" until some friends and neighbors replaced the lawn.
Among other places, in the United States. The CFAA [0] prohibits "access[ing] a computer without authorization or exceed[ing] authorized access". It's interpreted extremely broadly by the courts.
This means I could set up watchcutecatpictures.org with some fine print that prohibits watching more than 10 cats a day and call the cops on random people.
What legal reasoning exists to give companies this kind of unchecked power?
Just to make certain put a button that says "I agree" next to a link to the T+Cs.
>According to the court, DeJohn's claim that he did not read those terms was
>irrelevant because, absent fraud (which was not alleged), "failure to read a
>contract is not a get out of jail free card."
It doesn't need to be legally enforecable to be useful.
At the very least, it is an indication of where they stand. They are free to take any customers they want, and if they say they don't want customers with a certain use case, that's good enough for me.
> I've been saying for decades that benchmarks are almost always meaningless
I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred.
I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.
Oh absolutely, benchmarks are great for measuring your own performance and improving it.
I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks.
The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
It is extremely helpful to be able to estimate performance without building out a whole system. For instance 10ms vs 100ms vs 1s latency makes a world of difference as far as what is possible and what component might be the bottleneck. Though, I agree that the “horse race” between several highly optimized and broadly similar libraries may be far less interesting
Unfortunately not all companies follow your great obvious recommendation and listen to snake oil salesman dressed up as consultants and industry analysts:
So, I would encourage everyone to run your own test and don't believe what either side posts.
---
Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.
Strange statement. Didn't Cloudflare start the mud slinging?
I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.
Sounds like there's a lack of communication if Cloudflare published a benchmark blog post without the tech lead even knowing (better yet, finding out about the competitor's post first!).
I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post.
Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not.
Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.
I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.
I made a very small edit to one sentence to change "don't believe either side's benchmarks" (or something like that) to "don't just go on other people's numbers", since the latter better reflects what I meant -- not that the numbers are false or intentionally misleading, but that they probably aren't reflective of your particular use case.
I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all.
This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.*
A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened.
* intentionally vague here, in order to give you space to delete and leave no record
> their post doesn't come off like mud-slinging, at all.
The post's current title is Lies, damned lies... and the article's synopsis is,
> This nonsensical conclusion provides a great example of how statistics can be used to mislead. Read on for an analysis of Cloudflare’s testing methodology, and the results of a more scientific and useful comparison.
I think you misinterpreted me a bit. I don't believe CF's benchmark was designed to be misleading or biased. I think the benchmark had a perfectly reasonable setup and was actually representative of one relevant use case -- the case of an app that just returns a response without a lot of processing. I think Fastly's criticisms of the benchmark are surprisingly weak compared to the strong language they are couched in, and the strong language itself is what I refer to as "mud-slinging".
My point about benchmarks in general was that unless your application is extremely similar to the one tested -- which is unlikely -- then these numbers don't really tell you much about how your own app will perform. This is a criticism I have of every benchmark that attempts to compare performance between very different technologies. I don't think it makes such benchmarks unethical, just not very useful.
I don't understand why you're arguing for the parent comment to be deleted, it's useful to hear from someone, (especially an engineer and the tech lead) who is working on the product. I hope that the comment is not deleted.
But the title of the post validates his opinion and his claim about fastly and cold starts can literally be found anywhere in their blog and the compute@edge landing page.
I suspect you're mixing up microseconds with milliseconds.
5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...
> In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.
Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor.
You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product.
Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.
My manager's manager is the CEO. Yes, we all discussed it and everyone agreed the clause should be removed, but at the time of my comment I wasn't sure if I was authorized to announce that publicly. He has since posted in this thread.
If you bother to read the thread the CEO has already commented to say they're changing that ToS. Why give advice that no-one asked for and no-one needed? Isn't that pretty much the worst?
So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.
But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number?
We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons...
On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.