
How Much Traffic is Too Much Traffic For CloudFlare? - phoboslab
http://phoboslab.org/log/2013/02/how-much-traffic-is-too-much-traffic-for-cloudflare
======
eastdakota
This was flagged to my attention and I've reviewed all the interactions
between the author and our team. The site in question was using the free
version of CloudFlare's service. On February 2, 2013, the site came under a
substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for
all customers (even those on the Free CloudFlare plan), for the mitigation of
large attacks a site needs at least the Business tier of CloudFlare's service.
In an effort to keep the site online, our ops team enabled I'm Under Attack
Mode, which is available for Free customers and enhances DDoS protection.

The attack continued and began to affect the performance of other CloudFlare
customers, at which point we routed traffic to the site away from our network.
While we encouraged the site owner to take advantage of the Enterprise tier of
service given their needs and traffic levels, the site would have been brought
back onto CloudFlare's network if they had upgraded to the Business tier of
service ($200/mo) which included Advanced DDoS mitigation.

To be clear, CloudFlare does not bill based on traffic. However, resources are
not infinite and when an attack against a Free customer begins to affect the
performance of other customers we will take measures to protect the overall
integrity of the CloudFlare service.

Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)

~~~
autotravis
You keep calling it an attack. This was normal (growing) traffic for the site.

From the comments on the site: "I didn't notice any "attack" when CloudFlare
began to route all traffic directly to us. It looked like normal web traffic -
much of it, but no more than usual."

~~~
thezilch
He said; she said. A Layer 7 attack is not necessarily something one might
"notice." The very nature of such an attack is "normal" looking. I think it's
impossible for us to say who is right -- OP or CloudFlare -- without
substantial hard-data. Ultimately, it's not the basis of the article, and the
OP is looking for a different service -- unmanaged, limited-downlink bandwidth
-- than what CloudFlare is looking to provide -- managed, edge network.

~~~
ersii
Irregardless if it's an attack or not, CloudFlare's personell did not handle
this well.

Yes, I know they did offer one hell of a starter/"sweet lolipop to sucker you
in" pack - but that's still not what's being discussed.

It has been re-iterated many times in this thread - but CloudFlare had a sane
person on the other end that was willing to open his wallet - that's something
one should act on quickly.

------
kevingadd
On the one hand, I think it's insane that you hit 100TB/mo of usage on a
"free" service and expected it to keep going. Maybe you figured you wouldn't
rock the boat as long as it was free - I guess I can relate. But still,
really???

On the other hand: CloudFlare comes off as terrifyingly incompetent here -
which more or less matches with my experiences trying them out on a site
serving maybe 100GB worth of traffic at most in a month. They seem to have
missed dozens of great opportunities to upsell you on their paid service, and
when someone finally noticed how much bandwidth you were using, they
completely lost the plot.

What should have happened, IMO, is something like this: A panicked CloudFlare
admin realizes your site is using 100TB/mo. Their first step is to send you a
sternly worded email, explaining that for this usage level you need plan X,
and if you don't upgrade within... let's say 5, maybe 7 days? They'll be
throttling or limiting your service. Then you don't feel pressured to solve
the problem _right away_ , and they are trying to retain a customer that
(presumably) they value.

Instead, they haphazardly change your settings behind your back (???) and then
later take various steps to reduce your bandwidth usage before finally
deciding that you need to pay them, without even figuring out how much money
they want. Ridiculous.

To me this just says that CloudFlare is running on what may be a fundamentally
unsound business model, and that by claiming their free tier doesn't charge by
bandwidth (and not listing any limits) they're dramatically increasing the
odds that customers will suddenly discover there are limits after all, and
leave. If they were more up front about what the actual pricing structure is,
it'd probably be more likely that people would start paying for the value that
CloudFlare gives them.

~~~
Cymen
The author stated a couple of times that they would happily pay for the
service. The problem is it was free until suddenly it was $3000/month and it
took two weeks with an outage in service for that duration to learn that fact.

~~~
aaronblohowiak
Most reasonable people can be made ok with most things given adequate
communication. Expectation management is one of the most important aspects to
human relationships. As a business, what was a sales opportunity became a PR
issue.

------
thomseddon
I had quite a similar experience where our site (approx 10TB / month) was
taken "off cloudflare" and all traffic routed direct to our servers at peak
load (we were on the Pro plan @ $10/month after many months of free). They
cited network issues in the control panel and a "Layer 7 attack" in a support
ticket. We quickly upgraded to a Business plan ($200 / month) and traffic was
back through cloudflare within 10 minutes.

Also, similarly to OP, we are regularly (fortnightly) automatically put in
"I'm Under Attack" mode without any prior warning or consent which is quite
annoying as it tends to happen overnight so I am not alerted until someone
checks the live site in the morning (it still returns a 200 so current checks
don't pick it up)

~~~
dlss
Sounds like the $200/mo is really worth it...?

~~~
thomseddon
We use it mainly for the CDN, we have estimated a similar service using
CloudFront would cost over $1500 / month (750% more - I acknowledge AWS is are
not the cheapest)

------
jstalin
The TOS seems a bit too illusory:

"SECTION 10: LIMITATION ON NON-HTML CACHING

You acknowledge that CloudFlare's Service is offered as a platform to cache
and serve web pages and websites and is not offered for other purposes, such
as remote storage. Accordingly, you understand and agree to use the Service
solely for the purpose of hosting and serving web pages as viewed through a
web browser or other application and the Hypertext Markup Language (HTML)
protocol or other equivalent technology. CloudFlare's Service is also a shared
web caching service, which means a number of customers' websites are cached
from the same server. To ensure that CloudFlare's Service is reliable and
available for the greatest number of users, a customer's usage cannot
adversely affect the performance of other customers' sites. Additionally, the
purpose of CloudFlare's Service is to proxy web content, not store data. Using
an account _primarily_ as an online storage space, including the storage or
caching of a _disproportionate_ percentage of pictures, movies, audio files,
or other non-HTML content, is prohibited. You further agree that if, _at
CloudFlare's sole discretion_, you are _deemed_ to have violated this section,
or if CloudFlare, _in its sole discretion_, deems it necessary due to
excessive burden or potential adverse impact on CloudFlare's systems,
potential adverse impact on other users, server processing power, server
memory, abuse controls, _or other reasons_, CloudFlare may suspend or
terminate your account without notice to or liability to you."

In other words, you can't host non-HTML, but you can if it isn't
disproportionate, but if it is disproportionate, they can deem you to be a
problem and cut off your service, without notice. That's not a contract at
all. In legal parlance, that's an illusory contract -- when one side can
modify their performance in any way at any time.

I use Cloudflare's $20 a month option and it worries me now that I might be
_deemed_ to be using a _disproportionate_ about of space or bandwidth caching
images, and then be cut off without notice.

EDIT: I love the cloudflare service and I'm not complaining. I just think
their legal department needs to clarify this and the tech side of the house
needs to be able to warn users when they are exceeding the bounds of what is
acceptable.

~~~
derefr
My interpretation of this paragraph goes something like:

> _If and when_ your bandwidth usage gets high enough that one of our customer
> service people gets pinged about upgrading you, they'll also have a look on
> the dashboard to see what mixture of filetypes you're serving. If it's all
> static content, you're in trouble.

I'm not quite sure how else that could be phrased into legalese, than what
they already have there.

------
sauteedbiscuits
Based on what I have read on many HN articles, what Cloudflare offers and what
you receive are two very different products.

From being shut off when you have a incoming DDoS of some arbitrary size to
actually loading webpages slower than your server does vanilla, it seems the
benefits of Cloudflare are mostly hype.

------
raylu
While I agree CloudFlare handled this poorly, there is this reply on the blog
from Matthew Prince (CEO of CloudFlare):

Both enabling "I'm Under Attack Mode" or routing the traffic direct are both
supposed to generate an automated message to the customer letting them know
what happened. We've reviewed the logs and don't see a message having been
sent. I'm investigating why that didn't happen since I agree it is not
acceptable.

------
trotsky
Very interesting article, thanks!

 _At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer.
Let me know if this works within your budget._

An interesting proposition - If we take it at face value, $3000 for 100TB
works out to be $0.03 per GB. That's pretty high these days. If you are buying
downmarket (which is cloudfare like traffic quality) you can get a cdn deal
for maybe $0.01 on a 6 month term with these kind of levels, and somewhere
around $0.005 for an xc in the us or eu no commitment. Cloudfare should be
buying at substantially better rates than these (or at least, they seem to
imply it - calling bandwidth free) so it's it seems they have a similar
problem as many freemium models - when most of your customers aren't paying
you have to really hit the ones that do.

~~~
Terretta
Even at $0.01/GBT, you're still looking at $1,000 a month. His conclusion of
"two servers and 100 TBT for $200" seems unlikely to last long either.

> _somewhere around $0.005 for an xc in the us or eu no commitment_

From whom? Or is this reverse engineered from a fully saturated link?

~~~
trotsky
Not extrapolated from a fully saturated link, no. But I didn't mean to imply
that was usage billing.

I was thinking of fdc in amsterdam in the eu on a 10gb unmetered no bw sla,
but I also think they offer a similar (slightly higher) deal in denver. I was
thinking of somebody else for the US - in SLC - but the name escapes me. If
you actually could use the name I'll find it.

I shouldn't have said xc as really they expect you to buy power and space from
them. Clearly not what I'd try to run an upmarket video cdn off of, but i
would be surprised if people like fdc aren't who CF buys from.

~~~
Terretta
> _unmetered no bw sla_

Those kinds of offers are generally unusable _if the bandwidth matters_. This
goes back to the oversubscription model, selling the same resource to multiple
customers and letting the customers jockey for use, hoping most will never use
it enough to catch on.

See this thread for more:

—
[http://www.webhostingtalk.com/showthread.php?t=1159276&p...](http://www.webhostingtalk.com/showthread.php?t=1159276&page=2)

Note this analysis:

 _“The FDC offer is a shared 10 Gbps and i believe in another topic was
explained, that you're supposed to stay at 1 to 1.5 Gbps usage. Even a pure
10Gbps peering port will cost more then $500,- / month, so a true 10 Gbps for
anywhere close to $500,- cannot be expected from any provider worldwide.”_

—
[http://www.webhostingtalk.com/showpost.php?p=8153934&pos...](http://www.webhostingtalk.com/showpost.php?p=8153934&postcount=18)

And about the actual usable bandwidth:

 _“I always found this to be a problem with all FDC locations. So many places
have such horrible speeds that no matter what speed they offer, I have a hard
time making a good use of it.”_

—
[http://www.webhostingtalk.com/showpost.php?p=8258933&pos...](http://www.webhostingtalk.com/showpost.php?p=8258933&postcount=27)

Another comment:

 _“The server is great. The single-thread transfer speed isn't as much, but
it's reasonably passable most of the time, considering the server cost. But
I've been seeing some severe routing problems, with 5-10% of the net simply
being unroutable much of the time, as well as intermittent packet loss. Due to
this, the overall fail rate of this server is about 10x that of other servers
I have in NL.”_

If you're running an ad supported viral image host and can fit your popular
content in RAM, this kind of thing may be acceptable, though in the long run
users will tend to migrate to image hosts that serve their memes quickly.

TL;DR: Not all bandwidth is equal. You get what you pay for and what your
provider pays for.

------
anonymouz
Slightly OT but related to the Cloudflare business/enterprise offering: They
"guarantee" 100% uptime, does anyone know details of this? Obviously there is
no way to actually guarantee such a thing, but what kind of compensation do I
get when they do have downtime? I can't seem to find it mentioned anywhere.

"Service Level Agreement (SLA) - 100% uptime

Industry standard SLAs often feature 99.999% uptime, also known as the five
9s. At five 9’s your website could be offline for as long as 5 minutes and 26
seconds each year. All CloudFlare Business and Enterprise plans offer
guaranteed 100% uptime because we know that anything less than 100% is an
impediment to your organization’s success."

~~~
moonboots
Usually these guarantees mean credits toward future billing. Cloudflare's
enterprise plan offers 2500% uptime guarantee, which means if they are down
10% of the time, customers are credited 2.5 months.

It sounds a lot nicer than it is, much like "we never charge for bandwidth".

------
danso
It's not a directly comparable service, but serving 100TB/month over S3 would
cost you more than $9,000 a month. On CloudFront, depending on object size and
distribution area, it would cost more than $8,000

<http://calculator.s3.amazonaws.com/calc5.html>

~~~
sauteedbiscuits
<https://google.com/search?q=100tb>

There are many cheaper options. The ones you mention seem to be the most
expensive.

~~~
byoung2
S3 and Cloudfront are targeted at the dabbler, who will likely only be serving
a GB or two per month, in which case S3 and Cloudfront would be pennies with
no long term commitment. Once you move beyond that and are storing and serving
more than 1 TB, you'd be better off with a monthly commitment.

~~~
imsofuture
Just FYI, Cloudfront does heavily discount committed pricing.

~~~
byoung2
True, but that starts at 10 TB/mo

------
ghshephard
You get what you pay for. When I'm paying for a $16/megabit @95th on a 10 Gig
port, and I get hit with a DDOS, it's my providers problem, because I'm not
paying $16/megabit for DDOS traffic, so they need to stop that stuff before it
gets to me (ideally without impacting my customers)

Contrawise ,if I'm paying $2-$3/megabit @95th on a 1 Gig Port, the amount of
support I can expect during a DDOS is pretty minimal, so I end up having to
take the hit - but my damage is limited to $3000/month so I don't really care.

Any time I see a "We don't charge for Bandwidth" service, I interpret it to
mean one of (A) We'll throttle you once you exceed our unspoken limit, or (B)
We'll discontinue your service. (Drop your port from 1 Gig down to 100
Megabits, or slower, traffic shape you, etc...) once you breach that limit.

There is no sustainable third option for those who provision reasonably high
quality transit, and those who believe there is will one day wake up with
their internet property offline, or seriously degraded.

------
notlisted
This kind of whining and the comments on the page piss me off greatly.

A customer is a person who pays for a service. Someone who doesn't pay for a
service (yet) is a _lead_. Not all leads are good business. 100TB of traffic
does not sound like a good lead to me, not even at the $200 level.

Looking at the site, I see an IMGUR clone which was running for free off of
CloudFlare's cache servers. I really don't understand the nonsensical comments
on the article. WTF is wrong with people these days thinking that everything
is supposed to be free? Are you all 16 and on an weekly allowance?

Commenter Matt had a very valid point that some sort of optimization of the
stored (cached) files would have been a smart option for yourselves (less
local storage) as well as CF (less to cache, less bandwidth). I'd recommend
<http://www.jpegmini.com/server> (Oh wait, it's not free, now what... cry me a
river)

I have a question for the Phobos peeps. Were you making money? Seems to me
like you were... since you can afford the LeaseWeb servers. Instead of
bitching publicly, perhaps you should have reached out to the company when you
noticed your traffic levels were reaching antisocial magnitudes.

Grumpy Gramps (who used to pay UUNET $6k+ a month in 1997 dollars for the
privilege of hosting a basic database-driven e-commerce site for a luxury
watch brand on a guaranteed T1 connection)

~~~
throwawayG9
They weren't making money, and they did contact the company. Comments like
yours piss me off greatly.

~~~
notlisted
They contacted the company _after_ being shut down. Not before, when it should
have been pretty clear to them how many resources they were using. There's a
simple indicator in the dashboard. "BANDWIDTH SAVED".

This line in the article "I tried to monetize it through ads some time ago,
but failed. Advertising-Networks don't want us as a customer, but I'm fine
with that." was not conclusive that monetization failed via other means. Not a
user of the site, but at first glance some choice links to pr0n affiliate
sites would do the trick.

Throwaway ey... hmmm.

~~~
throwawayG9
Notlisted uh... right.

~~~
notlisted
Four years bro. Tosser.

------
vilda
Why is it that you'll always get the worst posible review by a non-paying
customer?

This is a common observation - less the user pay the worse and less objective
the feedback is.

~~~
cbs
There are a handful of contributing causes, just off the top of my head:

Once you get past the first few tiers of paid service, providers tend to reign
in the bullshiting "technically true by widely misleading" descriptions and
are upfront about what they provide. Properly managed expectations don't lead
to as strong of emotions when a customer finds out what they're getting
doesn't work for them.

Commercial users will save face by A) avoiding discussing anything that could
make them look bad, ditching a vendor is a tacit admission that you went wrong
when you chose their service. Additionally, if their service is built on top
of the ditched service, it casts quality concerns on their product too. B)
Avoiding disparaging anyone, ever, is pretty common play-nice-save-face.

Why should I give my competitor helpful advice? I got caught up in a web, I
don't need to help the competition avoid my mistakes.

In a multiple person organization, the party angry about the service and the
party who decided to buy/use it are different individuals and airing
grievances/venting is an internal process that they don't reiterated in
public.

In a free service, a failure to deliver is the only downside. If I paid for a
service, I'm going to be too preoccupied beating myself up for making a stupid
purchase decision and looking for an better alternative than bothering to
share my findings.

These causes and more are all often balled up and interpreted as free users
acting "entitled". Sure that exists, but its a classic case of reading too
much into imprecise metrics when someone assumes the cause.

Furthermore, the feedback from free users isn't inherently worse, and is just
as subjective as that from paying users, it just tends to be more negative.
For all we know, any given product could objectively be shit and because only
rubes will buy it, the free users are the only ones that can let you know.

------
iSloth
Interesting, I have been using their services for a while now but most of my
traffic (probably 99%) actually goes straight to my web servers as i'm only
really interested in the free DNS hosting service, however looks like i'm
breaking the ToS as well, may be time to rethink :S

------
Uchikoma
Could someone explain the reply from "Matthew Prince" to me?

They turned it off due to an attack and would turn it on again if upgraded to
business ($200/mo).

Does this mean turn it on while under attack would cost, and after the attack
would be free again? Or it would cost the upgrade to turn it on either way?

~~~
niggler
They claim that other customers were affected, so they stopped handling
requests.

The process would have been automatic if he was a paying customer. Because he
wasn't, the company felt justified in not trying harder to bring their
resources back up.

------
moe
Wow.

Imagine the origin server would have been Amazon S3. The webmaster would have
incurred an Amazon bill of $400 USD _per day_ after the switch.

------
goldfeld
One thing I'm yet to puzzle out is whether CloudFlare handles apex domains.
They say they do but DNSimple says in reality they just route it straight to
your servers so that it's pretty useless.

~~~
alax
CloudFlare does handle apex domains; when you turn the service on, it changes
the A records to point at CloudFlare. I don't know where DNSimple is getting
their information, you can see CloudFlare working on a naked domain just by
trying to resolve the hostname. It'll return two (or more) CloudFlare IP
addresses, not the address of your server.

------
24kpwn
And why is 4chan still hosted at CloudFlare?

~~~
thezilch
4chan (and its users) probably want a fast downlink. The author touts a
cheaper provider, but the service they are receiving is an order or more of
magnitude slower service from my anecdotal test through wget(s) of images on
both OP's and 4chan's sites. That is, a pair of dedicated and unmanaged
servers is not, AT ALL, comparable to a CDN's offerings.

4chan probably also expects to have more than an email (and timely) endpoint
with which to correspond with the provider of their most core site-service --
image serving.

------
throwawayG9
Thanks a lot for this article.

Were you already using http cache of 1 year for all the images when this
happened? Do you think it could have been avoided by setting it to 1 week, 1
day, or even 1 hour?

It's funny that I have to ask this to you instead of asking Cloudflare. They
have really messed up on this one.

------
unkoman
Where are their sellers? This is just a bad example of how they do not
recognize their customers.

------
codexon
You couldn't monetize it yet you can spare $200/month?

~~~
dangrossman
I've rented servers for high-traffic websites with no income for years. If
it's something you love to run, and you can afford it, then it's just the cost
of your hobby. Not everything is a business.

~~~
codexon
If he is as really rich as you suggest it seems odd to me that he would
complain about Cloudflare and not have rented a 100 TB server sooner. They are
not exactly difficult to find.

Whether he can tell us if he is rich or found another way to monetize adult or
potentially offensive content, it is up to him to share.

~~~
dsl
$200/mo is not "rich," it is what you would spend on a hobby (think a
membership to a climbing gym or buying a new gaming computer once a year).
Being able to spare $3000+/mo for CloudFlare enterprise is not even close to a
comparison.

~~~
codexon
Now this might just be my opinion, but considering many of the people I know
in the technology field have a disposable income of around $1k/month, I don't
think there's very many people that would throw away 20% of it on an image
dump. If you make more than that, then you are rich by my standards.

~~~
phoboslab
See, for me it's not an "image dump", but a nice hobby project. Managing a
community as well as the technical challenges this site presents is
interesting. I can spare $200 but not $3000.

Also, I learned a great deal of JavaScript while building that page and I just
now learned a bit about load balancing and setting up Varnish - something
which may come handy for godknowswhat. I never imagined making my living by
writing a JS game engine either.

