
Building fast.com - samber
http://techblog.netflix.com/2016/08/building-fastcom.html
======
nippoo
This is beautiful. By using Netflix's _actual_ CDN and requesting actual video
files over HTTP, they're preventing ISPs from whitelisting fast.com without
also whitelisting Netflix itself. Of course there'll be a game of cat-and-
mouse with DPI, but it's a super-neat way of making it much harder for the
ISPs to misrepresent actual video download speed...

~~~
IshKebab
Yeah right. Just wait until a user visits fast.com and unthrottle their
connection for 20 minutes.

I strongly suspect some ISPs already do that with speedtest.net.

~~~
nkozyra
Note to self: set up cronjob to ping speedtest.net and fast.com every 10
minutes, just in case.

~~~
figgis
I did that, works great and with speed test cli it (did it without that at
first) it's even better!

[https://github.com/sivel/speedtest-cli](https://github.com/sivel/speedtest-
cli)

If you are going to setup a cronjob/service for it, I suggest using the
--simple flag.

~~~
thecosas
Any before/after stats to show this made a difference in speed?

~~~
figgis
Could you clarify the question?

~~~
safewayclubcard
Do you have any proof that running a cron job impacts internet performance?

~~~
figgis
I believe you replied to the wrong person. I never made that claim unless I am
going crazy.

speedtest-cli has a lot to offer as a tool

~~~
Trundle
What did you mean "works great" for then? Just that your cron job works?

~~~
figgis
The actual tool I was suggesting was what I was saying works great. speedtest-
cli has a lot more options available to you that a simple wget does not have.

It provides cleaner information quicker. Yeah you could certainly setup a cron
job using wget and have it do everything you want it too down to the bits it
changes in memory... That just offers a quicker way to do all you may want too
with that information.

------
greenspot
First, I thought why another speed test, then I saw that they link their
results directly to speedtest.net (the reference) and finally, I realized that
fast.com is so much faster in detecting the download speed (but lacks upload
and ping).

Not bad.

EDIT: And they secured a fantastic domain.

~~~
twinkletwinkle
I think the point was they conduct the speed test from the same servers that
deliver video content. So ISPs can't throttle video streaming without it
showing up in this speed test.

~~~
greenspot
Makes sense but wouldn't they then use the netflix domain? Something like
netflix.com/fast? I could image that if providers throttle it would be a mix
of IPs and domain.

~~~
chrishacken
Some ISP's (we don't throttle anything) throttle based on several factors,
mainly packet signatures. Traffic coming from big/known sources, such as
Netflix, contain unique signatures (probably the headers, etc, that are unique
to Netflix Traffic). You can then check all incoming packets that match those
signatures and throttle them. So unless Netflix went out of their way to mask
every request to look like video content, there's not a whole lot you could
do.

~~~
IceyEC
The files that fast.com has you download ARE chunks of video files :)

~~~
zeristor
Does anyone know which video the test file is?

Perhaps it's encrypted, or a Netflix promo piece.

Wouldn't there be copyright issues to use their actual streaming licences
content?

~~~
corobo
If you have a Netflix subscription it's probably this or something like this
[1]. For non-flixers it's a calibration video similar to [2]

[1]
[https://www.netflix.com/watch/80103278](https://www.netflix.com/watch/80103278)

[2]
[https://www.youtube.com/watch?v=cGgf_dbDMsw](https://www.youtube.com/watch?v=cGgf_dbDMsw)

------
cocoflunchy
Fast.com is a lot slower than both speedtest.net and
[http://speedtest.dslreports.com/speedtest](http://speedtest.dslreports.com/speedtest)
for me... I get 78mbps on fast.com and 350 on the other two. Not sure what's
going on? Maybe they don't have servers near me, I'm in Paris, France.

~~~
the_mitsuhiko
More likely your ISP is cheating and you don't actually reach 350 outside
speedtest and some other routes they make sure you are unthrottled on. Would
not be the first.

~~~
cocoflunchy
Maybe they throttle Netflix yes... I do get 350mbps on every other test and
when downloading a test torrent though.

------
zackbloom
It's a little odd that they elected to invent their own way of requesting just
a range of bytes in a file. It's built into HTTP:
[https://tools.ietf.org/html/rfc7233](https://tools.ietf.org/html/rfc7233)

~~~
douglasfshearer
YouTube also put the byte range in the URL rather than using a range request
header.

Perhaps it mitigates against proxies stripping the headers, or it simplifies
caching.

~~~
a1k0n
I tried to use the Range header feature to resume downloads in an autoupdating
app once, and it worked beautifully...for about 90% of users; the others had
all kinds of weird issues thanks to transparent proxies and caches not
handling the Range header correctly. This was years ago, but I'm not surprised
they chose to be safer than sorry.

~~~
JoshTriplett
> I tried to use the Range header feature to resume downloads in an
> autoupdating app once, and it worked beautifully...for about 90% of users;
> the others had all kinds of weird issues thanks to transparent proxies and
> caches not handling the Range header correctly.

One of many reasons to use HTTPS: prevent tampering with your requests and
responses.

~~~
zeta0134
Youtube recently did a writeup on their long road to HTTPS: [http://youtube-
eng.blogspot.com/2016/08/youtubes-road-to-htt...](http://youtube-
eng.blogspot.com/2016/08/youtubes-road-to-https.html)

I find it particularly telling that switching to HTTPS caused many streaming
errors to mysteriously vanish, and I suspect ISP / proxy tampering to be part
of the cause. I'd love to know more about what their engineers saw though.

------
mofle
You can also run fast.com from the command-line:
[https://github.com/sindresorhus/fast-
cli](https://github.com/sindresorhus/fast-cli)

~~~
joeblau
This is cool; just ran it, but I couldn't tell that it was doing anything.
Maybe adding some soft of indicator/throbber that lets you know it's
processing.

    
    
      fast | cat
      860 Mbps

~~~
mi100hael
Why are you redirecting the output of the script which would otherwise be
printed to a program that does nothing but print the output?

~~~
corobo
Best guess is because it's what the --help says to do if you just glance at it

    
    
      $ fast --help
    
      Usage
        $ fast
        $ fast > file
    
      Example
        $ fast | cat
        90 Mbps

------
aggieben
1\. When will someone build a tool like this and then open up the measurements
data? A speed-measuring tool would be infinitely more useful if we could
investigate the results beyond just our one device for one test. A question I
really want to be able to answer: which ISP offers the fastest service in my
area? fast.com and speedtest.net could easily answer this question if they
made the data available.

2\. I think tests like this that rely on a built-out network and well-placed
CDNs and such are probably junk. A more useful test would be against a handful
of download targets that _aren 't_ optimized, and then average the results.
This is particularly true for Netflix. Case in point: fast.com says I download
at 78Mbps, but speedtest.net has me down around 57. I'm only paying for 30.

~~~
netinstructions
Google / YouTube has a 'Video Quality Report' which will measure your speeds
and will also show local ISPs speeds as well.

[https://www.google.com/get/videoqualityreport/](https://www.google.com/get/videoqualityreport/)

~~~
blakeyrat
"Reports from your location are not available yet." Uh, I didn't tell you my
location, Google... what the heck?

~~~
dpcx
They can guesstimate by geolocating your IP.

~~~
blakeyrat
The problem is I'm at work right now, and I wanted to get results for where I
_live_ not where I _work_. (Which might still be unavailable, given, but at
least let me type it in!)

------
comick
I see this as a tool for netflix to immediately raise concerns about ISP
neutrality. The link to another speed test (using some minor server) is there
not by chance. If speed test is far better than netflix fast.com the ISP is
clearly limiting the customer.

~~~
hardtke
I wish that my ISP (Comcast) would throttle Netflix in the evening. My
internet bandwidth gets reduced to 200 kbps during prime time (roughly 9 to 11
PM). It makes it challenging to work, especially if I need to stay connected
to a VPN.

~~~
yazaddaruvala
If you were wishing, why not wish for your ISP to aptly judge its peek traffic
and scale its infrastructure?

------
Buge
I seem to be unable to view the blog post in Firefox, it automatically goes to
https which doesn't load.

Maybe netflix messed up their HSTS settings at some point in the past?

~~~
ajamesm
I'm also experiencing the same (also using HTTPS everywhere). Disabling the
extension doesn't seem to work.

If you get a solution, please let me know.

~~~
Buge
I found that Noscript was causing it.

The best solution is to go into Noscript options > advanced > HTTPS and add
.netflix.com to "Never force secure".

The other way is to go into about:config and set noscript.httpsDefWhitelist to
false. But that's less secure.

~~~
__david__
Ahhh! Thank you! That was my problem as well. I didn't even know NoScript was
doing that. >:-(

------
daenney
Looks like I've hit a case of IPv4 vs. IPv6. I recently noticed that fast.com
starting showing me at about half my connection speed, 50Mbps. I only just now
realised while looking at the traffic that it was doing it over IPv6.
Disabling IPv6 in my network gets me a reported speed of 120Mbps on fast.com,
which sounds about right for a 100/100 subscription + some burst.

My ISP hasn't even officially announced any IPv6 support, I discovered more or
less by accident that if I enabled 6rd things "just worked" so the fact that
it might not go at the full speed of my subscription isn't something I'm super
worried about.

I've actually got a similar issue with Google. While using IPv4 I don't even
leave my ISP's network as they have those Google Global Cache boxes at the
edge and terminate my connection there but over IPv6 I do go all the way out
over the internet to Google.

~~~
developer2
Chances are you are not running native IPv6. Most customers who "have access
to IPv6" are running through IPv4 -> IPv6 conversion which adds immense
overhead and is not a real indication of IPv6 to come. For all such customers,
their pseudo-IPv6 will always be slower than IPv4. Most ISPs do not run any
form of native IPv6 yet.

~~~
daenney
> Chances are you are not running native IPv6.

Yes, that's the case since, as I stated, I got IPv6 connectivity through 6rd:

> It is derived from 6to4, a preexisting mechanism to transfer IPv6 packets
> over the IPv4 network

------
mshenfield
This does something good by providing a nice service for a common and
difficult to answer question, and also seems like a smart strategic use of
Netlix's resources.

The net result of applying their capital, engineering talent and existing
infrastructure here: a positive recruiting impact for engineers, a positive
branding impact for consumers visiting fast.com, and Netflix obtains
information about internet speeds for IPs that are not currently Netflix
customers, information that many businesses would find useful for things like
determining the market for high bandwidth applications.

------
Citrus2fizz
Hows this for a discrepancy
[https://i.imgur.com/7SX2Wro.png](https://i.imgur.com/7SX2Wro.png)

~~~
virtuallynathan
This is a bit of an unfair test, as it is not in Netflix's best interest to
provision enough CDN capacity for any user to hit 1Gbps, especially at peak
times. Netflix wants to buy/provision just enough capacity so that users don't
complain. Depending on the distribution of the users in the area, this could
5Mbps per user, or 15Mbps per user for a large amount of 4K streamers.

Netflix pushes their CDN nodes to the limit, 60-70Gbps per box for video.
Speedtests don't make them money, so why provision capacity for it?

~~~
sergiotapia
That's incorrect. I have AT&T Gigapower 1GB connection, and fast.com shows me
850-989 MB/s speeds.

It seems Netflix does shove info to you as fast as they can from fast.com

~~~
virtuallynathan
I'm not saying it isn't possible to ever hit 1Gbps (or whatever speed), I'm
simply saying it is not in Netflix's best interests to make sure this is
possible 24/7 365.

I have 1Gbps CondoInternet/WaveG at home, right now at 11PM PST I get ~400Mbps
from Fast.com and 800Mbps+ from DSLReports and the Comcast/Xfinity speedtest.

------
djhworld
My parents recently had 100mbps fibre installed, I told them to visit fast.com
to test their speed and they were pleased at how easy it is to use and hassle
free

Good work by Netflix.

------
e28eta
I'm getting a consistent difference between fast.com in Safari on iPad vs
their Fast app (90-ish vs 65-ish, across 5 interleaved trials). I was hoping
for _better_ measurement in the native app, but I suspect it might just be a
web view in a native shell. Anyone else seeing similar results?

I think this is an interesting idea, but I'm not convinced it'll actually make
a difference. From the consumer side, there's nothing to say that the slowest
link isn't Netflix's CDN when a discrepancy occurs between Fast and Speedtest.

I was receiving atrocious download speeds for movies from iTunes (18+ hours
for 4 GB), and also saw slow speeds from speakeasy.net/speedtest. The comcast
rep just pointed me at Speedtest and said "it's not a problem with your
connection, it's fast! Contact Apple." He dismissed the "other" speed testing
results. ISPs can continue to throttle directly or by refusing to upgrade
peering connections. fast.com providing a number to go along with bad Netflix
streaming quality doesn't improve the customer's life.

What I really want is something that empowers _me_ to be heard in my request
for the $75 worth of Internet that I pay for every month. I had high hopes
that the SamKnows program might yield something, until I received a docsis 3
cable modem in the mail from Comcast with a note: "since you're a participant
in SamKnows monitoring, we sent you this upgraded modem so you can take
advantage of better speeds". My first thought was "what about all of your
customers who aren't participating in SamKnows? When do they receive their
'upgrades'?" Do we really need the government monitoring _everyone 's_
broadband performance to ensure that companies are delivering?

------
manish_gill
I wish people would believe this. I had an hour long fight with my ISP the
other day - the customer support people are provided with a script that they
have to follow, and I'm trying to tell them that the speed isn't as good as
they claimed it was. It got to the point where I was sharing screen with them
on TeamViewer to show them the download speed I was getting.

Their response? "We use SpeedTest.net as standard, it must be a problem with
the servers that you are using".

------
pavs
I wonder if Netflix makes it as easy to apply for their local CDN nodes as
google does with GGC. I know akamai is super difficult to apply for and get
admitted.

Ever since Netflix became available in our country we have been hurting BW
wise. We don't throttle BW for any sites. Does anyone here have any experience
with open connect (Netflix)?

~~~
firloop
It doesn't seem terribly difficult so long as you have the traffic numbers to
support it (peak traffic of 5gbps) and enough bandwidth for the box to fill
its cache during off peak hours.

[https://openconnect.netflix.com/en/requirements-for-
deployin...](https://openconnect.netflix.com/en/requirements-for-deploying/)

------
breakingcups
"In pursuit of the design goal of simplicity, we deliberately chose to measure
only download speed, measuring how fast data travels from server to consumer
when they are performing activities such as viewing web pages or streaming
video. Downloads represent the majority of activity for most internet
consumers."

I'm willing to bet additional factors that went into that consideration were:

\- Netflix gets massive download traffic on the cheap due to their scale.
Upload, not so much

\- This tool is a clever way of preventing providers from throttling Netflix
servers without also throttling a (hopfully soon to be) popular speedtest.
Uploading isn't something they need their primary customers to be able to do.

------
tener
Speed testing is a good starting point, but on my LTE connection (rural area)
I find the latency to be very contributor to overall experience. Especially
that once the link is saturated the latency shoots through the roof (up to
x10). I wish there were some tools for testing more complex use cases.

~~~
djrogers
Latency is largely irrelevant to one-way video streaming (at least until it's
so high it affects throughput), and since Netflix designed this to test your
speed for their one-way video streaming service I think leaving out latency
testing is reasonable.

~~~
infereque
Latency is critically important...

You can easily determine how much of a problem it is in your case by
calculating your bandwidth delay product:
[https://en.wikipedia.org/wiki/Bandwidth-
delay_product](https://en.wikipedia.org/wiki/Bandwidth-delay_product)

~~~
djrogers
In the real world TCP window scaling solves latency induced delays for video
streaming in all but the most extreme cases (hence my caveat), and endpoint
buffering can solve even those.

For many activities on the internet a 1,000ms latency would be intolerable,
but for one-way video streaming it's unnoticeable to the end user. I don't
really care if a packet containing video takes 50ms or 1000ms to get from the
CDN to my TV, as long as it gets there and is reasonably consistent in it's
ability to do so.

For this use case, if my machine can get 20Mb/s on the test, latency will just
not affect me streaming a 10Mb/s video. Latency may be lowering my test
results, but that's the point here - testing the effective throughput.

------
newscracker
Unfortunately, it doesn't work for me behind a proxy (tried with three
different browsers). I thought I'd just check it out
([https://fast.com](https://fast.com) \- trying http redirects to https
anyway), but it says:

"Could not reach our servers to perform the test. You may not be connected to
the internet"

P.S.: It also bugs me that it says "the internet" and not "the Internet". :)

~~~
rahulgulati
:D AP and other publishers don't recommend capitalising 'internet' and 'web'.

"The changes reflect a growing trend toward lowercasing both words, which have
become generic terms"

[http://www.poynter.org/2016/ap-style-change-alert-dont-
capit...](http://www.poynter.org/2016/ap-style-change-alert-dont-capitalize-
internet-and-web-any-more/404664/)

------
aidenn0
Apparently my ISP upgraded my speeds; I tested at 170Mbps and when I last
checked my plan it was 80Mbps.

In general, I've been quite happy with Cox. The actual network service is
great. Customer service and pricing are so-so (I'm paying $60/mo. for the
above connection; customer service is very hit-or-miss depending on which
representative I get, but I haven't been outright lied to).

------
rfc791
Several years ago I used this wonderful tool.
[http://software.internet2.edu/ndt/](http://software.internet2.edu/ndt/).

It provides the ability to do lots of troubleshooting in a private network.

------
pbreit
Testing for speed is great and all but is there an app or something that can
monitor my Mac's end-to-end connectivity to an internet source? Like, I feel
like my connectivity is spotty but am not sure if it's Wifi, Comcast or other.

~~~
toomuchtodo
Trial/paid:
[https://www.pingplotter.com/download/mac/](https://www.pingplotter.com/download/mac/)

Free/dockerized:
[https://hub.docker.com/r/dperson/smokeping/](https://hub.docker.com/r/dperson/smokeping/)

------
Morpheus79
Nice and I think they really benefit on their existing infrastructure.

But tests shows 5mbits for me and all other 30+ (real). So not really a great
help till now.

------
randartie
What's stopping ISPs from slowing down other speedtests at the same rate as
netflix to eliminate your ability to have a 'control'?

~~~
dpcx
You can have any "control" you want by setting up your own "speedtest" mirror.
It's hard to stop all of them.

------
macandcheese
Great example of engineering as marketing.

