

Netflix begins the deployment of their own content delivery network - fendrak
http://www.zdnet.com/blog/datacenter/netflix-delivers-using-commodity-hardware-and-open-source-software/1409

======
mef
From <https://signup.netflix.com/openconnect/software>:

Operating System

For the operating system, we use FreeBSD version 9.0. This was selected for
its balance of stability and features, a strong development community and
staff expertise. We will contribute changes we make as part of our project to
the community through the FreeBSD committers on our team.

Web server

We use the nginx web server for its proven scalability and performance.
Netflix audio and video is served via HTTP.

Routing intelligence proxy

We use the BIRD Internet routing daemon to enable the transfer of network
topology from ISP networks to the Netflix control system that directs clients
to sources of content.

~~~
patrickgzill
They are probably using ZFS also. (my guess)

~~~
amalag
Is ZFS stable enough on FreeBSD? Seems to be an unofficial port.

~~~
duaneb
ZFS is very stable on FreeBSD.

~~~
awaythrow
But you need a lot of extra RAM compared to the usual filesystems. Is that
correct? If yes, is using ZFS worth the cost of the extra RAM?

~~~
gauravk92
You don't need the extra ram. ZFS uses it as a cache, more ram, more cache.
That's about it, otherwise it doesn't use much ram.

------
ChuckMcM
Interesting box. With an AFRR of 5 - 7% the box would nominally 'lose' a hard
drive or two a year. I wonder how they deal with transient failures (which
would be more like 2 - 3 month). If they do the 'triple replica' thing they
only get 36T effective per box, but I suppose at about 2G / movie in MPG2
thats still about 18,000 movies 'online' as it were. Now if they did something
wild and crazy like a form of RAID6 that would be even cooler (and give them
like 100T of actual space or 50,000 movies. Serious video, but really hard to
stream to multiple people.

The old SGI/TimeWarner Video-on-demand nightmare seems so quaint now. That had
a full cabinet (42U) of storage and servers and it poorly served about 50
streams of a couple of thousand 'titles' and cost > $2M each if my aging brain
remembers all the specs. Hmm these guys [1] peg the cost for the "Full Service
Network at $100M which sounds about right.

[1] [http://www.vizworld.com/2009/04/what-led-to-the-fall-of-
sgi-...](http://www.vizworld.com/2009/04/what-led-to-the-fall-of-sgi-
chapter-5/)

~~~
ropiku
Since it's a cache I don't think there's a big problem in losing a hdd and
some cache capacity. I would expect it to be more efficient to use as much
storage as possible even if a hdd will fail.

~~~
ChuckMcM
I agree the cache nature makes it easier, but consider that a 3TB drive, at
2GB per 'movie', holds 1500 movies. So when you lose a drive, if the only
content you have is on that drive you need to seamlessly switch from sucking
it off the drive and sucking it down the back haul network. I can think of a
number of ways [1] that might be done with varying degrees of 'hiccup' on the
user end, but it is a big chunk of cache to lose at once. Note that 3TB is
also 50 minutes of full on 10 GbE pipe bandwidth. But as you say it isn't as
if you _lose_ data, since its still there back at the head office, but you
lose that copy of it.

[1] Ways that come to mind are; multiple boxes with a diversity of the content
amongst them so that another box can 'pick up' the stream, keeping a stream to
the home office open but not pulling data until you lose the local source, or
even doing a 'buffer' to disk and stream from the buffer then switch disk
targets when your disk fails.

------
ghshephard
I've often wondered if CDNs are going to go out of business in the streaming
world, and no longer be relevant, in the world of (rapidly) falling transit
costs.

Once transit costs drop below $1/mbit @95th percentile (circa 2014 [1]), it's
costing you $1 to transfer approx 300 gigabytes/month. At that point, the
management costs associated with all of those distributed CDNs becomes greater
than your transit costs. This is somewhat offset by continuously dropping
prices of CDNS [2] which is as low as $0.02/Gigabyte on a 500 TByte commit.

What this does tell me is that NetFlix thinks they have an opportunity to save
money on CDNs and can beat that $0.02/Gigabyte.

Bandwidth Delay Product(BDP) - keeps CDNs relevant for _downloads_ (iTunes,
Software, etc..) - because everybody likes to max out their 100 mbit comcast
connection, but you don't need much bandwidth to stream a show. Maybe some
value for CDNs in HD or live event streaming.

[1]
[http://drpeering.net/AskDrPeering/blog/articles/Peering_vs_T...](http://drpeering.net/AskDrPeering/blog/articles/Peering_vs_Transit___The_Business_Case_for_Peering.html)

[2]
[http://blog.streamingmedia.com/the_business_of_online_vi/201...](http://blog.streamingmedia.com/the_business_of_online_vi/2010/06/data-
from-q1-shows-video-cdn-pricing-stabilizing-should-be-down-25-for-the-
year.html)

~~~
jedberg
Your calculation doesn't account for the artificial barriers that some ISPs
put up by charging the CDNs to peer with them (see Comcast and Level 3).

~~~
wmf
Isn't that orthogonal to the choice between serving from the edge or from a
single mega-datacenter?

------
juiceandjuice
Cool, maybe they can deploy it at comcast and deliver content via the same
magical content pipe comcast uses for their xfinity service.

~~~
mikeryan
Most of the large CDN's already deliver straight into Comcast's network. Last
I heard Netflix was already paying Level3 to do this. Xfinity doesn't have a
magic pipe, all the content and deliver is internal to the Comcast network so
its its own "CDN".

~~~
wmf
More info on Comcast's DiffServ magic:
[http://ber.gd/post/23025893856/comcast-traffic-
prioritizatio...](http://ber.gd/post/23025893856/comcast-traffic-
prioritization)

~~~
mikeryan
Which Comcast says isn't happening.

[http://blog.comcast.com/2012/05/the-facts-about-xfinity-
tv-a...](http://blog.comcast.com/2012/05/the-facts-about-xfinity-tv-and-
xbox-360-comcast-is-not-prioritizing.html)

~~~
wmf
I don't really want to rehash that whole discussion, but Comcast admits
they're using DiffServ tagging. People are just debating whether it's a form
of "prioritization" or not.

------
mikeryan
Makes me wonder how long until Netflix looks at doing managed services for
video delivery along the lines of Amazon's AWS services.

~~~
filmgirlcw
But would that have great business potential to them? For Amazon, having that
infrastructure helped create not just future products (Instant Video, Amazon
Music Cloud, Amazon Cloud Drive, the Kindle and Kindle Fire backend systems),
but was aligned as an additional revenue source.

You'd have to wonder if enough people would want to use Netflix's technology
stack in a way that didn't include distribution on Netflix.

~~~
ketralnis
The entire television/sports/web video/publishing industry may be able to find
a use for such a thing

~~~
filmgirlcw
That's valid, except most of them have their own solution partners. What I
mean is, does Netflix want to invest the resources that would necessitate
making this an offering for all but the biggest customers.

Keep in mind, Netflix is one of the BIGGEST online video providers, period.
Very few companies are going to need some sort of solution at their scale --
and those that do, likely have their own solutions. That means, your target
market are smaller companies.

I just don't see Netflix interested in getting into a space that Brightcove
and 5Min already dominate on the smaller web side -- and that custom Amazon,
Level 3 and Akamai contracts take care of on the high end.

Netflix built this solution because it hopes that it will save them money and
resources, somehow, I doubt that it's s scalable model (at least in terms of
Netflix's required investment for support and sales) to offer to other
partners.

I could be wrong, I'm just not sure I see the potential for a company that's
biggest competitor is HBO.

------
cluda01
Hopefully they should be able to use these cost savings to purchase more
content and get it into their system. Does anyone have any informed
speculation about how much they could potentially stand to save? I'm
unfamiliar with how peering agreements work and or the cost structure.

~~~
biot
It's interesting from a bit of a game theory standpoint. The optimum outcome
might be achieved if Netflix hands over the appliance and the ISP installs and
keeps it running at no cost to either party, except for maintenance costs
(hardware replacement, etc.) which Netflix bears. Both parties save.

If Netflix wanted to get clever, they could calculate the cost an ISP would
likely pay for the bandwidth their users consume, and charge the ISP say 40%
of this amount to host the appliance. Netflix saves 100% of its third party
CDN bandwidth costs for that ISP and the ISP saves 60% of their peering
bandwidth costs.

If the ISP wanted to get clever, they could calculate the bandwidth that
Netflix is paying a third party CDN for the traffic consumed, and offer to
host the appliance for 40% of this amount. The ISP saves 100% of its peering
bandwidth costs for Netflix traffic and Netflix saves 60% of their CDN cost.

What's likely to happen is a bit of cat and mouse over the costs. The ISP
peering is likely a lot cheaper than Netflix's CDN costs, so this could work
out as:

    
    
        Netflix CDN cost
      - ISP peering cost
        ----------------
      = Cost to Netflix
    

This is all pure speculation with made-up percentages, of course.

~~~
rdl
comcast is actually charging people more than transit rates to get access to
their network -- which is why (except for level3, who has a special deal)
video on comcast sucks even with large comcast pipes.

~~~
squeed
Actually, no, Comcast's paid peering costs are significantly cheaper than all
but the lowest quality transit. In other words, if you just have Cogent, you
get what you pay for.

~~~
rdl
That may be true at the 10G level (which I didn't price), but isn't true (in
my experience) at 1G. Plus, it's a lot more likely that you'd want 10G of
transit and 1G of comcast than 1G transit/10G comcast, so per-G, your decent
quality (Level(3), nLayer, etc.) transit is going to be cheaper. Comcast also
seems to negotiate less, since they're effectively a monopoly on routes to
themselves, vs. transit.

(Obviously the right thing to do is grow to needing 10G of comcast and 50-100G
of transit...)

~~~
squeed
Fair enough. The last time I was privy to this, it was more than a year ago. I
didn't expect things to change that much :-).

------
fendrak
Here's the accompanying Netflix blog post:

[http://blog.netflix.com/2012/06/announcing-netflix-open-
conn...](http://blog.netflix.com/2012/06/announcing-netflix-open-connect-
network.html)

------
thelastknowngod
I have worked on Netflix's servers first hand. After seeing how large of a
presence they have in their current CDNs, I can say that they have a hell of a
lot of work ahead of them for that to become viable. They are enormous.

------
ComputerGuru
Completely off-topic, but I _love_ the redesigned Silverlight player for
Netflix on PC. It's gorgeous.

