
Netflix Open Connect Content Delivery Network - nkurz
https://www.netflix.com/openconnect
======
curun1r
The one thing missing from all the articles about OpenConnect is any way for
Netflix subscribers to see which ISPs are participating.

I'd like to know if my ISP participates and, if not, be able to call in to
suggest they do. If they still don't join in a reasonable time frame, then I'd
at least have the option to switch providers to someone that participates.

A program like this will be more successful if they allow for consumers to
pressure ISPs to participate.

~~~
Alupis
Unfortunately, as we've discovered from the whole net neutrality debate --
netflix has offered these boxes for free to all major ISP's... yet the Big5
ISP's have all turned them down.

~~~
Thaxll
Yes because those box aren't sufficient for 1M+ sub. You're better off using
peering rather than cache boxes in your network if you're a large ISP.

~~~
dsl
That is simply not true.

A consumer broadband network often distributes out over a very large area and
condenses into micro POPs. In the Bay Area think Pleasanton, Concord, Marin,
etc. The ISP must then backhaul traffic from these micro POPs to a major POP
where the whole internet interconnects (Santa Clara/San Jose).

If Netflix were to install a million terabits a second of peering and a
hundred thousand caching appliances in San Jose, it wouldn't alleviate the
ISPs pain point at all, which is the backhaul to the micro POPs.

Putting OpenConnect appliances into micro depolyments "out in the field"
solves the problem, because it moves the content and caching out to the last
mile. However the ISPs are reluctant to do this because then what used to be a
half rack of telcom gear in a small shed somewhere has now become a defacto
colocation facility. You have Netflix, Hulu, Amazon, Vudu, YouTube, Microsoft,
eBay, and two dozen others each with 4u of gear in hundreds of locations that
you are paying for power and rolling out to service when they fail (because
you can't let competitors have physical access to each others gear, and a node
failure can saturate the uplinks from the micro POP).

~~~
Thaxll
So why then large ISPs would refuse it, if it's free?

~~~
dsl
The physical box is free, yes. Imagine if every website you visited wanted to
put a 1u box in your living room, which you paid to keep turned on and
connected to the internet. If a box failed you either needed to unrack it,
haul it down to FedEx, and rerack a replacement - or let a stranger in while
you aren't home.

If you don't get a box fixed fast enough, or choose not to install the MSN or
Baidu box, everyone in your house starts to complain about it.

I'm not defending the ISPs here, just simply explaining their perspective. I
am a huge proponent of net neutrality, and I'm currently in the process of
deploying something similar to OpenConnect for another large network.

------
mey
[https://www.netflix.com/openconnect/hardware](https://www.netflix.com/openconnect/hardware)
[https://www.netflix.com/openconnect/software](https://www.netflix.com/openconnect/software)

It looks like they are heavily redundant boxes with the goal of reducing field
maintenance, running FreeBSD as the OS.

~~~
Alupis
> running FreeBSD as the OS.

Presumably for ZFS

~~~
howeyc
Actually it turns out they don't really have a need for most of the features
of ZFS. So they don't use it.

[http://www.youtube.com/watch?v=FL5U4wr86L4](http://www.youtube.com/watch?v=FL5U4wr86L4)

~~~
Alupis
Hmm, do you know which spot in the video they discuss that (it's a long
video)?

Seeing that they are using just controller cards instead of raid cards, and
FreeBSD, it seems even if they were not using specific features of ZFS, the
performance and other automatic things ZFS does to provide superior (compared
to most (all?) other filesystems) data integrity and protection... they would
be using it by default...

~~~
howeyc
Why FreeBSD - 18:30

Why no ZFS - 21:30

------
condiment
For all the hand-wringing about ISPs creating an "internet fast lane",
OpenConnect sure seems like a step in that direction. If Netflix is allowed to
colocate their server in an ISP's facilities (effectively eliminating the cost
of their egress bandwidth), how does that not give them an unfair advantage
against other internet video operators who lack the leverage necessary to
broker this kind of deal?

OpenConnect isn't the first attempt at this sort of business model, either.
Back in the 90s, Akami tried this same thing with their "FreeFlow ISP" service
model - putting their servers in the ISP's point of presence in order to
eliminate their bandwidth costs. Then, as now, major ISPs refused.

------
mmalone
If you're a net neutrality advocate you should seriously consider whether this
sort of peering arrangement should be tolerated. This kind of peering is a de-
facto QoS arrangement that gives Netflix a performance and cost advantage over
other video providers that aren't big enough or rich enough to justify and pay
for direct peering with ISPs. It also gives bigger ISPs (that do more than
2Gb/s of traffic to Netflix and agree to peer, apparently) the same advantage
over their smaller competitors. Isn't that exactly what we're trying to avoid
with net neutrality?

This is not a new practice and I'm surprised it doesn't come up more
frequently in the net neutrality debate. It's been going on for years --
broadcast.com did it, YouTube and Google do it, and I'm sure others do as
well. We've never really had net neutrality -- peering agreements have been
happening forever in shady back room deals at NANOG. At least the more recent
shenanigans are happening out in the open.

~~~
Alupis
I think you miss the point of open connect.

It's no different than if Netflix was paying Comcast (or whover ISP) to colo a
server(s) in their data centers (you can do this as a normal citizen too).

The only difference, the idea is Comcast wouldn't charge Netflix for colo
space since Netflix is "earning" it via cost reduction for the ISP. So they
are still technically paying for the colo space.

Perhaps I missed your point though -- because this seems to be a much better
way to a neutral net than having ISP's charging both sides of the pipe (which
the small guys can't afford even more)

~~~
tzs
Is colo space at the ISP an unlimited resource? If not, then it will end up
priced out of the reach of the small guys.

I suppose the ISP could offer some kind of "virtual colo", where instead of
putting your servers in the ISP data center for fast access they offer some
kind of faster connection to your data center. This would have two big
advantages over physical colo. First, it would not have the space limitations
so it could be more widely available. Second, the price could be based on how
many customers you are serving at the ISP so that even small content providers
can afford it.

This now sounds similar to "fast lanes", which have been the major focus of
opposition to the proposed approach to net neutrality of trying to regulate
under section 706.

The problem I see is that in both the company specific CDN approach and the
"fast lane" approach it comes down to a content provider is making an
arrangement with an ISP to make that content provider's content better for its
customers on that ISP.

Is that against net neutrality? If the answer is "yes", then I don't see why
it should matter if it is done via physical boxes placed on the ISP's network
or by prioritizing traffic from that content provider or by purchasing a
higher capacity interconnect to the backbone providers that content provider's
traffic comes through. This seems to me to be one of those things where it
should be looked at as a black box, and net neutrality should be concerned
with the externally visible behavior of the box, not the mechanisms within the
box.

~~~
wmf
The small guys can use Akamai who already has many cache boxes in place.

My understanding is that private peering and caches are primarily a cost
optimization and are not about performance (as long as the "normal" Internet
isn't deliberately congested). This way of thinking makes CDNs compatible with
net neutrality.

------
rmrfrmrf
I read that entire OpenConnect Deployment guide PDF, and I have to say that it
looks like a really great, well-thought-out program. I'm impressed with how
many moving parts there are and the time it must have taken to develop a
system like this; they don't even touch on the details behind the Cloud
Control Plane, and I'm sure that must have taken a while to develop. Netflix
has said in the past that they only want the most skilled employees possible,
and it certainly shows here.

------
swasheck
it seems to me that calling this an "internet fast lane" is a false
equivalency. my understanding was that the internet fast lane concept was
something that users would pay more to use, which leaves the isps free to
render the "regular lane" unusable so as to bump the pricing. additionally,
there was the whole notion of the "bundles" of websites available at different
cost tiering.

openconnect appears, at least to me, like it's a collaboration between
providers of different services to enhance the value for their shared
customers.

edit: directed at both mmalone and condiment

------
13throwaway
I wonder what measures they have in place to protect against someone cracking
this open and dumping the entire video library.

------
whoopdedo
What's to stop Netflix from later on asking the ISPs to pay for the cost of
OpenConnect? The way that satellite TV networks charge a subscriber fee to the
cable operators. Netflix has stated that they want to become like HBO. I don't
imagine they're unfamiliar with how HBO become so successful.

~~~
wmf
_What 's to stop Netflix from later on asking the ISPs to pay for the cost of
OpenConnect?_

The fact that ISPs will unplug it five seconds later.

 _The way that satellite TV networks charge a subscriber fee to the cable
operators._

HBO only charges each customer once (via the cable company). If Netflix tried
to double-dip by charging the customer directly and also charging the ISP
they'd get rightly flamed.

~~~
whoopdedo
> The fact that ISPs will unplug it five seconds later.

At the risk of angering the many people who watch Netflix.

I don't get the double-dipping argument either. We already have that with
television, newspapers, and radio being paid by advertisers yet still charging
a subscription to viewers.

------
rdl
I wish I could buy these custom chassis; I didn't see a link in the article.

~~~
mbell
I would think in low volume a supermicro chassis would be more cost effective,
they have many 4U models that support 36+ 3.5" drives.

------
Duckpaddle2
Very interesting to see what hard drives they "blessed".

~~~
nacs
The hard drives used are listed here:
[https://www.netflix.com/openconnect/hardware](https://www.netflix.com/openconnect/hardware)

    
    
      Hitachi Deskstar 5K3000 3TB
      Seagate Barracuda 7200.14 3TB

~~~
ihsw
> The following system was developed and first deployed at the end of 2011.

This is _very_ out of date. Their latest OpenConnect appliances use SSD
drives, have higher capacity, and use less power. They're heavier, too.

------
nodata
Funny name: is it open to other CDNs too?

------
StavrosK
Can someone explain what this is? I don't see that mentioned anywhere.

------
blerk
> For the operating system, we use FreeBSD version 9.0. This was selected for
> its balance of stability and features, a strong development community and
> staff expertise. ...or due to the less restrictive licensing, it's ok
> netflix everybody does it...

~~~
wmf
AFAIK Netflix contributes back all their changes to FreeBSD, so I don't see
how the license matters.

My impression is that companies that don't like the GPL either deal with it
(Android) or just violate it (almost all embedded devices). Nobody is actually
willing to switch to BSD for licensing reasons.

~~~
blerk
Sony switched over from Linux (ps3) to kfreebsd (ps4) due to licensing. Sure
they (netflix) contribute back to FreeBSD, I don't doubt it, but don't act
like licensing doesn't have anything to do with it.

~~~
ksec
The PS3 CellOS is based on FreeBSD, not Linux afaik.

~~~
blerk
xmb yes, development kits no

~~~
dogshoes
I believe the only portions of the development kits which used Linux was the
communication processor, an isolated computing platform bolted on to the
hardware to facilitate certain debugging functions.

Sony releases the GPL source code here:
[http://www.sony.net/Products/Linux/Others/DECR-1000.html](http://www.sony.net/Products/Linux/Others/DECR-1000.html)

