Hacker News new | comments | ask | show | jobs | submit login
Netflix and FreeBSD: Using Open Source to Deliver Streaming Video [pdf] (fosdem.org)
132 points by vermaden 12 days ago | hide | past | web | favorite | 61 comments





Why is there such a visceral hatred for non-GPL software? Seems that every time the GPL vs. BSD (or equivalent) issue comes up there is much anger about companies doing proprietary work on top of FreeBSD. I'm curious about the opposite: code taken from BSD (or equivalent) projects and put into GPL code bases and then extended such that those improvements cannot be ported back to the BSD source. GPL proponents say this never happens but in multiple forums I've heard from folks who have had their permissive-licensed projects killed because their work was effectively GPLed and "extended" against their will.

To me this boils down to how you see humanity: if you think we're essentially good and willing to help each other, permissive licensing makes more sense. If you think morality has to be enforced by contract because we cannot trust one another then GPL is the only choice. I chose permissive licensing because I don't see my fellow humans as essentially evil.


As user of commercial software and ex-Linux zealot, I have my own glass ceiling, so take it with a grain of salt.

Had Linux and GNU not been GPL, and we would all be using Aix, HP-UX, Solaris, ..., because none of the BSDs would have gotten back the contributions like it happened to Linux, and BSDs would still be just another UNIX derivative among many.

Just look at the optimizations that some embedded OEMs, or even Sony, don't contribute back to LLVM. And speaking of Sony, not everything from PS4 OS has gotten back to BSD.


The Playstation 3 and Vita also run/ran NetBSD, as well

Sony also helped spawn "ToyBox". A BSD licensed replacement for the GPL "BusyBox" https://en.wikipedia.org/wiki/Toybox


The problem might be that companies are considered to be people and it seems easier for companies to be evil.

I really like the work Netflix has done and open sourced back into FreeBSD [1], But I do take issue with one of the slides, where it says "Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections". Sorry but that's not FreeBSD, that's a Netflix proprietary fork of FreeBSD serving all that, until me and the rest of FreeBSD community can actually download and use TLS sendfile, Netflix should probably stop saying FreeBSD serves all this TLS traffic.

[1]: https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s


The most important part IMHO is here https://github.com/freebsd/freebsd/tree/master/sys/netinet/t.... The TLS sendfile piece is a bit specialized, mainly interesting if you are doing the exact same workload i.e. a CDN on UFS disks. I can say on some authority (I had all the relevant code at my last job) NF will give it to you if you ask and have a need/competence. The reason the TLS bits aren't upstream is that they aren't general purpose or "finished" yet. It relies on some third party code outside of src like boringssl and Intel's isa-l.

I clicked on that video and at the timestamp in the link the guy says they like FreeBSD because it's not GPL. Not that we have anything against GPL or the Free Software Foundation, but the BSD license is more collaborative. I'm paraphrasing here.

Is there any actual evidence for that claim?

I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you".

Why does FreeBSD then not have your TLS sendfile code? Linux has TLS sendfile. It was contributed by a corporation, as far as I know.

Personally, I have always really liked the BSD projects. The people working on them have been nothing but professional and yet courteous, they tend to fix their security issues quickly. I would love for them to get paid properly.

With money. Not with "exposure" or "we'll test your code".


>I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you".

Where are you seeing these corporations ripping off FreeBSD? Netflix has contributed code in the past, currently over a thousand commits in FreeBSD, and there's no reason to doubt their intention to continue doing so.


I worked for 5 years for a company taking from *BSD and not contributing much. Lately they started monetary donation, but no code.

Weirdly, I actually think that's a feature not a bug. It keep ill intentioned idiots from gumming up the works because they are forced to contribute and really don't have the corporate beliefs or culture that would make them productive and not a distraction.

The problem is that BSD problem is not so much money, but manpower, on top of that the cathedral model does not really scale. Just from a technical standpoint, no BSD is using any modern DCVS, beside dragonfly.

Money is always a problem, and fixes a lot of manpower issues when you can pay developers. The whole DCVS is a bizarre argument. I would have to see actual studies that show the BSD run organizations do not scale.

Lack of proper DCVS hinder the possibility of committers to publish early version of their work outside huge diff. Getting the code and submitting changes are a huge PITA. With the git/github/gitlab workflow, you can clone a project, fork it, push a fix and submit a PR faster than a CVS / SVN checkout. It's not hard to find out why 90's cathedral style hierarchies hinder development process.

That being said, unless you are already a committer, it's gonna be very difficult to get your code blessed...


That being said, unless you are already a committer, it's gonna be very difficult to get your code blessed...

Are there examples of this? Since I see people sending patches that aren't committers. I honestly would want to see some actual data.


Frustrating experience.

>because they are forced to contribute

but they aren't. GPL forces to release source, not contribute.


That’s a contribution and then they feel entitled because folks who don’t see the business case in the first place are going to be a pain.

Apple, Sony, a couple of companies selling routers,...

They are on their right though, that is what the BSD licence is all about, code for free no questions asked.


I'll grant you there are probably router companies ripping them off, though that's true of Linux too. Your other two have both contributed much back to FreeBSD, so I'm unsure what you mean.

With Linux there is a legal way to get hold of those contributions, to anyone willing to invest time and money into it.

Yes, they have contributed back and I referred that in another thread, just not what they were supposed to under a copyleft license.


Long time ago, didn't Microsoft pulled the complete TCP/IP stack from OpenBSD/FreeBSD? I think they still have an /etc/hosts hidden somewhere in their filesystem...

Yes, it got rewritten on Vista.

> Where are you seeing these corporations ripping off FreeBSD?

It's not like FreeBSD has a shiny DE courtesy of Apple...


FreeBSD doesn't even have a DE, I have no clue what you are talking about here.

That is what I'm talking about. Apple has benefited from FreeBSD.

(There is Lumina, in terms of DEs.)


This doesn't make any sense. FreeBSD never had a desktop environment, so how is Apple ripping them off by not giving them the one they made?

The only way for me to make sense of your comments is to imagine that you're asserting not only that no-one would want to run a DE with FreeBSD but in fact that there is no way to run a desktop environment with FreeBSD.

However, that is clearly not the case, since Xfce and KDE and so forth work on FreeBSD. Further, there is even a DE developed for TrueOS, which is FreeBSD-based: Lumina ( https://www.lumina-desktop.org/ ).


If you download a release of FreeBSD, that image will not contain a desktop environment. If you look through there source code, no desktop environment, something also true of Linux. Of course you can install third party packages, including a range of desktop environments.

So, considering FreeBSD has no desktop environment, and likely little desire for one, how are they being ripped off by Apple's failure to give them their DE?


> If you look through there source code, no desktop environment, something also true of Linux.

Right, a DE is not part of the kernel, or even perhaps the base install. But DE is something people do use with FreeBSD - there are people who run DEs.

> FreeBSD has no desktop environment, and likely little desire for one

The development of Lumina is a BSD-associated endeavour, so the desire part does't seem quite true.

> how are they being ripped off by Apple's failure to give them their DE?

The DE is just an example of something that could have been useful. The point is that Apple hasn't really contributed back to FreeBSD - the real-world behaviour of large corporations highlights the drawbacks of non-reciprocal licences.


>Right, a DE is not part of the kernel, or even perhaps the base install. But DE is something people do use with FreeBSD - there are people who run DEs.

The base install is FreeBSD, it is the code and system the team created and what the FreeBSD foundational raises money to support. If Apple ripped off FreeBSD, it would these people that are getting ripped off. You can install a wide variety of other software on your computer alongside FreeBSD, but that doesn't make them a part of FreeBSD.

>The DE is just an example of something that could have been useful. The point is that Apple hasn't really contributed back to FreeBSD -

As far as contributing back unrelated things that would be useful, they have contributed back LLVM.

>the real-world behaviour of large corporations highlights the drawbacks of non-reciprocal licences.

This wasn't real world behavior, and FreeBSD using the GPL wouldn't have forced Apple to give them their DE. A program, say a DE, on a GPL operating system does not need to be distributed under an open source license.


> This wasn't real world behavior, and FreeBSD using the GPL wouldn't have forced Apple to give them their DE. A program, say a DE, on a GPL operating system does not need to be distributed under an open source license.

Depends on how the DE integrates with the other components.

Real-world behaviour: the ecosystem that has a GPL-licensed kernel has much wider adoption that the ecosystem which uses primarily BSD licences. I know this isn't fully attributable to licensing, since the BSDs were afflicted with legal issues during the time when Linux was taking off, which surely contributed to the situation, but still the BSD licences encourage certain types of entities to take without contributing back the ecosystem. Though I admire the idealism of BSD-type licences.


> Maybe we mean different things with the word, sharing

Pretty much this. The BSD community doesn't see code being used in proprietary solutions as "ripped off".


I am convinced that if the BSDs would be the only available free UNIX clones, most commercial UNIXes would still be a thing.

> but the BSD license is more collaborative. I'm paraphrasing here.

It's pretty much impossible to get anything significant in BSDs without a commit bit, Linux is much more open to external contributions. Moreover, any "large" work is mostly discussed privately and only a huge patch (ie. thousands lines long) will be published publicly. Reviewing such patch independantly is pretty much impossible.


To be fair, they do say "It is our intention to upstream any code which we can." and give the 3 reasons they have local commits, which all seem reasonable. I expect they will continue making contributions as they make finish features and generalize the features.

Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. So the fact that FreeBSD has made an amazing code base that can and does serve as the foundation of Netflix's distribution network is great; it gives visibility to the OS and also meaningful code contributions from a large player who pushes it to the limits. We may not get all of those _right_now_ because of NDAs and the realities of life, but it is all helping to make it more awesome.


> Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license.

There is no "distribution" happening; all of Netflix's use of BSD is internal. Even if they had chosen to modify Linux instead, there would be no requirement to share their changes.


Pretty sure they distribute these appliances to ISPs...

> Pretty sure they distribute these appliances to ISPs...

They co-locate them at ISP as a form of extremely distributed peering. Netflix, however, owns the boxes.

In the context of licenses such as the GPLv2, distribution happens when software transfers from one party to another. This doesn't happen: the software always stays on Netflix owned devices.


So the implication is sendfile does not use tls over an https connection for example? That is a shocking revelation to me if true!

TLS libraries like OpenSSL operate in user space (in-kernel TLS is very a recent addition to Linux).

sendfile(2)'s benefit is that it doesn't require transferring the data back and forth between kernel space and user space. You point it to a file descriptor and a socket, and it simply copies the bytes -- all in kernel space.


> in-kernel TLS is very a recent addition to Linux

Linux has gotten in-kernel TLS? That's slightly horrifying. Seems like it's hard enough to get right without being allowed to overwrite any user page you'd like.


Yes, but only the symmetric part. You handshake in userspace, then you pass the negotiated cipher keys/IV to the TCP socket with `setsockopt`. Last I looked, the only cipher supported was AES-128-GCM.

That's right. And if anything else happens on the connection (like a renegotiation) then it is just punted back to userspace.

Which is the correct way to handle, fast data path in the kernel, slow control path in userspace.

Ah, that's a lot more reasonable then.

Doesn't a lot of that come from the very long development history of OpenSSL (so best practices have developed over its lifetime), and the many, many compatibility hacks they still include in the codebase?

It also contains a LOT of different things in there, whereas "just" a TLS stream with limited ciphers should be a lot simpler.


Supporting TLS directly in a kernel is horrifying regardless of the legacy of OpenSSL. Having userspace do the session establishment (where all the weird stuff happens) and letting the kernel do the bulk ciphering is reasonable, especially if you're already using sendfile so the kernel can handle the bulk data sending. If something goes sideways, punt the socket back to userspace. Lots of kernels already have bulk ciphering for encrypted disks or ipsec (which is often very similar, userspace negotiates sessions and kernel handles per packet ciphering)

AFAIK that's all Linux's TLS implementation does -- ciphering & encryption, not session setup

Don't know about Linux implementation, but sendfile would only need AES, and today's CPUs even come with hardware support. I can't talk about security aspect, but it is not like the openssl is implemented in kernel or hardware, just the symmetric encryption and that only transforms the data.

There is ZERO technical details in this PDF beside 'we track head closely'

I was a bit naive when I said I wish Netflix had use Fast.com as their brand of CDN. Turns out their whole CDN are tailored to their special needs of Video and Video only. So it wouldn't work nearly as well if there were many small files or other CDN use cases.

What I really wished was I could have a CDN based on BSD. Even though you can't actually use BSD for lots of other reason, you could at last contribute a little by having some infrastructure that is built on top of BSD and you are paying for it. As far as I know only Limestone CDN are on FreeBSD, but unfortunately they only offer business to Enterprise.


All of the discussion points in this thread are valid but what the slides omit is whether they using ZFS?

Pretty sure it's still UFS+J. They've mentioned in the past that the benefits of ZFS don't really apply to the CDN boxes: if a drive dies, it's simply removed from the local cache, and files are re-downloaded/redistributed as necessary across the remaining cache drives the next time the appliance refreshes from its upstream origin. Additionally, ZFS prefetching was actually less performant in some cases from what I remember -- or at least providing no benefit -- because of how it couldn't really anticipate the I/O patterns of video segment requests from users.

Additional reference:

- https://news.ycombinator.com/item?id=8077594 ("…but caching is futile.")

- https://www.youtube.com/watch?v=FL5U4wr86L4



Thanks. ZFS doesn’t have to be used everywhere. And in media like this it probably makes little sense as they have all the tools custom built to deliver stuff to it and such.

The last I'd heard, they were using UFS.

Is ZFS that interesting for their use case?

Built in compression (probably not given the uncompressibility of video) but the protection against bit rot would be huge imo

Absolute shame flaunting open source as a backbone for DRM delivery.

Using Open Source to Deliver Streaming Bytes You Hope Are Just Video

^ better title, please update OP



Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: