

How Netflix deploys 2 Terabits in 12 hours [pdf] - aditya
https://www.nanog.org/sites/default/files/tue.lightning1.general.temkin.pdf

======
donavanm
i didnt catch it in the pdf but a single deployment is supposed to push
200tb/s. Id ball park the capital cost aaround 1.25M USD. the original 4u
boxes do/did ~9gb/s of prod traffic each. peak/lab util was about 15gb/s. IIRC
two of these boxes could cache nearly the entire library for a region.

The interesting bit is the new 2u hosts used specifically for hot content.
only using 3x 10gig interfaces because the bus is too narrow. im guessing
these hosts are pushing 20-24gb/s at peak. theyll have to be using a zeroish
copy like sendfile() to just dma everything from disks to nic. looks like the
14 ssds can push 450-500MB/s, so theres the bus limit again. read scheduling
will probably suck at that rate. but id wager that most of the bits served are
super hot content being dmad straight from memory pages.

------
bradhe
Amazing to see the on-prem chops of a company who is so cloud-oriented.

~~~
Corrado
Yes, I thought Netflix was completely cloud located. I realize now that I've
been naive in my thoughts. It looks like they developed their own CDN, which
makes me wonder what it was about the existing solutions they weren't happy
with. Too slow? Too expensive? Not enough control?

Ah, I bet its the control one. The slides mentioned that they actually knew
what the customer wanted to watch before the customer did. This allows them to
push content down to a local node in advance of the customer requesting it.
Wow, that's amazing and scary all at the same time.

