It's really two questions: Why choose Ubuntu for the cloud, and, why choose FreeBSD for the CDN. We believe that's the best choice for both environments. I was trying to type in an explanation here, but that's really something that will take a lot to explain (maybe a Netflix tech blog post).
If you do write that blog post, it would be cool if you not only covered the FreeBSD vs. Ubuntu aspects of the choice, but also the Ubuntu vs. other Linux aspects (particularly Debian).
Netflix CDN (Open Connect Appliance): lots of physical boxes, FreeBSD.