Do you think they have a special deal w/ Amazon? Otherwise I imagine that's a really expensive bill.
Do you think they're doing it because its cheaper when you account for things at a larger scale? Like, reduced liabilities, payrolls, CapEx, whatever? Or is it more expensive but still "easier" and they go with it?
I really REALLY would like to hear everything from everyone about this, because right now I'm facing that decision for my company. I'm not as big as Netflix obv, but it is a big deal for me. It is really attractive to have all the scale you need at will and forget about maintenance at all; even if it is more expensive I would gladly pay as much as double my operating costs if I can rest easily at night. I would even eat as much as 10%-20% of my profits if I don't have to deal with that.
But the devil is in the details, I think is relatively easier to go full-cloud than to later find out it was a mistake and try to migrate back to your private premises. So, what could be the downsides of going full-cloud just as Netflix did? Also, taking into account that I'm not a personal friend of Bezos neither another Netflix, I would be just another smallish guy and I'm pretty sure that won't get me all the perks that come with the former.
You should talk to people who have done this IRL, not on HN. This is a difficult thing to pull off, and like any complex decision, there's a lot of trade-offs. Without knowing your specific situation, it's fair to say that volume helps out Netflix a lot, and you won't get anything close to the unit costs they have.
The major thing that sways people to one side or the other is the kind of problems they want to build an organization to cope with. Do you want to build an engineering team that can handle a whole availability zone in Amazon going out, or do you want to hire one that eeks performance out of bare metal? Do you want a sales organization that has to go to the mat over security details, or a marketing organization that has to cope with justifying why "you're not a cloud company, you don't use the cloud" in their brand? There's going to be bumps either way.
Source: I used to sell monitoring software, so I got to talk to a lot of people making this decision. We tended to find big opps when people made the move.
As someone who works on a far smaller video platform than Netflix commands, I can definitely say that AWS has given us plenty of freedom in places that planning our tiny slice of a DC has not. We dabble with dedicated hardware, but it takes so much more planning and budgeting - mostly of human resources. The budget we stick to is barely monetary, as it eventually gets quite a bit cheaper to run things (especially video transcoding) upon metal, but hours and mindspace are far more costly. The things we have started moving to metal are tested on the cushy and forviging pillow that is "AWS". We build and monitor there long before we consider moving the services to our own hardware and that conversation includes total cost, including those that are sunk into the massively larger upfront investment (buying / maintaining servers).
That said, we try not to build our system _within_ the world of AWS, but rather "on top" of it. Our backend software is written to run on any recent ubuntu distro, and moving elsewhere, whether it be another cloud platform or dedicated hardware, is more of an Operational issue than something requiring significant changes in our soft machinery. For instance, we stick with our own messaging services rather than SQS as it's more portable. And our databsaes are RDS (MySQL). I assume this is why AWS sticks to well-used protocols for its larger services, because it allows us to use their services with an open mind and not worry as much about vendor lock-in. This gives us room to grow and worry about the expense when the expense becomes a valid issue.
We're allowed to build on their platform without worrying about whether we can move on to another, provided we keep that in mind. As it stands, we're almost completely on AWS, but I run our entire platform on my [ubuntu] desktop without issue. The Ops are different, as I'm installing our software and dependencies manually, but the software itself can live on any server. And even as we consider dedicated servers, we keep in mind that we may want to continue using AWS for scaling during spikes, and so we continue to write our software and manage our systems accordingly - keeping in mind the idea that it should run on any system that meets our minimum requirements, whether it be bare metal, managed, or in the "cloud".
This depends on your current infrastructure, your expected growth over the next few years and obviously your software stack. In my experience, unless you are doing something with extreme latency or scaling requirements, AWS is likely cheaper in TCO, at least in the beginning where your fleet is less than a few hundreds machines. Also note that AWS is not just EC2. Other services like S3 or Dynamo can be life savers. The downside of AWS is that you can become over reliant of it.
Last first: Getting enough competent people to whatever you have. Just getting people that knows the difference between rm -rf / and rm -rf ./, not to plug both ends of a cable into the same switch/router, ... can be quite hard. If you have a premium product running on a few hundred machines, AWS doesn't actually eat that much margin compared to having your own 24/7 staff & all that entails.
And when the company then decides to open up "somewhere over in that big asian market", you've got to do it all over again. You can get a lot done with "remote hands" and whatnot, but again - something that eats your margin, just as AWS does...
you still need ops guys just not hardware guys. for a smaller company it would be the same guy anyway no? a company with 10 or 20 servers doesn't need a dedicated hardware tech.
If you watch any of Cockcroft's presentations you can see that there's nothing easy about it. They developed a significant amount of tooling just to work around quirks in EC2, but apparently the elasticity is worth it.
Assuming Amazon can do it slightly cheaper than Netflix could (i.e. Netflix doesn't have to duplicate Amazon's fixed costs, such as engineering), and that Netflix has enough bargaining power to pay Amazon very little over what it costs them to provide it....seems like an economic decision to me.
It depends on your IT architecture and more importantly on your software architecture. I did this IRL and I did it both ways and I'm doing it again.
If you want some ideas or guidance or want to ask specific questions - feel free. My email is in my profile, "about" section.
Disregarding the fact that the E stands for Elasticity...
Yes, Elasticity is THE advantage for Netflix when it comes to using EC2. Consider the vast differences in usage they have in North America during the evening after dinner vs during the middle of the night. I can only imagine their fleet capacity goes up and down drastically every single day. This amounts to huge savings over having the bare metal.
At work, we use a dedicated server provider, so we're; when we identify faulty hardware, we file a ticket, and they swap it out. We still need to have enough bare metal machines allocated so we can turn one off while they fix it, etc; but we don't worry about staffing or supplying the datacenter.
I wouldn't want to switch to AWS from this; bare metal plus a good account manager gives us a lot of visibility and reliability; Netflix blogs a lot about the chaos that is AWS, and we save a lot of effort by having machines that for the msot part just work, consistently; instead of having to deal with instances where performance is variable, etc. Running our own datacenter might be nice for some reasons: more control and information about the network, ability to make hardware choices that don't match the providers offerings, etc; but running a datacenter is a big change.
Sorry, I don't get it - why would you consider using cloud servers if you don't have significant spikes in your traffic?
Regarding hardware failures - it's not that hard to arrange proper redundancies in your systems, and it will certainly be cheaper than outsourcing it to Amazon. Cloud only becomes cost effective when your spikes are several times larger than your normal traffic, and you must maintain good QoS at all times.
Because sometimes cloud servers are very cost effective, easy, and fast to deploy: Digital Ocean for example.
$40 / month for 4gb of ram, 4tb of transfer, and two cores, is very cost effective. Linode is also very reasonable.
If all I need is that scale of computing, what should I use instead that makes sense other than cloud services? Setting up colo or a dedicated box for that would be more expensive and a much larger hassle.
Arguably, Digital Ocean is more like a regular hosting provider with better onboarding. Plenty of hosters offer comparable infrastructure for the same price (you can even get something dedicated).
They're also far cheaper than AWS, where you pay a premium for the elasticity and access to the ecosystem. If there's anything to compare AWS with, it would be Google Cloud Platform, Azure and maybe Rackspace.
Well, I admit, those prices are pretty good. I looked at AWS pricing 3 years ago at a SaaS company with steadily growing traffic, and at the time it didn't make sense. It might now.
That really depends on a type of workloads and requirements you have. But basically you (your team) have to be smart about how AWS or any other cloud provider is used. No cloud buzzword alone will help you avoid cost, availability issues etc otherwise.
As for the "perks" i.e. discounts talk to AWS account rep.
Netflix is still running its own CDNs / using CDNs outside of Amazon.
"The company, which competes with Amazon in the video-streaming business, elected to keep control of its own content delivery network."
So Netflix is still probably doing the bulk of their traffic themselves. Application servers with AWS is probably not a huge bill versus massive video traffic.
Fascinating. I bet they are running their own CDN to prevent Amazon from having full access to their traffic volume/streaming demand. Would be huge competitive advantage for amazon streaming if they could see per-title demand.
I was wondering about that. Open Connect was significant in the disagreement between Netflix and Comcast that spurred the net neutrality debate. Unless I'm mistaken Netflix ended up agreeing to the peering fees, though after negotiating down a bit. So does using AWS for mundane traffic actually save them money by avoiding the CDN bandwidth that Netflix has to pay ISPs to use?
There's literally no way to operate a CDN-like service from AWS; no matter what discount you get from Amazon, you're paying per-byte instead of 95th percentile billing and settlement-free peering.
This is very strange. Amazon is in the pay per view streaming video business itself.[1] Netflix is outsourcing their entire business to a direct competitor. At what point will Amazon decide that they don't need Netflix?
Or perhaps there's an acquisition or merger coming.
> But many companies still run sensitive software in their data centers or in private clouds, in which a company has dedicated cloud resources from a third-party or within its own premises.
(Emphasis mine)
Ok, honest question: If I have a "private cloud" on my own premises, how is that not a data center?
> If I have a "private cloud" on my own premises, how is that not a data center?
It is, but "private cloud" on top of that implies a lot more about how you run that datacenter: using virtual machines, abstracting hardware, using APIs to manage servers and storage (ideally the same APIs that would let you just as easily run on someone else's cloud infrastructure), and otherwise trying to minimize your reliance on any particular piece of hardware.
Thanks. I was looking at it from the power, cooling, and square-footage perspective—didn't even consider the sysadmin portion—but I see the massive savings that could be had by uncoupling metal pizza boxes and one-off hardware requirements from the applications that run on them
The free market is the basic guarantee Netflix is counting on.[1] If Amazon 'jacks up it's prices', Netflix will simply move to another provider. Amazon is also unlikely to make sudden moves which disadvantage its customers, as any such moves will come at a tremendous cost of reputation, and many customers will leave and never come back (; as Amazon has no power to compel customers to purchase their services). In a market, no one has leverage, everyone simply has a position and an offer.
> If Amazon 'jacks up it's prices', Netflix will simply move to another provider.
Like hell they will. The switching cost for Netflix, and really most heavy AWS users, would be extreme at this point. A good portion of the engineering org would pretty much have to dedicate themselves to the task of switching. That isn't to say other organizations can't be nimble between providers, but it would be hard for Netflix at this point.
AWS is already engaging in a commodity hardware market in order to run the DCs running. AWS fears becoming a pure extension of the same thing, and they have a decent bit of lock-in designed to their services for that reason.
I remember seeing a talk by a Netflix engineer (can't find it unfortunately) where he talked about multi-cloud strategies and how they aimed towards decoupling some of the components of his project from implementation details specific to AWS. So it's quite possible they're already slowly paving the way towards becoming a cloud-agnostic or multi-cloud platform. Not to say that it wouldn't still be a huge undertaking, but I don't think this never crossed their minds before.
Considering how many millions Netflix pays Amazon every single month, a team of highly skilled engineers is a drop in the bucket, and if switching means they can save enough, they'll do it.
Compared to the running cost of their streaming service, yes. Besides, I like to believe (yes believe, because non of us outside of Netflix would be able to verify) that they've spend that much time such that they're far less dependent on the underlying architecture.
> I've never understood why so many companies are comfortable giving Amazon such leverage over their business infrastructure.
You have to be a pretty big company in order to not let another company have leverage over your infrastructure. Hosting your networked servers in your own office? You're depending on Verizon, AT&T or Qwest, even if those are not the companies selling you your pipes.
Co-locating your servers, getting feeds from 2-3 carriers, while doing BGP with your high-end Cisco routers? You're depending on the hosting facility, such as Telehouse.
So you want to survive a blackout in one section of the US. Now you need at least two server facilities. Your remote facility needs at least one full-time tech.
Corporate infrastructure can get in the way as well. Your division of the conglomerate is in San Francisco, but another division is in New York. Perfect, your division's remote IT person can work out of that division's New York office. But the conglomerate might be structured so that is not really possible. The other division may as well be another company.
So a small company can't afford to do this. A big company can get so big that it's not feasible either. If the big company has one central IT department that all divisions use, it becomes more feasible, but how many big companies only have one centralized IT department? Even if such a creature exists - the big company buys a small company, or merges with another big company - boom, two IT departments again.
Wait... since when has AWS ever raised their prices? They've consistently lowered prices for better services... something like 50+ price reductions in the past 7 years.
Total guess on my part, but I'd imagine Netflix has locked-in long-term pricing. Of course, there's the opposite danger of the rest of the world getting cheaper...
If they're savvy, they'll have locked in long-term maximum pricing. Something along the lines of "prices will at most be the minimum of this discount price, or x% above the standard retail price".
From my understanding this is a simple decision: do not focus on something that isn't your core business. Amazon is doing it well, Microsoft is, the cloud is becoming more competitive every day, and they're big enough not to use the public pricing, so it's even more economical for them.
In addition to this, they only need a few "machines" (https://openconnect.netflix.com/hardware/) to have full ISP-local caches of Netflix; why bother with a full datacenter ?
People on HN just don't understand that the Basic Hosting company won't give a scalable Network for free. And with scalable Network I mean a Physical / Virtual Network where you could connect / disconnect Nodes and connect it with arbitrary Networks. And that's basically something which costs a huge amount of money especially when you want to connect two Locations.
When you need HA and don't get such a network you would need to create a Overlay Network over VPN, etc by yourself which could be quite hard to manage (even for a low amount of servers)
In germany we could buy Virtual servers as low as 5-10 Euro / month however you can't buy 3 in different locations and they are connected with an private internal network for free (there aren't even many hosters that will do that, i don't think there are any outside of the saas platforms) however you could rent non virtual servers and buy additional stuff on these Hosters which adds a huge costs per month. So running on AWS / Google Cloud / Azure / etc could be really cheap. It just depends on the Things you need.
Edit: Also AWS gives you a lot of credits for just doing surveys for their new stuff. I have so many credits so that I run my private gitlab instance on AWS and it costs me nothing. I never got a credit of my previous Hoster which I used over 10 years. On AWS I just got a 200 USD Credit at the start since I missed my Free year (also only cloud providers giving you credits, the most other Hosters won't do that and they bareley give you something new on AWS. You could now run a Website for basically nothing just with AWS Lambda which costs nearly nothing.)
For better or worse, "public cloud, in which users share the resources of a service provider" is now an accepted term of art in the industry. It doesn't really matter whether it agrees with people's commonsense understanding of the word "public".
I'm on FF without adblock extension, so at least for me it wasn't. I'm in Japan however, so I'm probably completely irrelevant to the operators of that webzone. "What are you telling me, Johnny? There's people outside the US who have access to the intertubes? Getoutofhere!"
Most newspapers don't show paywalls to users coming from a search engine, as part of a compromise with Google to prevent them from being penalized in rankings.
Do you think they're doing it because its cheaper when you account for things at a larger scale? Like, reduced liabilities, payrolls, CapEx, whatever? Or is it more expensive but still "easier" and they go with it?
I really REALLY would like to hear everything from everyone about this, because right now I'm facing that decision for my company. I'm not as big as Netflix obv, but it is a big deal for me. It is really attractive to have all the scale you need at will and forget about maintenance at all; even if it is more expensive I would gladly pay as much as double my operating costs if I can rest easily at night. I would even eat as much as 10%-20% of my profits if I don't have to deal with that.
But the devil is in the details, I think is relatively easier to go full-cloud than to later find out it was a mistake and try to migrate back to your private premises. So, what could be the downsides of going full-cloud just as Netflix did? Also, taking into account that I'm not a personal friend of Bezos neither another Netflix, I would be just another smallish guy and I'm pretty sure that won't get me all the perks that come with the former.