I didn't meant external networking.
Connect two datacenters, see what costs come up.
Connect machines inside your datacenter, etc..
Your maintenance cost will raise.
AWS is cheap. Whatever you did, you operated on a really slow scale where money didn't mattered. Or you completely forgot a lot in your calculation.
> AWS is cheap. Whatever you did, you operated on a really slow scale where money didn't mattered. Or you completely forgot a lot in your calculation.
I've managed thousands of instances in AWS, and I've managed thousands of servers in colos, both owned (DOE facility, web hosting startup, consulting firm) or with the space leased (Equinix, Savvis, Cogent, IBM).
You could most definitely run elastic workloads in your own datacenter now using Docker containers, and one of the many orchestration frameworks that exist for it.
As I mentioned above, if you don't have a predictable workload, AWS is great. Exceptional even. But if you know your workload and can hire 2-3 people to manage it (which you should be doing at scale), its cheaper to get your own gear and colo it.
> You could most definitely run elastic workloads in your own datacenter now using Docker containers
Have you actually done this though ? Because I have extensive experience with both Docker and OpenStack and I am sorry but neither comes close to AWS for stability and reliability. They are both really buggy.
Also you seem to be completely missing the either side of AWS which is the software. Using ELB, SNS, SQS, RDS etc. can save a lot of money.
> and I am sorry but neither comes close to AWS for stability and reliability.
All of my instances in us-east-1 lost outbound connectivity Sunday morning for half an hour, and an hour last night. Tell me about this stellar stability and reliability.
As long as you design your application properly, it'll be reliable wether you run it in AWS, on OpenStack, or even bare metal. StackOverflow does it, you can do it too (they colo all of their equipment in NYC). I have not done this with Docker or OpenStack, but I have using Xen and KVM virtualization (Docker would be about the same as using lxc).
ELB? Haproxy. RDS? MySQL/Postgresql/MS SQL (in that order), unless you absolutely have zero management experience with it. I concede SQS and SNS have no open source counterpart; you'd have to roll your own.
I am not talking about the individual nodes. I am talking about the overall platform.
This idea that you can design your application properly and it will be reliable anywhere is ridiculous. If the underlying platform is fundamentally flawed (i.e. more than just nodes coming/going) then your options are very limited. Docker and OpenStack are examples of platforms that are really buggy when used in Production settings.
AWS as a platform is very stable and of a generally high quality. And the components like ELB, RDS etc are managed. Trying to manage the hardware AND all the core infrastructure components on top is more than just one person's job like you mentioned before.
> This idea that you can design your application properly and it will be reliable anywhere is ridiculous.
Thats not true at all. If you've designed your application that any instance could fail at any time, it doesn't matter where your instances are hosted: AWS, Digital Ocean, OpenStack, or your own bare metal servers. As long as you've architected the environment to pass traffic to only health nodes (you're doing service checks, aren't you?), healthy microservices (again, service checks), you can promote slaves to masters for your datastores, and you have sufficient capacity for failures, yes, you can design your application to be fault redundant on any infrastructure.
TL;DR If your app is designed for high availability and is already structured to scale out horizontally, its doesn't matter what the underlying infrastructure is.
managing != operating them.
You could manage a thousand servers, but ONE PERSON just can't operate thousands of servers.
also how did you managed file access in a scaling manner? did you've setup hdfs? and when yes how did you provided your developers that space? did you created some kind of infrastructure for that (libraries, etc)?
There are so many things, which just doesn't work out.
As already said, you've just managed it. But other people did the work, that this could work out and they needed a lot of time to get things done.
Take a look at google, they always try to bring costs down, through things like borg, kubernetes, etc. but they throw thousands of people at those problems, also at the networking thing, they have a huge team of PROGRAMMERS that try everything that the management of their network will be easier.
also if you don't need things all the time, why should you buy it?
(The below was ~5 years ago; it would be even easier today)
> You could manage a thousand servers, but ONE PERSON just can't operate thousands of servers.
Yes, I literally managed AND operated ~5500 linux servers in a facility. Remote power management, serial console access, PXE booting. I would replace any failed hardware when convenient, usually after my morning coffee or in the evening if I wasn't in a rush to get home.
> also how did you managed file access in a scaling manner?
Shared NFS with work distributed to servers on-demand or scheduled, based on workload.
> and when yes how did you provided your developers that space?
Limited SSH access to their environment; diskspace was provisioned in minutes. Scheduled jobs could be executed using an open source scheduling framework.
> did you created some kind of infrastructure for that (libraries, etc)?
No, the majority of the tooling was a few thousand lines of bash scripting
> There are so many things, which just doesn't work out. As already said, you've just managed it. But other people did the work, that this could work out and they needed a lot of time to get things done. Take a look at google, they always try to bring costs down, through things like borg, kubernetes, etc. but they throw thousands of people at those problems, also at the networking thing, they have a huge team of PROGRAMMERS that try everything that the management of their network will be easier. also if you don't need things all the time, why should you buy it?
Not knowing how to do something does not make it impossible or hard, simply an unknown how to do it.
Are you saying of something major broke at 2AM on Sunday you where the only person who would be available to show up and fix it? If so, your adding a lot of risk for major downtime ex: Your hit by a bus. If not, then you did not manage 5500 servers by yourself.
PS: I could see a team of 10 people handling 55,000 servers though. But that's serious scale and fairly lean.
I never said I was the only person; we were talking infrastructure cost savings, not human resources. Using AWS doesn't magically mean everything manages itself.
To directly answer your question, people from other teams were available if for some reason I was not.
Human resources are a major component of infrastucture costs. Sure, for small teams you can have someone doing everything but at scale you lose the fudge factor.
I would also be suspicious of 1:10,000+ numbers as they often don't include people that are part of the server team but not standing in the data center. Aka Facebooks 3 person dev team, a manager, whoever designs there HW, the team spun up to do mass HW installations etc.
In the end AWS covers a lot of seperate costs which are only really obvious as you keep scaling up.
Most applications/companies don't need to scale to hundreds of servers, let alone thousands... Having the number of employees to setup and manage that level of infrastructure has a not insignificant cost. If you can have your dev+ops team not have to have as much specialized knowledge and get more done with AWS, Azure or similar, then there are definitely costs to be saved. Of course the more of AWS/Azure infrastructure you leverage the more locked in you are...
I'd like to see a similar analysis for say 50-100 servers, and see if AWS really costs more vs. colocation when you add in the resources to setup and maintain the infrastructure.
> As mentioned, Etsy and others have gone against the cloud tide and seem to be making it work. Etsy CTO Kellan Elliott-McCrea told me that the online marketplace has discovered "very real cost savings" and higher utilization by running in its own data centers.
Even your own article contradicts your claim. :P
Yes, for people who can't capacity plan effectively...paying someone else to have idle capacity makes sense. For the rest of the world, it doesn't.
Most of the world doesn't need hundreds or thousands of servers... most businesses need fewer than fifty or a hundred. And growing is pretty easy... Once you hit a certain scale it absolutely makes sense to consider migrating, but that's the point where you have teams of dedicated employees.
And even then the costs of moving off of the cloud solution may never be worth it.
Umm. Look, I really don't think you know what you are talking about and you seem really eager to defend your whole "USE AWS BECAUSE ITS SO EASY" mantra as you replied to every almost all of the posts I made in this thread.
So I'm going to distill this into one post:
> Most of the world doesn't need hundreds or thousands of servers... most businesses need fewer than fifty or a hundred. And growing is pretty easy... Once you hit a certain scale it absolutely makes sense to consider migrating, but that's the point where you have teams of dedicated employees.
Fewer nodes actually means the cloud makes less sense not more...often you can get away with 10 1Us in different DCs. At that scale, "the cloud" makes even less sense than you think it does. Leasing dedicated servers is cheap compared to the cost of automated deployments of 50 VMs, especially in labor costs. At which point, we are just talking Active/Passive deployments across 2 datacenters and such.
Amazon really, at this scale, isn't giving you anything you can't do for yourself. The only time AWS seems to make sense is as a 1-2 man shop where you literally don't have enough time in the day or at extremely volatile loads where you can't capacity plan. Most businesses are neither of those.
> And even then the costs of moving off of the cloud solution may never be worth it.
Yes, that is called vendor lock in and that is what AWS, etc. rely on. If you don't develop the in-house talent, you can never move off without problems. Waking up one day and going "LOL I WANTS TO MOVE" is a bad idea.
> Really? could you elaborate on these other providers? I would expect at least, VPS, remote files (S3/Azure-Blob), distributed key-value store as a service, some SQL variant as a service, and load balancing as a service. Most of the alternatives I've seen don't offer those services, and most only do VPS.
Honestly, if you can't figure this out on your own without relying on AWS at the scale of 3 people...you have no business discussing this, frankly. This is a case of the blind assuming everyone else is.
I have hobbies with 99.5% uptime that essentially have all of those functions serviced for cheaper than AWS.
> Linode, and most alternatives, don't have S3/Blob storage as a service.. also missing are distributed key-value storage as a service and sql as a service.
...this is one of those times where I'm tempted to just run a site that includes all these things as a hobby just to point out how stupid this comment is. I don't really need more hobbies but this YC fanboy behavior is ridiculous. Well, except for the "kv/sql as a service" because that is just silly and can be done just fine with traditional clusters that have existed for years.
> Generally, with any given service as you can scale it makes sense to run your own instances.. but realistically someone who knows insert-tech better than AWS/Azure admins is a pretty rare breed, and having to in-source those specific skills costs in terms of time and money, redundancy even more so.
> If you have a team of 3-5 people, you can do far more with AWS or Azure than you can most alternatives... Employees aren't free.
It seems rare to you 'cause it isn't something you are good at. AWS/Azure simply aren't cost effective. Fully loaded sysadmins cost for someone compentent is like $30/hr. Setting up an internal cluster for any of those things is much, much cheaper than AWS is after a year for literally every business I've ever dealt with IRL.
If I'm running all my own services... now I need to Hire people to manage my OSes for the app, as well as systems for databases, key-value stores, backup, mail, load balancing, etc, etc... few people are good at more than one or two of those things, which means I'd have to hire 2-3 people to cover those skills. 2-3 people at even 100k ($30/hr + employer taxes, etc), is quite a bit expensive than AWS services extra costs. Those are people you wouldn't necessarily need if you leveraged those services from AWS.
And when you say, you can just hire people to setup, and run maintenance via contracted services... I've tried hiring a PT postgresql dba, and it was pretty much impossible to do... those services I did reach out to mostly just plain didn't respond, even with a quote.
> as well as systems for databases, key-value stores, backup, mail, load balancing, etc, etc... few people are good at more than one or two of those things, which means I'd have to hire 2-3 people to cover those skills
That just isn't true. If those people were such unicorns, everyone would beg to hire me because I'm worth 2-3 people. :/ I'd be able to get a job anywhere with no trouble.
Given that isn't the case and most people find me unimpressive...yeah. I just think the problem is you, honestly.
Oh and here is a Link from Zynga: http://readwrite.com/2015/05/12/zynga-data-center-amazon-aws... My next examples would be Gilt, etc. You can't run elastic in your own datacenter.
From your article:
> https://blog.serverdensity.com/saving-500k-per-month-buying-...
> Dell R415 with x2 processors each with 8 cores, 32GB RAM, a 500GB SATA system drive and a 400GB SSD
This one made me laugh...