All the original internet companies run their own hardware. They rent out excess production capacity to us peons in the form of cloud services.
These companies that all run their own hardware exclusively are telling everyone that it's stupid to run your own hardware... Why are we listening?
Look to the newer tech giants as a prequel of what's to come. Netflix for example, is completely at the mercy of Amazon. They might as well be "Netflix brought to you by Amazon". Their edge is in software alone, nothing that causes a huge barrier to entry like custom hardware. This makes them much easier to dethrone. Hilariously, they rent all their hardware from a direct competitor who has access to all their software secrets. Does anyone else see something wrong with this?
The cloud as a money saving venture is and always has been a damn lie.
It's the same tactic as when automotive companies paid off local governments to destroy Americas public transport many years ago. All the big tech companies have their hands in the cookie jar besides Facebook, who remains mostly silent but runs their own hardware as well.
The major tech companies have every incentive to make you think running your own servers is nigh impossible. Don't drink the fucking koolaid. If the industry continues to consolidate rapidly AmaGooSoft will be the ONLY places you can have web servers in ten years
"Netflix operates its own content delivery network (CDN) called Open Connect. Netflix manages Open Connect from Amazon, but the storage boxes holding videos that stream to your house or mobile device are all located in data centers within Internet service providers' networks or at Internet exchange points, facilities where major network operators exchange traffic. Netflix distributes traffic directly to Comcast, Verizon, AT&T, and other big network operators at these exchange points."
Everything else - such as the Netflix UI, your account management, billing, etc - is on Amazon. But Netflix has an enormous amount of hardware distributed around the world. If you wanted to compete with Netflix, you would need to do a lot more than uses Amazon's cloud services.
Yes you'd need multi-million dollar contracts with all of the major studios. Netflix isn't as much a technical problem as it is a business problem.
HBO has a far worse network, UI, and technology in general but its still competitive simply because it has Game of Thrones and other shows.
After pioneering streaming and realizing that the studios were asking themselves why they don't just load up their own computers with movies and get paid to stream them out, Netflix was forced to start its own content division to keep subscribers onboard and prevent the studios from poaching all of their customers by failing to renew contracts for content.
We have a similar problem in online services. It's impossible to compete with Facebook because the law allows them to hold data hostage.
We badly need copyright reform. The current incarnation is impractical in the digital age.
Copyright should be adjusted so that people can make money off of their creations without allowing the culture to be held hostage to corporations. If the copyright term was shortened to, say, 10 years, a Netflix-with-only-content-from-10-years-ago would be totally viable without having to ask for anyone's permission.
Copyright is a government-granted monopoly. It's gotten totally out of control and severely restricts competition in this type of space.
The movie industry has Long spent its years fighting piracy instead of providing a solution to a problem. Now there are solutions to the problem and the industry is refusing to adapt and continues to hold the content for themselves instead of licensing it to the likes of Netflix.
We know the studios have the right to license content because that's exactly what they do when they sell licenses over seas to broadcasters in other countries to air content.
This isn't about copyright. It's about the studios lack of ability to license content.
This is why in 2017 piracy is still a major issue.
This isn't even getting into exclusivity licensing that the studios seem to be real keen on doing with Netflix, Amazon, Crackle, VUDU and a jillion other streaming services. You know those shows that are branded "Netflix Original" on Netflix? Those aren't literally made by Netflix, they've just paid for some/all of the production and licensed them exclusively for some period of time. They're still produced by all the same studios that have been producing shows and movies for years.
Right now, competition is hamstrung because people are prevented from entering the market with anything that the consumer would find desirable, barring permission from the rightsholders.
Competition is not real competition if the same data cannot be accessed. For example, Chrome and Firefox compete because they both access the same internet, and whichever browser provides the better experience for that internet will win out. That kind of thing is not possible on walled gardens like Facebook because the law allows them to obliterate anyone who would create an alternate "Facebook browser", and it's not possible with things whose primary offerings fall under copyright protection, like Netflix.
We need to open these things up so that people can compete and provide the consumer with the best experience without needing anyone else's permission. It's true that most people will want access to new releases, but if copyright expired after 10 years, you'd at least have a secondary service that people would actually want to use, which would compete with Netflix et al for viewership on a lot of the content.
One of the big points of the cloud is that is gives one the ability to do things that one would trouble doing without the cloud. A small business with a single part-time developer can deploy an app to Heroku, and not need an ops person. That sure beats the heck out of the small business not being able to deploy software at all. As a former small biz owner, I did not want to have to manage a bare metal server in a colocation rack with a not-cheap data pipe. I just wanted a CRUD app with a minimal feature set.
When Heroku went down, I sat around wishing it would come back up, and eventually it did. That was way better than having to service a physical box when it failed.
Heroku cost me a few hundred a month, and freed me up to earn much more than that. In my eyes, it paid for itself many times over.
They employ their ops team and their ops team has equity stake in the company's success as well as their career riding on the line.
If a chunk of Amazon goes down and takes out your website, all you can do is wait and hope Amazon is prioritizing the fix.
>When Heroku went down, I sat around wishing it would come back up, and eventually it did.
This method of recovery just doesn't cut it in businesses with lots of transactions 24/7. Just sitting around can be flushing thousands per minute down the drain. In those deployments, you need to build across multiple cloud providers; and once you've reached that abstraction, you might as well have your own hardware as well and just burst to providers.
A company like Netflix can get by because they have their own hardware for streaming so an outage only affects people browsing for a movie at the time. And on top of that, their only interactive transaction is signing up for the service, which is extremely infrequent compared to the rest of the traffic. So essentially it's safe for Netflix to use AWS because they don't need high reliability/control of their own destiny.
>A small business with a single part-time developer can deploy an app to Heroku, and not need an ops person.
This use case I understand. However, anything at the scale that is frequently discussed in these contexts (gitlab, etc), they need ops people.
Well. AWS went down (part of it), but Netflix stayed alive.
As soon as you can confidently predict sufficient demand, the economically rational decision is to hire a good ops team and run real hardware. But to do so, you need to either have money or get money - and the costs of cloud service are zooming up, potentially leaving you without the margin to invest.
The question is what that level of sufficient demand is.
I feel this is one of those "easier said than done" statements...
An organization of even medium size? Not a chance.
Sure, paying extra for a dynamically scalable service is inefficient if your resource needs are flat 24/7/365, but is that realistically the case?
(OTOH, one must not forget that there is a cost to engineer systems to take advantage of dynamic scaling to minimize costs while meeting requirements.)
Yes, the vast majority of apps are not 12 factor architectures designed for scaling so nodes can't be brought online and killed at the flip of a switch.
Most tactics in business are money-saving tactics, and cloud is clearly one: dynamic scaling means only spending now for what you need now rather than spending now based on anticipated future need, and someone-else's-server means economies of scale in ops.
This also explains why older tech companies disproportionately use their own hardware, even those that aren't all that big. They all made that decision at a time when it made sense and the switching cost is sufficiently high to prevent them from turning back now.
I doubt there are many rational cases now for newer companies to use anything but the cloud. Even at the scale of Snapchat (>3B$ infrastructure spend over next 5 years) and Netflix, it appears the public cloud is still the rational choice.
So while it may be worthwhile if you're huge enough to get a big discount, that doesn't mean it applies to small or medium size companies
If it requires no or a smaller ops team then it seems like it might be.
Could be solved by more staffing, I suppose...
If you needed a team to manage it before, you still need Ops with "the cloud"
What do you mean by this, because it sounds like a straight up lie?
Amazon and Google are in an ongoing price war and cut costs regularly.
There is no economy of scale with the cloud. As you scale up your service, Amazon/Google happily scale your bill right up with it. When you roll your own datacenter, things get much cheaper per unit (bandwidth, storage, etc) as you get larger.
Obviously if you're growing your costs will grow, but that's basically a tautology. Bare metal costs also rise as you grow.
Except you're gonna need between 10 and 100 people to recreate the service in-house, not just one.
Trying to re-do something from scratch in-house is always more expensive than just buying the product.
while true; do
I don't mean to say that shell scripts always can replace more complex systems, but sometimes they're enough.
Actually, they are a media company. Their "edge" is in content creation and licencing deals. They might have been first-to-market for mass-scale streaming, but streaming video technology is a commodity, and they know that they will live or die by the desirability of their media catalog.
> Netflix for example, is completely at the mercy of Amazon.
Amazon and Netflix were symbiotic for years, as Netflix was a headline corporate customer, and I'm sure they received preferential pricing. And given the quality of Netflix's engineering team, I would be very surprised if they don't have a plan to migrate by the expiration of their contract if necessary.
That's only partly true. But if you considering a lot of other "media companies" haven't been able to compete with Netflix's ease of use, then you have to ask the question how is Netflix different from any other media company. And the answer is because they are also a software company, where their successes is primarily attributed to making DRM video in browser easy and seamless for average folk.
The original success of Netflix depended on being the first company with a fixed-price-no-ads streaming business model in addition to really good user experience across a wide variety of platforms. I remember being astonished at watching a Netflix movie on my iPhone (when I was still an iPhone user) while riding a train in 2011'ish. It was miles ahead of any other competitor in providing solid, usable clients on multiple platforms combined with reliable streaming in most cases and networks. This was as much a triumph of technology and product design as it was of the content available. If it was only about the content, it would be Hulu (at least in US) instead of Netflix who would be in pole position today.
Actually, the original success involved putting DVDs in mailboxes.
> I remember being astonished at watching a Netflix movie on my iPhone (when I was still an iPhone user) while riding a train in 2011'ish.
Right, but you're going to be equally astonished in 2017 if you don't get the same level of service from Prime Video, Hulu, HBO GO, YouTube Red, et. al. And in 2019 a YC company will offer a one-click media publishing company that will allow any random joe on the planet to publish a DRM-protected movie available through a world-wide CDN with a pay-per-view payment system.
In 2011 tech was a major differentiator, today not so much. Netflix is moving with the times, that's why they are offering original content instead of being content to provide a good platform to stream other people's IP.
> This answer is in the same category as "Actually, Google is an ads company". That is to say, it makes a plausible case but ultimately wrong.
https://qz.com/607378/were-live-charting-googles-first-alpha... (tl;dr: the overwhelming majority of google's revenue comes from ads)
> Prime Video, Hulu, HBO GO, YouTube Red,
So Netflix has a six year lead and attendant network effects in it's favor. If you don't believe me, I created a facebook clone in PHP last weekend which works just as well (or better) than the original Facebook. Do you want to join it?
I completely agree with you. I personally have a Netflix account, and do not subscribe to other services.
My point is that right now, today, this very instant in time, Netflix is not primarily differentiated on technology, but on having content that their subscribers want; that online video distribution is now a commodity.
This is where we disagree. I don't even know what content Amazon or HBO or Hulu has at this point. Netflix has already made me a lifelong customer with their product and user-experience in the past 10 years or so that it'll take an earth-shaking difference in content for me to switch (or add) a second monthly bill for content. Netflix is part of social fabric (at least in the US). All of this was won by relentless focus on an excellent product and user experience... and perhaps only a middling content catalog.
The question is, does it make sense for businesses with a handful of servers running their office environment, to a few thousand servers running production for a moderately popular web service, to pay consultants to drive out and tend the servers in their office closets/colo racks, rather than piggy-back on someone else's (mostly) well-oiled machine with economies of scale.
If you can build an in-house equivalent of the AWS team, do it. If you're going to pay someone to do it for you, AWS might be a better deal than your neighborhood business IT firm.
Using Netflix as an example again, if they're pitted against a company that runs their own hardware but is otherwise equal, they will lose. A portion of their profit is siphoned off as Amazon's profit. What's already happened is Netflix is directly funding development of Amazon Prime Video.
Or a traditional business renting a storefront? Something that happens all the time. Even huge companies like Apple use OEMs. Samsung manufactured Apple chips for years despite putting up a direct competitor to apple in the Galaxy S.
Systems like this allow people who do not have capital to scale up. Just like developers build malls and lease space to retailers or OEMs build factories and lease them to companies to make equipment.
Meanwhile, the most successful hardware vendors are all farming production out to third party manufacturers...
The only clear differentiator then is actually owning a datacenter. And the Total Cost of Ownership of that isn't easy. For someone who doesn't want to be in the datacenter-ownership business, opting in to cloud would make sense. Concerns for privacy and government intervention are a valid concern though.
Traditional product companies very rarely own all their own factories.
Do you honestly think Netflix management is incompetent? They've run the numbers on this countless times. They have way more insight into their spend and operations than a random anonymous internet commentator.
Most companies at even 1k servers will get enormous cost savings from O&O.
A former employer, then in the 600-ish box range, costed out moving to ec2. Our ops actually brought up our internal hadoop cluster on ec2 as well as ran parallel workloads for a week. Amazon at the time offered price deals to attempt to get them as a marquee customer, as well as a bunch of engineering time. Even so, ec2 cost 3-4x as much for 1/3 the throughput. ec2's prices have since fallen, but so has the price of bare hardware.
People also overestimate the cost of hands-on; this company contracted for 4 hours / week with a local person. If you get serious about puppet/chef you can successfully run lots of servers in a remote datacenter.
so, you were saying?
Or do you think facebook,google,aol,tripadvisor and pretty much anyone beside HotNewStartup are incompetent?
The video is the most important part of their product, so keeping it at your ISP or a local exchange not only lowers latency, but most likely also gives better throughput since it only has to travel through your ISP's network or local exchange links without having to go through the rest of the internet backbone.
Latency ( 30ms vs 300ms) doesn't matter much in video playback, only throughput. Once you hit play, it isn't bi-directional.
But you're just nitpicking. This IS their core business, and they realized its "too core" to give to someone else.
So yes, you do "outgrow the cloud", and well, part of "outgrowing" can be cost structure. Gitlab offers a free service, their main competitor, github, runs their own datacenter. Why do you think that is?
Put another way, if there was a cloud DC in each major city and Netflix hadn't already started using their own edge nodes, then there's nothing stopping them from spinning them up in each cloud DC as opposed to their own hardware. There's nothing inherently special about their setup except location.
Nothing about this situation says that Netflix "realized its "too core" to give to someone else." I don't quite understand how you're making that leap. Instead, they had a very specific requirement (that most companies don't have, mind you) that current cloud hosts can't provide. Nothing else.
Besides, here is a list of companies using AWS:
You think Google and Microsoft is incompetent when they offer cloud services to their customers? There are a lot of companies that do know want to own any datacenter related infrastructure because it is irrelevant to their business and there is no on-board expertise. While there are companies that can afford to hire staff for running large DC operations. I thought this is pretty obvious to everybody.
> Or do you think facebook,google,aol,tripadvisor and pretty much anyone beside HotNewStartup are incompetent?
Far from it, and that's precisely my point. At Facebook scale, it certainly makes sense to run your own data centers. At HotNewStartup, it obviously doesn't. Competence is finding the solution which works best for your scale and organization, not following some generic rule.
The cloud offers convenience that all the `original internet companies` wish they had when they were starting out. There may be value in switching when you hit a certain scale, companies like Netflix will have already done cost-benefit analysis to that degree. But it's a massive undertaking that introduces substantial risk for long periods of time. Many companies are just not in a position to do that.
Their original proposal on bare metal was for 64 servers. They need an ops team regardless of whether they control the hardware or not.
Really, what software dev hasn't put together a desktop PC and plugged in some Ethernet jacks? That's all you need to Colocate.
Your response echos what cloud providers want everyone to think. Hardware is too hard for us, let's pay someone to do it and make 50% profit margins on us
Plugging in 150 machines is significantly more challenging. Providing disaster response, backups, stable power, and VLANs to secure services, quickly ratchets up the cost. And nobody wants to pay one engineer per machine to babysit one machine, x 150. That wouldn't even be a good idea, because engineers get bored and start mucking with stuff that needs to be left alone.
Data center administration is a first class professional grouping. Software engineers who don't know that are a real hazard to any business.
When you're a business, you have to pay people to handle hardware no matter what. The question is not whether you need to pay someone to administer the hardware -- the question is who you are going to pay. It can be an employee, contractor, or third-party service company like AWS or remote hands at a colo.
I've seen multiple companies move to cloud merely because they were annoyed at having to go down to the rack to deal with hardware, often making oblique justifications about difficulty when they really meant that they don't like driving down to the colocation center.
Those companies have spent millions more paying AWS than they would've if they just hired a couple of hardware jockeys.
The argument basically becomes a question of whether you know how to hire someone who knows how to deal with hardware. If you don't, it's better to go with Amazon even though they're going to rake you over the coals cost-wise. If you do, then practically speaking, the differences should be small. Your hardware people will automatically fix the hardware issues just like Amazon's hardware people.
The bulk of the work of running servers is at the OS admin level, which is still the customer's responsibility with cloud servers. These arguments about difficulty of administration would work for something like Heroku, but they don't work for Amazon.
I'm in agreement, after the hardware deploy, you're pretty much left with what you would have in a cloud environment: a bunch of VMs (or in this case OS instances) that need care and feeding, and only rarely, much rarely than many would think, do you have to replace a disk, or recable because requirements changed.
Till you're the guy in your 20 people startup who has to do it and crawl around the racks with 100kg of equipment to assemble, cable and power up everything.
It's absolutely hard to run your own hardware. There's a reason that companies which run their own hardware have entire teams devoted to it.
You seem to be thinking like an engineer, not a business person. For most businesses, cloud providers offer a substantial cost savings.
The echo chamber needs a devil's advocate and sometimes the resulting discussion gets interesting enough that I question my original beliefs.
In this case I believe owning my hardware is cheaper for me because I already do. I might be a rarity though, or just old fashioned and wrong
There are some scenarios where it makes sense to run bare metal, but they're few and far between. If you've evaluated the (capital, operational, and human) costs of cloud vs. bare metal and bare metal came out on top, that's fine. But don't assume that other mature businesses haven't also done so.
For companies with AWS bills less than $10k/m, it'd be ludicrous to even consider bare metal.
I've been experimenting with "cloud over" where I only use cloud servers for spare capacity. With docker it's getting easier. My baseline CoLo has the same performance as a super high end AWS instance and costs a third as much after my fixed investment.
I still have the cloud if my box goes down or takes a big traffic hit, but I'm not paying for a bunch of cloud servers all the time either.
It's ludicrous that you suggest that anyone who is not willing to dive head-first into 100% cloud everything should be fired. The cloud has a place, but there's no reason that cloud should be the only option, as you appear to suggest.
If you're spending millions on infrastructure, you should absolutely do a rigorous analysis of whether the cloud is better or worse.
I've seen way too many companies with full-time sysadmins maintaining only a few dozen servers. The significant cost savings come from eliminating those positions.
If you have a bunch of idle sysadmins because you thought you'd have to replace a system's drive every week and hired 4 extra hardware-only sysadmins, you can probably fire them without moving to the cloud and save even more money, right?
Significant savings is not something you get from the cloud unless you were massively over-provisioned with your hardware. In nearly every case I've seen, moving an equivalent workload to the cloud has resulted in more costs for the same size workload. But the company still made the decision to do it because it was less stuff to worry about and it made expanding that much quicker.
You pay a premium for the flexibility of the cloud.
Those "old-school sysadmins..." get hired by large cloud companies. Amazon has plenty. The cloud is just somebody else's computers.
Yes that's a loaded statement but so is comparing "the cloud" to "a desktop".
I've had degraded instances screw my customers websites many many times in AWS. On our VMware cluster I never have any such issues because I make sure not to touch anything running production sites
Except that most cloud providers offer replication and load balancing, integrated directly into their other products already. They spent a lot of time making sure that it works so you don't have to.
Yes, shit happens. But shit happens everywhere, and it is far more likely to happen when you run your own bare metal if you don't pump sufficient time and resources into it, than at the company that has hired thousands of engineers and ops people specifically to provide these services.
They have stuff that will allow you to make your stuff redundant if you know how to architect, configure, and deploy it, but you don't get these benefits just by becoming an AWS customer. You have to know how to set up an ELB just the same as you have to know how to set up haproxy on bare metal.
Is it easier to configure an ELB than a haproxy instance? Probably, but it's not that much harder, and configuration is mostly a one-time cost. Pay someone on staff to spend a couple of hours testing and configuring haproxy or pay 3x more than you should every month to rent VMs on Amazon? Which one is cheaper over "the long term"?
All you're doing is renting some space in a server. The configuration is still mostly up to the user. Hardware requires maintenance, yes, but people are really hamming up both the frequency and difficulty of hardware maintenance.
Have you ever used EC2 with traditional apps hoisted onto the cloud? Just because you can click a button for ELB doesn't mean anything when the system wasn't horizontally scalable to begin with. EC2 is just VMs, it doesn't provide any high availability at the VM level.
With local solutions (e.g. vmware/kvm/openstack), you can live migrate these annoying (but critical) pets around as you phase out hardware with very limited traffic interruption. With EC2, the thing eventually just shits the bed and you get a notice saying the instance will be retired even though it's already in a worthless state.
Glad to hear the vmware cluster is working out for you. Anecdotal but I've seen people having issues without touching vmware too. One personal experience doesn't diminish the cloud. Maybe different business needs? One sysadmin maintaining customer websites vs multi-person team managing a web app?
I'm not against running your own servers. Bare metal is fun, interesting, and possible. I'm just against blanket statements that "it's bare metal or nothing!".
> The cloud as a money saving venture is and always has been a damn lie.
I personally have conducted migration savings assessments for companies going from data centers (owned/lease hardware) to cloud services. I can tell you with 100% certainty that the savings are there and have seen financial proof of the savings.
It's either that, or I'm a liar.
To go to Amazon EC2 let's see... 64 GB of RAM for $0.862 should do the job there, but just for that I'm at $620/mo. Then let's add bandwidth, 10TB out at $0.01/GB = $100/mo. Now for disk I have to use EBS which runs (let's say st1 will do the job) 0.045/GB/mo*3TB = $135/mo. And that's before any IO.
So to move from bare metal to amazon I go from $1.5k upfront + $90/mo to a total of $855/mo on Amazon.
I can buy a new server every few months for the same cost of hosting that at Amazon - unless I'm severely missing something. Maybe it's worth if it you're a very small installation or one with very unpredictable traffic? But I just don't see it.
1. Reserved instances. If you've got a running business and are expecting to be around for 1-3 years, you get 40-60% discount on that 4xlarge.
2. Do you want to move the service exactly as it is? Maybe you don't need the large ebs? Maybe you can rewrite the storage to S3 instead which is much cheaper? Do you have heavy, sporadic tasks that you can move out of your main system and into lambda+queue and use a smaller instance?
3. How much do you spend on people to monitor the hardware, source and replace the disks, do firmware updates, etc. ? How much on external services to monitor your box which could be replaced with integrated AWS solutions at free tier?
Basically what I'm trying to say is: if you just lift your current system and move it to AWS, you're ignoring lots of opportunities. You need to consider much more than a 1:1 hardware requirements migration.
It's really amazing that, when faced with clear evidence of the expense involved in moving to AWS, you suggest that he prepay for a year of service up-front and redesign his application just so Amazon's bill doesn't look so egregious anymore.
My experience aligns with his. We moved from racks with 20-something boxes to EC2 with 100+ instances. Our monthly bill was 80% of the cost of the hardware in our data center.
How did we solve this problem? Why, move to Docker and Kubernetes of course! Over a year of manpower has been devoted to that task. What kind of savage would ever return to bare metal in this enlightened age of expending millions of manhours redoing stuff that was already working perfectly well?
If you want to autoscale, autoscale on the cloud and keep your primary nodes on bare metal. There's no need to start forking over millions of extra dollars to cloud providers to host all of your infrastructure.
Cloud has some unique benefits, but it should be used for those unique benefits only. There's no reason everything has to be moved there.
I'm not advocating to rewrite just to give Amazon money - you're creating straw man arguments here.
I'm just saying that if we're comparing different ways of hosting, we shouldn't pretend they work the same and have the same tradeoffs. Maybe one server approach is optimal. Maybe the cost would be smaller if the service would be split into various components. That's all part of a proper comparison. Including the cost of getting from one arch to the other.
> How badly will moving to cloud storage via S3 affect his performance v. having the files on a local disk?
Depends what they do with the data and how the users interact with it. Maybe there's no data that could be migrated (all records need to be available in memory for the web app), maybe it would be slower but have a trade-off of using smaller instance, maybe it would improve the performance considerably because file-like blobs are not stored in the database anymore and users can get them quicker via local cloudfront caches. There's no generic answer!
As for the outage... S3 was down for a few hours. And for many people it happened when they were sleeping. If your single server goes down in a data centre - what's your cost and time of recovery? S3 going down once a year still gives many companies better availability than they could ever achieve on their own.
If the need to reassess the architecture doesn't convince you this way, think about migrating from AWS to own hardware. If you were using S3, sqs, lambda and other services, are you going to plan for standing up highly available replacements of them on separate physical hosts? (Omg, we need so much hardware!) Or will you consider if it can be all replaced with just redis and cron if you have relatively little data?
You could make the same argument going the other way though - why should GitLab spend money recruiting/hiring people, leasing space, setting up monitoring, etc, if their solution today works? Why should anyone re-architect their system to give $COMPANY money? When your bare metal's RAID controller craps out and you have to order another, will his customers be understanding of his decision to move to bare metal?
It doesn't makes sense to factor in fixed costs of such a migration.
Well, that reason would be, at a minimum, a halving of their hosting expenses.
It also doesn't necessarily take much refactoring to move either way. Even if you're heavily dependent on cloud storage, etc., you can access that from an external application server.
The person I replied to was suggesting that the parent refactor in order to make AWS costs less egregious without articulating any particular reason that the original commenter should do so.
> When your bare metal's RAID controller craps out and you have to order another, will his customers be understanding of his decision to move to bare metal?
His customers won't need to know, assuming he has a standby that can take over. Even if he doesn't, he can rush down and install one and move the disks over. With an AWS outage, you can't do anything but say "I hope Amazon fixes it soon". The bare metal equivalent is a power or connectivity loss at the DC, which is much rarer than AWS outages.
So, pay for the privilege of being vendor-locked to Amazon. What a wonderful idea.
(There are multiple services providing S3 interfaces BTW)
Also, most businesses don't have demand that steady. Nor is it advisable in a cloud architecture to rely on a single large server.
Most of it does need to remain online at all times including several large databases. Just running my logstash server alone eats 8 GB of RAM and 200+ GB of disk.
In bandwidth and storage alone I'm past the cost of the server - so even if I could split this up and keep it mostly offline it still wouldn't be close to worth it unless I'm missing something more significant?
> You're not including the time costs of operating that server.
Do you mean swapping the drives about once every 3 years with free DC remote hands? Operational costs there are almost nothing - maybe 10 hours a year of my time. Surely much less than it would take to get everything running on a cloud architecture. And that cloud architecture would still require some level of maintenance and monitoring I'm sure.
Nope, you pay more for a single large machine than multiple small ones.
If your service is truly so stable that you only need to spend 10 hours a year maintaining it (and demand isn't growing), then maybe the cloud isn't right for you. But that's not true for most companies.
As for personnel, if I had to, I'd hire mostly devops people or pay freelancers for jobs as needed until hitting a limit where you have to dedicate a large portion of someone's time to it and then repeat the math - I fully suspect that this would mean sticking with bare metal. I suspect a 2 or 3 person group dedicating their time to managing only hardware and basic infrastructure for something like Kubernetes could probably handle hundreds of servers without issue. Even if that averages out to an additional cost of $1000 per server per year it'd only be approximately a $80 per server month, nothing like the increase of going to EC2. And renting entire racks gets cheaper than the individual colocation I'm paying for at the moment. It'd surely be interesting
I have no idea where people came from when they moved to AWS/etc but I always have (and still do) picked up the phone and got someone on the line who usually fixes whatever the issue is with me on the phone.
As far as demand spikes - even smaller hosting companies will have hardware they can spin up pretty quick.
So ~$900/month for bandwidth alone.
The disk and CPU costs for me are worth it, much better to pay $600/month and let that be someone else's headache. But the bandwidth makes this totally a nonstarter.
Out of curiosity, what are you moving?
I'm also not sure how these smaller datacenters somehow charge so much less for bandwidth.
On paper, the cost differences between renting/colocating traditional dedicated servers and using cloud service with similar performance/capacity is huge, so it's pretty unusual to actually save money by migrating to cloud. I'd love to hear where the savings actually come from.
At a small scale, you're paying the cloud less than 2 or 3 competent admins with datacenter and networking experience cost for salary. In that situation, you're saving money, because you're saving the hardware ops team. You're paying more dollars per iop, core and GB of ram, but the alternate method of obtaining these resources has more surrounding costs than "ze cloud".
On the other hand, once you're shoveling several hundred kilo-dollars per month to a cloud provider, it makes sense to throw a million or 10 at dell and hire those operators, because it will save money within a year or two. This only makes sense if you need the resources, but if you need those resources, it helps being more efficient.
But if the problem is in the software stack, one would still need competent sysadmins/system engineers with skills to diagnose and resolve the issue. When (just a random example) oom-killer wreaks havoc and free(1) insists there's more than half of physical memory still available, it doesn't matter whenever one's in the cloud or not.
I believe, skilled system engineers are still a requirement for any large project, be it in cloud or not.
And that's why cloud will win every single time, forever, on all metrics.
Because the cloud requires less engineers to achieve the same work, as it takes care of the low level hardware work.
That used to work well before the "cloud" era, and still does.
If you use some VM-based cloud infrastructure you will still need a bunch of good system/linux admins to make your application work on the linux in the cloud. Without these, you're toast.
Bare metal requires you to have both good system and linux admins, and on top of that, good hardware admins, networks and datacenter hands. These are different skill sets.
So again: If you do the bare metal thing right at the scale it pays for itself, you'll probably end up paying both the skillset for the cloud deployment, and add the hardware skillset on top.
I believe, usually, "bare metal" hosting means you order the hardware, get it installed, but networking/cooling/power supply/etc are done by the datacenter people, not your own staff (your own staff may be not even permitted to enter the server room). No less-common specialized hardware to deal with, either.
System engineers must known OS internals well. If they do, it's unlikely they can debug, say, a kernel memory leak (which can be a thing in a VM), but not a lockup in a network card driver's interrupt handler (which is close to impossible in a cloud VM, but I saw this on a bare metal). Maybe I'm wrong, but that would be, like, too specialized and just weird sort of specialization. Am I wrong?
My attempted joke wouldn't make sense outside of this context
When you are in business, saving money is not always the most important thing. In reality, you usually spend 95% of your revenue. The question is not "How much money can we save", but "What investment will bring the greatest return".
It's a bit like buying the building that you are working in rather than renting it. Or hiring full time cleaning staff rather than hiring a firm to take care of it. The second choice is more expensive, but requires less capital/time up front. If you can use that money and time to better effect then it is a no brainer. As companies get bigger, they have less opportunities to make large returns on their money/time, so it often makes sense to start focusing on those smaller returns.
As an aside, this is also why you shouldn't go to your CTO and say, "I can save $X by refactoring/rewriting/testing our code". The CTO can usually always find things that appear to be better ROI opportunities and so you get into a situation where refactoring/rewriting/testing becomes forbidden. Rather you should always attempt to use appropriate techniques to maximise throughput in development and to keep your stake holders happy.
What are some examples of companies that started in the cloud then switched to their own metal? Surely there are some blog posts about costs before & after?
Exactly and most of the companies are not original internet companies, and they use the cloud.