I think the cloud is great; however, most people go way overboard for what they need. You don't need 15 lambdas, 4 micro-services, dynamoDB, API Gateway, etc. to build your internal app that gets 10 TPS. Most engineers over build things because it is how you show "complexity" to get promoted. When in reality a monolith running on some bare metal will work just fine.
I agree with others that the cloud vendors make it complex just to setup a simple service and push you to have these complex architectures. While they are beneficial for availability reasons, I question how often it is really needed.
I remember running apps on shared servers that were far weaker than any micro instance you see now. Many services were quick side projects by one or two employees. They didn’t have the same quality as pro teams of 250k / year employees but the business moved forward and rarely got pwnd by some internet exposed endpoint running unpatched Django.
> You don't need 15 lambdas, 4 micro-services, dynamoDB, API Gateway, etc. to build your internal app that gets 10 TPS
I keep trying and failing to make this argument at work. The majority of the team are primarily PHP developers and if you want their best and quickest work they can write you a perfectly good internal service in PHP.
But someone several levels above me drank the kool-aid and now everything takes four times as long and turns out crappy because half the team is spinning wheels learning new proprietary cloud tech and reimplementing the base-level tooling we already had. I am yet to hear any explanation why this cost is worth it beyond vague mutterings of “scalability” and “enterprise”.
We had a “monolithic” on-prem application that was forced to rewrite for cloud.
The original system was blazingly fast — it was actually a set of composable TCP workers and coordinators, but repeatedly called dirty names like monolith, legacy, synchronous by ambitious managers trying to demonstrate the need to migrate. Proceed to bring on legions of contractors and offshore teams to move everything to HTTP and AMQP, fully coupled to the cloud vendor’s mechanisms for these with all of the goodies.
Today they are paying a fortune for a slower system, full vendor lock-in, real customers dismayed and prospective customers aghast at the prices.
Recently they came back to the original engineering teams to “make it a real-time system” and panicked at the realization that this capability was unceremoniously sacrificed in the name of microservices, the cloud for the sake of hot-shot aspiring upper management to make their name in “modernizing” the tech. Yikes
You gotta remember that the optimal career choices of an engineering manager in medium to large enterprise are technically highly destructive to the health of the company and productivity of the people they're managing.
They need to balloon the amount of people they manage and be involved in as many projects as possible for bullet points on their CV when they're applying for the next position.
The cloud is absurdly overpriced. The margins are huge for cloud companies, especially on bandwidth. The gains brought by Moore’s law are pocketed by could providers and barely passed on because once you go “cloud native” they have fairly strong lock in. Even if you can move your work load doing so is painful and usually requires downtime.
It’s also designed to encourage and then monetize a certain style of complex, over-architected, high resource load development: microservices, use of single-cloud-proprietary services, tons of loosely coupled systems, etc. Cloud native is the J2EE of the present era.
This is an anti pattern that keeps happening over and over in our industry. It’s partly a result of the middlebrow meme— that mid developers love complexity— and partly something that is pushed because someone (consultants, cloud providers) are profiting.
Edit: I don’t mean to blast mid developers too much. One must often journey through the middlebrow meme to make it to the hooded dude on the right. The meme describes a learning trajectory. But we should try not to let the complexity explosion of mid developers become industry norm.
>This is an anti pattern that keeps happening over and over in our industry. It’s partly a result of the middlebrow meme— that mid developers love complexity— and partly something that is pushed because someone (consultants, cloud providers) are profiting.
Pretty un-fixable without better stewardship over the software industry. That won't come for decades, so you may as well join the dark side and make the money and practice the way you want with hobbies or start your own company to enforce it.
I guess the stewardship can happen on a team level, but someone who's being restrained can always go find another job where they'll build the complexity they desire that ultimately gets maintained by someone else.
The gains aren't even "barely passed on". When new generation chips are released, they'll oftentimes release a new instance type at roughly the same price, then re-allocate the older generation chips to power internal services which don't promise a chip family. That internal service might be e.g. S3, so who cares, but it might also be Fargate or Lambda, where you're paying per second of wall time and thus slower chips actively cost you more money.
Tbh, the flexibility can make up for it if you know how to use it right. Sure, the servers might cost 3x as much as what you would pay at a VPS, but you can spin them up and down by the minute.
For one of the projects where the demand appeared during working hours in my timezone, I did the math, and found this to be the case.
Do you count VPSes from Vultr or the like as "the cloud" for this purpose? And, incidentally, are there any VPS providers that you would recommend over Vultr when one doesn't need a whole dedicated machine in a given location?
I would add that a huge selling point is the on-demand aspect. You can get VMs/containers/functions running in a few seconds. This is great for experiments, bursting, etc but if you are using compute capacity for a year, it should be understood that you should rent a box instead.
Like, you wouldn't rent a dozen ZipCars every day. If your business required a fleet of cars, you would quickly look for a better arrangement than pay-by-the-minute "convenient" app services.
That's why people here love HetznerCloud. You can create a VM in seconds and be absolutely sure that you will pay no more than €4/month for it, something that AWS can never guarantee.
Exactly. You can run almost for free on Google if you use things like cloudrun and you don't have a lot of traffic. We developed against that for over a year without ever hitting their freemium threshold. This was just a simple development server of course. We transitioned that to a bit more flexible traditional setup with some vms and a load balancer. You can just start a vm with a docker container in Google cloud so it's stupidly easy to deploy dockerized software and do things like rolling updates. With AWS you need things like ECS and some other cruft. Also not the end of the world. But more work.
The mistake most companies and developers make is mapping their organization structure to their deployment architecture. And then over provisioning everything. Kubernetes is the goto solution for companies like that. We spend less on our cloud bills than it costs to spin up even a simple Kubernetes cluster that does basically nothing. It doesn't solve a problem we have because we have a monolith. All I need is two cheap vms and a loadbalancer. If it gets busy, I'll add vms.
There's some other stuff we need, of course. The most expensive thing is actually managed search and databases clusters. They are nice but also pricey.
But from a cost perspective the whole point for me is actually minimizing my time investment. A managed database means I don't need to obsess about uptime, backups, and other things you would normally pay some team to obsess about 24x7. I don't have that and I can actually go on vacation and it's fine.
Of course all this stuff is way overpriced. You get the equivalent of a raspberry pi worth of compute for a monthly price that would allow you to buy several of those. Cloud providers actually put multiple customers on their hardware for the cheaper vms. So, these things are basically paying for their own cost pretty much instantly. When you use a managed database, it runs on shared infrastructure that minimizes resource usage. So why is that more expensive than a non managed one that has more idle resource use? Convenience. It doesn't actually cost them more to manage it. But it's more convenient for you. So they just charge you more for less.
Cloud pricing bears no relationship to actual cost. Which is weird. There should be way more competition on price. But it seems that there just isn't. Of course, people just mindlessly buy AWS because they heard it's good. That's why Amazon is so rich. The retail business is a side hustle for them at this point. And you can get some amazing deals from smaller providers. Hetzner gets mentioned a lot and they are pretty aggressive with their pricing. But it's less convenient.
I've been running Pirsch [0] on Hetzner Cloud for 3 1/2 years now on a self-hosted HashiCorp Nomad cluster. It has been super stable and very cost-effective. The Hetzner VMs are really cheap and a lot more capable at the same time. You can find everything on our blog article [1].
In front of it are two Caddy load balancers, also running on VMs (Hetzner offers load balancers, but we wanted to support custom domains [2]).
The databases are running on root servers though, to get the maximum out of them.
When we started, we used Google Cloud. I think we would be paying at least $2,500/month for the same setup, compared to ~$400 now. And I have happily managed this all by myself. I think a proper sysadmin for a larger corp is worth the cost!
We back up simple dumps to a Hetzner storage box (5€/month for 1 TB), which is fine for the moment. I download and restore the dumps locally regularly.
In the future, we will need a more robust and scalable solution of course.
So I decided to try it to see it with my own eyes (EKS takes often half an hour or so...) and lo and behold, they use k3s! I mean, there is nothing wrong with that, but it's not mentioned anywhere on their product page so that was a bit misleading.
In any case they have a verification process that I haven't managed to pass. I was surprised because I used the same credit card for Gogle and AWS, but for some reason they closed my account.
VPS are good, but the fact that you have to enable 5 different services to do stuff on most clouds and deal with semi-opaque billing is pretty annoying as someone who lived through the colo and dedicated hosting days.
At least for us, the promise was not a lower bill, but faster time to market. The requests to on-prem engineers are just _not a thing_, which is much faster for our developers.
That that sounds like you are solving your internal organisational dysfunctions by outsourcing the function rather than trying to fix the organisation.
I understand your point of view, but the BigOrgs are strange. They say they want to move fast and succeed, but they are also magnets for people who would much rather accumulate money/power through politics than work. This often sets up perverse incentives to do the opposite of what would make sense.
For instance, the IT department manager claims to need more headcount and money to fund it, because they believe that having more budget and employees shows how important their function is. Engineering requests a new VM, and the request sits and sits and sits, so long that you start believing that they are overloaded. Engineering complains that it is taking too long to provision machines, and IT (for the 100th time) says they need more headcount to keep up with internal demand. To fix the problem, management grants one or two hires, things get marginally better for a while, until the cycle repeats.
In this scenario, Engineering is trying to get their product completed, and looks bad because the schedule is slipping. If you are the Engineering manager, do you care more about fixing the organization, or sidestepping the issue by swiping a credit card?
Fixing things gets political fast, spending money less so.
Not managing your own physical infrastructure is not a dysfunction; unless you have a very specific hardware requirement, or are very large, on-prem is just never going to be cost effective.
Where it is cost-effective, it can still be incredibly strategically limiting.
Yep. The business case for cloud solutions is TTM, availability and capacity. The argument that on prem never delivered on its promise to be cheaper is a lie by omission. It CAN be cheaper, but that was never the main reason.
That's my main understanding as well. You're effectively outsourcing the hardware and a lot of software infrastructure and it will vary a lot based on what SaaS offerings you're using with the cloud provider.
Personally, I tend to push for PostgreSQL primarily with near self-service containerized services for the applications and mostly will just leverage the cloud's queues and blob/s3 storage, which are easy enough to replace in a software stack. I'm not big on lock-in.
It's about TCO and time to market. Depending on what you're building, it's all YMMV. This article just quotes some individual costs
As somebody who worked at a mid-tier company trying to run databases, I can attest that RDS was a godsend. Trying to hire both a DBA and an Ops team that knew how to write the chef cookbooks for a proper multi-node cluster postgres was a nightmare. Like, we never succeeded, and something that's ootb with RDS
You must not forget that also, many (most?) companies that run things themselves do not do it right. Like, with proper off-site backups that you're regularly testing and know you have options to easily spin up replicas or restore point-in-time backups
Jeff Atwood's been saying this from the initial SO podcasts from 2008. If you have the right people who are motivated, and provide the right equipment and resources, you've always had the opportunity to have lower TCO doing it yourself
I have since moved on to a small top-tier company and still prefer to "outsource" my DBA work by way of using Aurora. Yes Aurora is more expensive. No, I don't have the mental or monetary budget to hire up a proper ops team. I know my limits
The cloud, while expensive, gives flexibility. We had x machines running at a space where you rack machines.
First it took 3 weeks to decide what we needed, then 3 weeks to get a quote for new machines, then 3 weeks of negotiating and penny pinching because this was a 3 year lease. Then it took 3 months for them to get the machines in there.
In the cloud it's 3 clicks and you have the servers, and even better, scale on demand once you have good metrics in place.
Flexibility has a cost. But not being able to do anything also has a cost.
Oh wow, imagine that, a forced meme in tech pushed primarily by marketers, this is totally not a trend at all. /sarcasm
this is like when UX designers mess with a GUI just because they need to make themselves look useful on occasion.
Like-for-like, those costs go down over time and have since we started building datacenters. Electric costs roughly track inflation, but the price of a TB of storage or 10GB switch port or a 24 core CPU goes down over time. That's why it's hard to align the price of cloud services, the underlying fundamentals are understood and they generally track downward, yet somehow AWS and Azure and GC etc all find ways to increase their pricing.
There's still the middle ground, either virtual or real machines hosted somewhere else. Pay fixed monthly amount, get a machine somewhere, don't have to deal with heating, cooling, power, internet, anything, but still noone will charge you anything "per database lookup", per IO cycles, etc.
Instead of having to hire a full sysadmin and buying infrastructure, you just click a server, enable snapshoting and you are already 10x better of of what you had before.
The biggest issue with these 'cloud' discussions and the pricing is simply solved: EITHER you have a good team who understands it and can and should determine if you are better of onprem or on cloud, or you do not have this team and you are not wasting money you are just paying for a lot better system which you would otherwise never had.
I have seen plenty of tremendes shitty setups in small companies from people who should know better. Databases reachable on the internet, slow ticket systems for getting hardware, costly upgrade prices (and partially internal cost centers were an server upgrade costs a few k).
You use too many slow lambdas? your Database costs tons of money? You can't control your micro services anymore? And you think your company would have been able to setup a better thing on prem without the help of cloud? Never
> and its not like a dev can't do anything system related
I'm glad to read this! Unfortunately the industry has become so fragmented that a very significant percentage of younger developers no longer have any knowledge nor any interest in understanding how systems work. There is this idea that a developer only writes code and has no need to understand threads or processes or userspace vs. kernel or networking or anything at all other than the language syntax. It is very sad.
The place I work at now wouldn’t exist without the cloud. We simply don’t have the money or manpower at this stage to run everything in house relative to the growth.
That being said, a design philosophy I’ve imposed is KISS with largely off the shelf OSS solutions, that way we have the ability to move the software elsewhere in the future (I’d love to run in house if we get big enough). Of course, nothing is that simple but it’s much harder if not effectively impossible with something built significantly out of vendor-specific libraries and platforms. I don’t mind managed solutions as long they are completely swappable.
Unfortunately, the people before me used DynamoDB as our persistence layer, but I digress.
While I think civo are doing some pretty novel things to allow kubernetes to run cheaply for small clusters, it’s a mistake to take a vendor’s “report” at face value. It’s a press release.
I've always found the best cloud path to stick with basic VM/VPS offerings, blob storage, DNS, CDN and IdP.
Getting into things like functions as a service has always felt a little too hot for me. I don't mind integrating with cloud services though (e.g. S3-compatible APIs).
Hybrid seems to be most compelling if you can keep the monster in the box. Sign up for something like Azure or AWS to get at their IdP, DNS and CDN but keep all your actual machines in a cheaper provider (hetzner, et. al.).
My rule is to use services which could easily be replaced. For example, on AWS using RDS is a gimme because you can always spin up your own database with the same engine but not having to spend months building the equivalent battle-tested configuration is a pure win. ECS Fargate is great because you can run containers for at one order of magnitude lower ops investment than Kubernetes but if you had to leave, it’s vanilla containers on Linux. Something like DynamoDB, though, I’d want to think about what a migration plan would look like since databases are core to your application but the semantics are not as interchangeable as SQL databases are.
The 'cloud' always was - at its core - a mechanism to take your sovereignty away. Like the social media or IM walled gardens that you like. And like a lot of things that came up in the last ~15 years. Subscription models as well...
All that will not fail, because there are strong forces that explicitly want this to happen.
But it's always nice to hear that there are at least problems...
We need to first differentiation between "public compute cloud" (AWS, GCP, Azure) and "cloud services" (GMail, Dropbox, iCloud, etc). The fine article is concerned with the former exclusively, and I think mixing in the latter dilutes the point it's trying to make.
So unless your personal homelab needs to be hyperscalar, you don't need a public cloud; your needs will be served perfectly by a $5/mo VPS/PaaS, and/or a LACK rack.
Do companies need sovereignty? No, companies need to make money, that's their primary and main reason to exist. The choice of public cloud vs dedicated servers vs colocation vs on-prem vs etc amounts to whatever best fits your operating model. The public cloud was overhyped (like everything in tech), but smart companies chose the right tool to do the job.
Your power, water and gas supply also come from the 'cloud'. If anything, localized computing devices (Phones + PCs) are significantly more popular than localized power generation/water treatment (garage diesel generator/wells).
Historically, the main economic impetus for centralized government, is to coordinate the construction of irrigation infrastructure. The more dependent a society is on one big river (Ancient China/Egypt), the more centralized it becomes.
How about the cost of needing to hire more admins to handle on-premises machines, and their time spent trying to save data from faulty end-user machines instead of just replacing them and thinking twice about it?
Either you pay admins to manage your on-premise systems, or you both pay admins to operate your cloud systems and also pay cloud providers for their admins to provide the cloud systems your admins manage.
I run all my shit on a VPS (which could be called a cloud) or a dedicated rented server but that is so easy to setup and I can run all projects on the same server. Easy, simple and if I need to scale I just rent a bigger server.
Scaling vertically is easy, scaling horizontally is hard. Most people never need to scale horizontally but does so anyway because they think they do.
You also get like 10x the perf for the same money. Using SQLite makes it easy to have backups and even time-specific testing databases.
Because most companies/startups have this in-built assumption that they will eventually grow like wildfire. Obviously this is because they are selling to investors who want this to happen so they can cash out on their unicorn. So they just build it in the cloud in the first place with that in mind.
Now in some miniscule amount of cases this is true and probably did help some people who's business 100x'd overnight, but in the vast majority of cases your business just will never get to the point it needs to be "cloud scale" in the first place. Nevermind accidentally shooting yourself in the foot with a recursive lambda here and there in certain instances or a misconfuguration causing a huge bill.
Edit: Another is because lots of companies who do actually end up succeeding negotiate a shit-load of credits with cloud providers so they can basically grow their business for free for a while. That is until those credits run out and they get hit with the actual costs.
For most tech startups, when the growth period hits, its vital there must be NO DELAY to scale. Miss the timing and its permanently gone. If word of mouth leads to a sudden surge of orders, you must be able to supply the good, or you get very angry customers who walk straight into the competitors door.
Most of the value generated by startups is highly concentrated in the few that succeed. So naturally the industry should optimize itself to go big or go home, not penny pinch. That's also, why they hire expensive engineers rather than offshored developers, because speed matters more than cost.
As for non-tech companies. Their demand is more stable in the long run, but they are not tolerant of outages. Amazon cannot have its servers go down during a big sale, too many physical ongoing costs that gets wasted for every second the central nervous system is down. So the cloud is good for its reliability.
When AWS was first getting big though they solved genuine, really hard problems for a lot of organisations that were large or growing quickly. NVME drives didn't exist, SSDs were expensive and a lot of servers still had spinning SAS drives – A little box with some ram and some NVME drives didn't scale as ridiculously far as they do now.
I do think as computers keep getting faster and smaller the number of use cases that need a 'cloud' shrink very quickly though.
The initial rush for what is now called the public cloud was caused by many factors, the main ones being "you don't need to manage things as we will do it for you" (=you save on operations) and "you can scale up and down how much you want without any commitment" (=you save on recurring costs) with success stories like the NYT resizing tons of images fast without having to rent servers, configure them etc.
Fast forward and what we now have is a terribly complex beast with nets of dependencies that got developed partly by marketing and product teams, partly by demands of larger customers. And it's more or less clear that if you are small, you will be much better off using VPS (that's why Amazon decided to offer Lightsail), and if you are very big, you will save a lot of money moving at least part of your infra away from the public cloud.
But what remains is a large part of the market: medium sized businesses and large organizations that depend on the public cloud for many reasons. But they are not stupid: once a project becomes expensive, someone starts asking questions. And after you've exhausted the path of reserved instances, spot etc., and still burn a lot of money with not-always-stellar performance, you'll find a way of moving these workloads where it makes business sense.
One recurring pattern I have seen at multiple customer sites is that scaling makes the engineers lazy to optimize. One production performance calamity requires the team to add CPU as a quick fix, and from that time on the baseline for the product's requirement has been set to the new number of CPUs.
"Back in the olden days", if your product was slow but the number of CPUs was fixed (or could not be increased instantly), the solution was to go and fix your code.
Basic system level skills are now no longer taught or practiced at the appropriate levels, so teams end up without engineers who actually know how to profile and optimize.
I've lost track of the times I've heard "compute is cheap! engineers are expensive!" Except... that compute cost will live forever. The time it takes someone to debug a bad loop or poor query is at worst a one time cost. Longer term, it may even make other stuff faster in the future.
Looking at some numbers on cloud, yeah I don't believe in that statement anymore. End device compute might be cheap, but cloud services certainly are not.
Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. This wasn't the case a couple of decades ago.
It also means less engineers are needed for most companies.
> Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue.
It's not cheaper, it's just more opaque.
Back when your service was deployed on that 2 CPU box and it was too slow for obvious reasons, you optimized it and then it was good.
Today you just shrug and increase that kubernetes cluster from 16 to 48 nodes and forget about it. Costs a lot but the bill shows up somewhere else, in most groups the engineer doesn't even know what it is.
For large companies, a big reason is to transform capex into opex, and the predictability. Moreover, large organizations tend to favor predictivity over levels, i.e. are ok to increase average if variance is decreased.
This. The beginning of my career on cloud was a POC, where the director shared with me this was a major driving factor (capex > opex), as well as some of the fringe benefits.
I got to see close up that a team of devs ran their whole solution (with a bunch of paying customers and everything) in the cloud, because cloud automation was good enough that they didn't need dedicated ops people.
Now I work for a cloud provider. I can't say that if I was running a business that I'd build it cloud-first instead of OnPrem. Certain use cases, sure. If I didn't need a lot of horsepower, I might build it on a cluster of VM's with some segmentation of duties - not quite microservices, not quite a monolith. Most likely if I was hosting on the cloud, I'd use the provider I work for, just because I know the system and how to get things done and how to talk to support.
I will say though - learning the ins and outs of cloud computing has made for a great career. Challenging, but lucrative.
FTA:
> Microsoft and Google decided not to officially comment on the survey's findings. However, a representative for one of the hyperscalers retorted that the figures seemed cherry-picked and pointed out that, as an example, customers using reserved instances could realize significant savings.
Reserved instances are a thing for sure. There's lots of other ways you can control cloud spend (enterprise agreements, dev/test subscriptions, spot instances, automated shut down / scale down, etc.) - it's enough complexity by itself that big companies hire entire teams of people to just work on tracking, projecting and controlling cloud costs.
The value proposition of "the cloud" is to ease all the "not important at first but you'll need one day" things: logging, alerting, availability, backups, SSO etc which usually requires different know-how from what devs have.
But it has become so complex that instead of an OPS team you now have a Cloud team. With a huge wallet.
Best of both worlds is to setup your own Cloud on multiple VPS which is relatively easy nowadays: HAproxy, Rancher, Kubernetes, , Keycloack, Openwhisk, Gitlab, Harbor, Opentelemtry + Prometheus + Grafana and your devops will feel right at home.
If you can run everything on just a couple servers then the cloud doesn't make sense. There are a lot of companies that have tens of thousands of servers scattered around the world. just tracking them is a headache and you can offload a lot other overhead as well. Remember you are paying for the people full time to be in that computer room either if you do it in the cloud or not, but the cloud allows sharing those costs better when you don't need the computer.
yeah, that wonderful time when you needed a three year contract + support + licenses to get a startup going off ground
sure cloud benefit might not look like a lot if you are an enterprise customer, but for startup, along with open source software, it changed the game and is still an enabler of what you see today in term of supporting startups from bootstrapping all the way to unicorn.
I agree with others that the cloud vendors make it complex just to setup a simple service and push you to have these complex architectures. While they are beneficial for availability reasons, I question how often it is really needed.
reply