> The serverless concept solves this problem, because it requires no DevOps to build and deploy applications at scale
This is wonderfully naive. A great goal, but hopelessly unrealistic. E.g. currently it is not handling data persistence at all, so for starters it's targeting "the easy bit", yet there's hardly any mention of things like logging, or what do to when their Consul cluster decides to start complaining because membership has gotten messed up (ever had a Consul node rejecting most messages as "suspect"? ever had the same node get registered twice? all kinds of fun stuff), how to analyse performance issues, networking problems etc.
It looks like an interesting platform, but anyone thinking this gets them out of understanding ops concerns will have tough times ahead..
> anyone thinking this gets them out of understanding ops
I haven't looked at it specifically, but given how much the blurb focuses on 'you don't need ops, developers can do this, its all automatic and built-in' (thankfully they don't use the term 'magic') makes me think that when said devs inevitably do need ops help to make it work, the required work for said ops will be increased because they can't just say "well the problem is your load balancer is doing X instead of Y because of config Z" - its going to be a shit fight of fighting against the 'automaticness'.
Maybe they want to be the new cPanel (the web hosting product, not the generic nickname for a control panel).
Hey, I look forward to the day I can charge high rates going in to fix underlying problems for this stuff... So far, those of my clients doing cloud setups typically end up spending 2x-3x as much for my services to sort out their systems than those doing dedicated hosting, on top of usually spending more on hosting as well...
There are certainly exceptions that are using cloud technology for the things they are genuinely cost effective for (e.g extremely spiky traffic; bulk processing; or to manage their own dedicated hardware), but the amount of money that's blindly thrown out the window because the people authorising the expenses don't know how to estimate the costs of these things is staggering.
Well, yeah. The whole idea that you can somehow "save" on ops by doing X is just accruing technical debt, moving the point at which you will need ops into the future and also thus making it far more expensive when that time comes. It will be a fire that maybe only expensive outside (ops) consultants can put out. Imagine that: A fire that no one at the startup can put out. That's a lottery I don't want to play. If that doesn't scare people out of this kind of "we don't need ops anymore!" hooey, I feel like they deserve whatever comes their way down the line.
"We rely heavily on Docker and Docker Swarm for containers, orchestration and networking. Other pieces of tech we use are Consul for service discovery and gRPC with Protobuf for the RPC."
Awesome that it's using gRPC under the hood. But, I'm still missing if there's an orchestration service running somewhere that takes care of the load balancing, or if it's distributed?
"When deploying Lever, you allocate a number of servers to it. Lever can then manage resources and auto-scale services running on it - but this is only limited to the servers you have allocated to it."
It would be awesome (and critical to our use case) if it could provision instances, even if it's via a shell command manually entered into a config file. From the stack description quoted above, it sounds like it would do okay if it was coupled with an autoscaling group that does this function for now...
Finally, I love the sticky resource routing, with the use case being data sharding.
Thanks man. To answer your question about load balancing, it is made via the 'leveroshost' agent itself. Each contains a HTTP/2 proxy that routes and loadbalances traffic as needed. Each RPC goes through 2 of these agents: one on the node originating from and one on the node going to. The decision of where to send the request next (load balanced / routed) is based on information from Consul.
Isn't a core tenant of DevOps that your developers are the ones doing the Ops too? I keep hearing things like "Hiring DevOps" or "According to the Stack Overflow survey there is 1 DevOps for every 30 developers.", and that feels like... an Ops role, right?
The day I see most developers doing ops well will be the days pigs fly.
My background is development, yet I've always done ops too, but what I see from most developers is that exceedingly few of them even know the extent of what taking responsibility for ops includes.
E.g. I strongly believe that one of the drivers for a lot of the extremely costly cloud solutions people use is that most developers these days have an acute lack of knowledge of hardware. But to do the ops side right you need to be able to choose cost effective solutions and you need to understand the trade-offs, or you may easily end up costing your business 10x as much in hosting for no good reason.
Thinks like capacity planning, risk management related to the hosting, backup strategies, failover/ha are areas very few developers have much experience with.
I spend a lot of the time I do consulting on mentoring development teams and/or providing advice in those areas, and most places are _very_ far away from having integrated teams that can safely and cost-effectively handle the ops side.
> exceedingly few of them even know the extent of what taking responsibility for ops includes
When you don't know the extent of your ignorance, everything seems trivial. A.k.a. "how hard could it be?".
I have a feeling the more ops-y things I do gets more in demand as organizations move towards the cloud. Measuring performance is not trivial in a shared environment and failure scenarios can get very complex. And every development decision suddenly affects the bottom line when billing gets granular.
Yeah, I keep getting clients come to me because of the complexity of their cloud setups, and they invariably end up paying far more than the guys running stuff on dedicated hosting. Most of them also pay 2x-3x as much for the hosting itself...
I think a lot of these setups grow by accretion because people are able to push things onto them within purchasing mandates they have because of monthly fees that can slowly grow through discrete approvals rather than having to justify large capital expenditures upfront, or even seeking approval for contracts for monthly payments vs. running up per-minuter or per-hour or per-GB charges...
I suspect a lot of the enthusiasm for these services would vanish if more engineering teams actually had to produce cost estimate, budgets and after-the fact accounting of operational costs for the stuff they build...
I think as a startup, though, offering these kind of platforms as open source is ingenious, because you get mindshare, but people soon enough will run into enough pain to want to offload the pain on someone else - either hiring devops/ops people, contracting it out, or coming to your startup begging you to run it for them.
Certainly, it's great for people like me who do consulting part time that people don't know how to do this, as it's far easier to justify my rate when I can ask to see their hosting costs and be able to point out to them that the monthly cost savings they'll get will cover the fees in 3-4 months.
> Isn't a core tenant of DevOps that your developers are the ones doing the Ops too?
Im sure it means that to a lot of developers, and definitely some managers.
The developers see it as "no more ops telling me that I need to use a TLS connection to the database because it's over a WAN", or "no more ops telling me I can't use the 0.0.0.1-pre-pre-alpha version of <insert runtime/framework/daemon> in production".
The managers particularly see it as "we don't need those pesky ops people, so we've just saved ourselves $X per year". They don't tend to factor in the lack of relevant skills by their remaining developers, which tends to result in less efficiency (i.e. taking longer to do shit) and greatly increased risk.
And then the developer says "hey we'll just use AWS because thats what HN all suggest and it's cheap" and you end up paying a heap of extra cash each month because they just run 10 on-demand EC2 instances 24x7 and a host of assorted services to deal with the lack of persistence on EC2 instances, when previously you needed just two servers in failover setup, with capacity to spare.
I've seen what happens when developers "do ops", as I was leaving a company. When you're asked by someone "can you help, I'm trying to install this SSL cert on a server (for a soon to be production system) but I don't really understand how any of this SSL stuff works", there is just one realistic response. Run. Then run some more, hide, change your email and don't answer your phone. Because that shit is about to go down like the titanic.
> And then the developer says "hey we'll just use AWS because thats what HN all suggest and it's cheap"
Never understood this, except from developers who've never actually had to pay for anything out of their own pocket. AWS is nice, and has a place.
And I do hear it from people, a lot. It's cheap compared to buying your own hardware and dedicated staff, yes. But for a lot of the standard webapp stuff you might want to do starting out, it's not as cheap as just getting a dedicated VPS someplace.
I've been independent/self-employed for a long time, and while I use various AWS services (including EC2 for some projects), it's rarely ever cost which is the driving factor when AWS is chosen. But it's because the $ is coming, directly or indirectly, from me.
Running your own stuff becomes more expensive when you want monitoring, Load Balancers, HA, VPN's into your corporate network and constant database backups. And then making it really robust.
Hiring all the staff for that can be expensive.
Problem is there are lot of developers who think setting up a virtual machine putting a db on it, and then a webserver on another without thinking about all the other stuff that can go wrong.
There's a balance to be had, and there's very likely different periods in a business lifecycle where some environments make more sense than others.
There's a pretty big cost to learning how to manage and get value all the AWS stuff, and in the early days of a project - especially true in the 'startup' world - there are probably more areas where $ can be spent to create more value.
I've seen far too many people spin up multiple AWS and tie themselves to expensive AWS setups when a small linode or other VPS (managed or otherwise) would have been more than adequate for the first year.
There's this weird view people that have it's either 100% AWS or hire full time developers and buy a bunch of bare metal. There's a lot of middle ground which can provide a good amount of value while giving you time to ramp up.
The fact that you're referencing tying VPNs to a corporate network means you're already far more advanced than a lot of the smaller orgs I've run in to tying themselves to AWS when far less expensive alternatives exist.
Again, saying this as someone who uses and likes Amazon stuff, just not exclusively.
It can still be an issue. It would still be a disaster if in the first year, the database got corrupt or hacked?
Lone developers might tempted just to setup database without setting up a backup solution and HA. Amazon has that at click of a button.
For a developer/infrastructure guy probably 1/2/3 days if you know what your doing. If you don't know what your doing for your specific brand of database, you might not set it up right which is a big risk which only shows up once it's failed. There's a reason DBA's can get paid a lot, because there's a lot to know.
I think as a startup, you need to take all the shortcuts you can grab. Getting it out, and time to market is very important. Focusing on price at first can cost more than its worth.
It depends on if you want to pay more for having quick setup and having amazons engineers looking after backups and HA or do you want to pay less, spend more time doing it yourself and getting the skills to make sure its setup correctly.
Competent, experienced staff to run the core of your business has a cost. You can replace ops with managed services no more than I can replace devs with cheap, untrained labor in Asia.
I never said replace, just augment. You may need fewer, but you still need people to monitor, deploy and orchestrate these different services. At the end of the day the extra cost is going toward hiring engineers at the hosters side for managed services but benefiting from their scale.
Either way your going to have to pay for skilled operational engineers.
"when previously you needed just two servers in failover setup, with capacity to spare."
I don't know if you were there but I said the same shit to people here on HN in a hosting discussion. Some company had downtime on AWS and/or a bunch of servers. I said something along lines of how hard can it be to buy four servers at $1-2k each, set them up as HA pairs of pairs, send two to a colo, and manage that? And optionally send another pair to a third, geographical area if one wants more uptime? It's not that hard given we've been doing it since friggin' VMS clusters on refrigerator-sized VAXen. Just dirt cheap now for basic solution. OSS tools are even good enough for homebrewing it with little risk now.
Nah, though, let's pay for a bunch of VM's we have to manage anyway that we don't even control on sites with less uptime. (rolls eyes)
"but I don't really understand how any of this SSL stuff works", there is just one realistic response. Run. Then run some more, hide, change your email and don't answer your phone. Because that shit is about to go down like the titanic."
LOL. That's great. Someone with a meme generator needs to make an image out of this: "DevOps: Replaces Admins, Can't Deploy SSL." I can imagine quite a few more in capacity and disaster planning. Now what image would go with that...?
To me this sounds like the developers weren't tested for ops skills before being brought on to a team while everyone knew that developers are required to do ops.
Of course, specialisation helps. But the most pleasant work experience I've had was at companies where there was no distinction between dev and ops.
I worked for the company ostensibly as a developer, but my formal qualifications are in network engineering, and ops (we used to call it infrastructure) is what I did before I jumped to web work.
When none of the developers are particularly experienced with ops, how do they know whats required to do ops?
Im not saying someone *can't& be skilled at dev and ops. I'm saying the 'normal' result I see, is that management (and some developers) assume any developer can do anything an experienced ops can do, and write that fancy javascript widget too!
If library is done well, then you shouldn't have to know much about SSL to be able to use it. Just an abstract idea of it plus how to use the library. SSL from an ops perspective is quite different. That's why it's better to have pro's in each working together on it.
I would want a developer who knows the best practice keeping the ciphersuite up to date and understands stuff like client side certificates if they going to work on any security conscious application.
I'd prefer it but one you mention can be done upstream by library developer with secure defaults. Think NaCl. Client-side certificates might come with best practices or checklists that show proper use with optional data on the way. Neither issue requires understanding SSL in common deployments.
Note that I say this while agreeing with you that I prefer someone who knows what they're doing. Especially as SSL support isn't usually at the usability standard I'm describing. Yet, it's not intrinsic to the problem: just crap UX.
DevOps is one of those terms, kind of like Agile or XP, that means different things to different people. To me, DevOps means Developers and Operations working more closely together, and walking a mile in each other’s shoes (you’ll hear the word empathy thrown around a lot). Developers should support their applications in production (including pager rotation!) and architect, design, and build applications with infrastructure supportability in mind, and Operations needs to engineer infrastructure using better development techniques: test driven development, automation, and with developer ergonomics in mind (providing a standardized, scalable platform and not just isolated snowflake systems).
> Isn't a core tenant of DevOps that your developers are the ones doing the Ops too?
This isn't how it started.
"deliver a seminal talk known as “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” The talk is an energetic presentation in which Allspaw and Hammond basically act out the classic “fingerpointy” conundrum of Dev versus Ops—“It’s not my code, it’s your machines!” (and vice versa)—to a roomful of developers. They make the case that the only sensible way to build, test and deploy workable new software is to make development and operations transparent and integrated.
Read more at http://rewrite.ca.com/us/articles/devops/a-short-history-of-...
Honestly if you go back to the early 2000's Operations and Engineering were often playing the blame game if not at outright war. I can't tell you how many SA's told me to "file a ticket" when asking for basic factual information (not changes).
The idea of a "full stack" engineer is mostly a myth. If your starting out small hiring a generalist who can do it "all" is probably a good thing. The issue is finding folks who can do everything well is HARD. Once you hit a certain scale, you need people dedicated to each JOB (Front end, back end, operations, database, networking). When you get to the level of having a lot of specialists, having someone do "DevOps" to act as a bridge, to speak able to speak to specialists and convey issues can be vital.
I don't think the term is well-defined. At first, it was about liaising between developers and operations. These days it seems to be about infrastructure-as-code like Chef.
To avoid confusion, I assume you mean "tenet" as in principle, not "tenant" as in "occupying someone else's property", which could get confusing when talking about containers, cloud computing, and using some provider's resources.
Originally yes, it was about being more cross functional instead of separate silo's.
However operations people had an issue with that(not in a bad way, just observing where stuff is going), and decided to obtain devops skills. So you ended up with pure devops people.
A lot the first devops people were developers, who also knew about infrastructure issues and been in those roles. They were fairly rare.
No. Originally it was about operations and developers collaborating more closely, including cultural change (devs don't throw things over the wall to ops, ops doesn't hide prod from devs), increasing automation (applying development methodologies to operations), improving learning between the ops and dev teams, and such.
These days it's sometimes used:
- to mean traditional ops
- to mean that devs do all of their own ops
- to mean that there's a separate silo between the dev and ops silos that make tools
- to mean the original thing about collaboration and a shift on the ops side to be more programmatic.
Your explanation is a complete and total myth, though. It's just some garbage from a programmer who didn't know what it meant and decided to be a huge jerk about it.
I didn't realise I was offensive, sorry. If operations is moving towards being automated by software it's only natural to want to move more towards programming side.
I'm not saying this a bad thing, only natural. If infrastructure people see traditional server management moving towards automated deployments/monitoring/automated server setup. Then a lot of them would want to move into those skills, which a lot did.
Is there anything wrong in wanting to cross-functional? Most DevOps people are developers who automate infrastructure, or Infrastructure people who move into programming so they can automate.
You need to have both to be really good at it. That's what crossfunctional means.
Btw I am actually devops, and I've been working on it for a few years.
> Your explanation is a complete and total myth, though. It's just some garbage from a programmer who didn't know what it meant and decided to be a huge jerk about it.
Thanks, as an ops guy. I work hard every day to make the best tools available for our developers, and to ensure we have the highest levels of reliability.
If you look at the early history of devops, a lot of it was developers who had previous infrastructure experience and roles writing software for automating deployment, and sever creation.
It sounds to be me you have chip on your shoulder, any mention of programmer help makes you angry? When in reality programmers appreciate modern DevOps people. What makes you thing programmers dislike you?
No chip on my shoulder, nor assumption you're a developer.
You said: "Originally yes, it was about being more cross functional instead of separate silo's."
Which strikes me as the continuation of requiring someone to be familiar with everything in the stack. So you get people who are mediocre at every skill they need, and not particularly good at any single task.
Also, you said: "However operations people had an issue with that"
Not in the slightest. As an ops person, I embrace the DevOps model. My problem is when someone like yourself makes a broad, unsubstantiated generalization such as this.
Ops people want systems to run smoothly, and also want to have a life outside of work (I'm assuming? Perhaps another broad generalization). DevOps means I work with developers to ensure their code works as expected in production as it did in development and staging, and that they're not just throwing it over the fence to me and hoping for the best.
Just that a lot operational people noticed it was trend occurring and adapted to it. They didn't avoid it.
As a DevOps can you develop code to automate things, like deployment, builds, ci, server setup? Can you setup servers, networks, monitoring? If you can do all of these things your cross functional.
Originally before devops, operational teams and developer teams were completely separate. Often separate rooms.
At the company I'm at devops and developers are actually on the same team. Same room and attend the same stand up meetings. We have devops people that have helped create code, and we have developers that are on call for making sure that they are on duty in case their new code falls over. Skills gradually start to overlap. We also have people have switched their focus, developers who moved to more ops skills and vice versa.
We aren't different groups of people. That' what I mean by not separate silos.
This is really great initiative! Despite some fundamental critics and misconceptions, I'd like to say: Congrats, Vlad Alexandru!
In my humble opinion, this is not even close to "serverless", but rather "cross-platforms". The best description to current "serverless trend" that I've heard is that "infrastructure comes with no-ops and pre-scaled at massive scale, that only providers like Amazon's AWS, or Google's GCP, or Microsoft's Azure and alike can provide" ;)
So... this is backend infrastructure (i.e. it runs on a server) to let your team deploy "severless" apps...
So they're serverless. Except the server you're running it on. And the lamba/style code they wrote and uploaded to it.
But serverless, because no ops staff required. Except the ones who installed and maintain this.
This is like a snake eating its own tail and wondering what hurts.
Edit: despite my sarcasm, it's always good to see open-source solutions to reduce reliance (or risk of reliance) on closed vendor solutions. Just stop calling it serverless, and I'll stop telling you how fucking ridiculous that name is.
Seriously. We got serverless being the most BS term I've heard in a while. Then DevOps: a term I still don't understand as I've never seen a concrete description of it, its advantages, and specific examples. We have microservices that people either brag or warn about in HN articles.
Combine them to get... something! Um, something I'll avoid for now until the Cloud of confusion clears up.
Hey, at least I didn't use "cloud-native" or "12-factor app"! On a serious note, though, yeah I can't say I'm keen on the terms. I've known DevOps since it was called SRE 10 years ago, at Google. Just trying to use same terms other people on HN seem to use.
Haha, I hear you. Yeah, cloud-native would've been worse I agree. What's SRE mean? Site Reliability Engineering? Or is my guess way off?
Btw, I really enjoyed reading 12-factor app [1] compared to most since the page was a clear description of obviously good practices and debatable recommendations that were at least clear. It was also about apps with a list of 12 things to consider. That fits the name. Page was over 70% specific justifications and technical answers with almost no hype or zealotry. I was impressed the cloud crowd was capable of that. Mainstream needs to do terms and explanations that way more often. ;)
"Serverless" is wrong in a very technical way - they have a very distinct meaning in their head, but they're just choosing to wilfully ignore a huge part of the stack so they can call it that.
DevOps has as many different definitions as practitioners, and few will completely agree on the same definition.
Cloud is one of the worst frankly, because it is so hand-wavy and it's made it into every day language. My assumption is that this name came from the 'cloud' icon used in a lot of technical diagrams to represent something outside of your control and outside of your scope for the current project. I guess in some ways that makes "Cloud" the perfect term to describe things like AWS/et-al: something that is out of your control.
You hit the nail on the head for each one. My thoughts exactly. Especially serverless, though, as we already did serverless back in our day: it was called P2P and helped start a revolution. What they're describing don't look like no P2P to me. :)
It was a little PCI card. Even so, I'd ask the marketer or developer: could you please draw a logical representation of your client and services with lines showing data flows? Followed by (points) "there's the server(s)." :)
This should be called what it is: a PaaS that automates some of your continuous delivery pipeline, specifically the deployment of your containers across a cluster of servers.
It's disingenuous and not likely to generate a lot of enthusiasm when you glom onto the flavor of the month (serverless) and try to recast a technology that pre-dates it (PaaS) as being serverless, when in fact its not.
Agree this is not serverless, but still looks awesome. Just remove the serverless piece. I'd say this is a platform for running "lamba" on your own infrastructure.
I was developing something similar with effe[1] e effe-tool[2], the idea is similar.
You wrote your service in go following in such a way that it expose 3 function and it get compiled down to a single binary, then you can dockerize your application (images usually are smaller than 6MB) and deploy it, wherever and whenever you want.
I don't know why HN is being so critical, because this sounds awesome and really useful! Similar to what I've been trying to do with databases ( http://gun.js.org ).
It is sad to see so much elitism in the tech world that is dismissive of anyone/anybody that knows less than them. This same attitude seems to come out a lot to dismiss really cool tools like this ( http://leveros.com ) that help solve frustrations and problems. Could we try to be more encouraging of newcomers and of tools that automate (laziness is a programmer's virtue) those gaps?
LeverOS seems to be an open source implementation of a lambda service for running event-driven, "serverless" applications, aka an implementation of the idea that drives AWS Lambda, Google Cloud Functions, Azure Functions.
The Serverless Framework is one of several frameworks for writing serverless applications. The name of this framework is unfortunate as "serverless" is a generic term referring to event-driven applications, but has certainly benefited from the confusion that its authors have caused.
Those less familiar with these products can look at this as "RPC as a Service".
A big part of the value of AWS Lambda is that it allows you to glue together other AWS services with custom application logic, similar to stored procedures in a database. Without the other AWS services and the event triggers they expose to Lambda, "serverless" is a lot less interesting.
So I know something like this (or AWS Lambda) is good for quickly putting up an endpoint and scaling it to theoretically unlimited nodes.
Could anybody shed some light on how this might be used in practice? It seems at the point that you have multiple endpoints, you may as well just deploy a full application.
This reminds me of Zappa [0], only Zappa actually runs on AWS Lambdas. Although they readily admit it takes a lot of hacks to get WSGI working within a Lambda function.
This is wonderfully naive. A great goal, but hopelessly unrealistic. E.g. currently it is not handling data persistence at all, so for starters it's targeting "the easy bit", yet there's hardly any mention of things like logging, or what do to when their Consul cluster decides to start complaining because membership has gotten messed up (ever had a Consul node rejecting most messages as "suspect"? ever had the same node get registered twice? all kinds of fun stuff), how to analyse performance issues, networking problems etc.
It looks like an interesting platform, but anyone thinking this gets them out of understanding ops concerns will have tough times ahead..