> I've never come across a devops person who'd say "Hey, that sofware is too simple to prematurely scale for sudden spikes of irrational amounts of traffic, so why not just deploy it on a cheap vps?"
Was a devops engineer at my previous job.
We already had k8s clusters setup, pre-made CI templates and pre-tailored helm charts (along with monitoring and much more). All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place).
Creating (and maintaining) a single vm for a pet project is way more work than using the pre-made and pre-customized and curated toolkit.
This was at a 100+ developers organisation.
If you think you could easily get off with a single vm then you've never seen devops done right, i'm fairly sure.
EDIT: I probably fell for the bait, but the post i'm replying to really made me remember why we went on a killing spree in order to eradicate everything that was not k8s at my previous job and removing as much developer access to prod as possible. Some idiot developers think they know better, usually end up re-inventing a square wheel that breaks as soon as it's not running on their laptop anymore.
IMO the vast majority of software development happens in much smaller organisations than that. Dev Ops still matters there, and the requirements are different.
I am working in an organisation with one and a half developers. I am lobbying that the third of fourth developer concentrates on Dev Ops here.
It is very important. Look at the back up/recovery procedures at your organisation. Has there ever been a fire drill? Are you sure the back ups are sound? Can you recover? What if data corruption occurred a week/month ago. Have you a back up of the uncorrupted data?
That is a very unsexy aspect of Dev Ops, and without somebody dedicated to the job, your backups will not be any of those things.
> I am working in an organisation with one and a half developers. I am lobbying that the third of fourth developer concentrates on Dev Ops here.
not sure you're doing the right thing here. you might want to consider hiring some kind of linux guy that can can do some basic devops. or maybe hire some devops contractor that can work with you on a part-time basis and "curate" some specific aspects of your operations.
I've seen this done in the past: so you've got this consultant on retainer, and you tell them something like: "i've got this issue, can we do something about it? our constraints would be x y z" and the consultant would make 1-3 proposals (different approaches, different pricing levels, different ETAs etc) and then you agree on what gets done. The key aspect here is that a good devops consultant can get stuff done very quickly.
> It is very important. Look at the back up/recovery procedures at your organisation. Has there ever been a fire drill? Are you sure the back ups are sound? Can you recover? What if data corruption occurred a week/month ago. Have you a back up of the uncorrupted data?
Yes (to all questions). I ended up in working in heavily regulated environments. All the things you mentioned were not just niceties, but mandatory by legal requirements.
> That is a very unsexy aspect of Dev Ops, and without somebody dedicated to the job, your backups will not be any of those things.
That's basic system administration. Most devops engineers are former sysadmins.
> IMO the vast majority of software development happens in much smaller organizations than that
I guess it's okay to have an opinion about that, but this seems like something that should probably be a fact. Unfortunately, not sure I can find reliable stats on sizes of engineering organizations.
The thing is, while there are obviously lots of small companies, there are also some really big software development organizations out there. A company like Netflix has 2,500 engineers. Microsoft employs over 100,000 engineers. Walmart employs over 15,000 software developers.
You need a lot of little 10-50 engineer dev shops to add up to the combined size of the engineering orgs of the Fortune 500.
According to https://www.statista.com/statistics/507530/united-states-dis..., at least, 29.4+25.8 = 55.2% of the US "IT Industry" workforce are employed in companies with >100 employees. That's a Long way from telling us about sizes of engineering orgs though.
But still... I'd be careful assuming that the vast majority of developers are in organizations of less than 100 people.
Why do you assume that only three US landscape is discussed here?
Plus I'm not very sure those statistics are reliable.
Anecdotal evidence: I have 22 years of career and I've only worked in big organizations twice, for the total of a year. Everything else was much smaller.
But you’ve got 100 developers. That’s not “most web apps”, that’s firmly in the set of companies that need standardisation and potential scale, where the devops team makes the life of many devs far better. When it’s just a few devs and cash is limited, the business doesn’t need the complexity. It’s mostly for ego and branding.
When the company is set up well on k8s, then choose k8s. If the company is set up well on VPS, then choose VPS.
If it got neither, I'm unsure what's the better way to go on a greenfield.
k8s has nice tooling, but part of that is required because it is massively complex.
But with managed k8s providers (e.g. GKE with autopilot) you can just put in proper CPU and memory limits and essentially get yourself a VPS provisioned without having to worry about anything by k8s measures.
If you have different pricing or location constraints, then it might be better to go with different models and some custom orchestration.
Yeah, I'm an independent, selling these solutions. My minimum stack costs about $500/mo in AWS costs- and you could save 90% of that. But then you'd be paying me $20k more to set it up again when you expand to another dev team, while this way I can add your second dev team for $10k and practically zero additional AWS cost.
Going straight to overkill is good business sense for any company that's going to make it to a medium-sized business, and it's not going to be the differentiating factor for a company that burns out.
> All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place).
How did/does the devs create, test new code and debug issues? Can they do that locally on their local laptop? If so, how?
>> How did/does the devs create, test new code and debug issues? Can they do that locally on their local laptop? If so, how?
I used to buy the idea of this when there were monoliths.
Then I had the joy of running a shop with dozens of web properties.
Local dev became untenable. My systems admin was an early adopter of Xen. That shop ran like a dream... devs could come in and have a new environment update in place and just start to work. Staying in sync with prod was never a problem.
By making systems guys keep devs fed, by making devs work closely with systems folks you get better software. Containers just hid developers shitty decision making in a wrapper that Systems folks can tolerate.
And how DO you debug... cause what you do to figure a problem out locally is not how you trouble shoot when the shit hits the fan in prod. These tools should be the same, sane and well used and loved by everyone. Local debugging is part of the problem, even more so if your service based app that lives and dies on the wire.
Then I assume you have a custom Kubernetes LB that can handle non-HTTP TCP and UDP traffic because you choosing Kubernetes and the design restrictions that comes with it does not affect how the dev solves problems?
The underlying orchestrator definitely affects how the software needs to behave and is definitely not irrelevant.
> Then I assume you have a custom Kubernetes LB that can handle non-HTTP TCP and UDP traffic because you choosing Kubernetes and the design restrictions that comes with it does not affect how the dev solves problems?
nginx-ingress-controller does that. no custom stuff required.
Was a devops engineer at my previous job.
We already had k8s clusters setup, pre-made CI templates and pre-tailored helm charts (along with monitoring and much more). All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place).
Creating (and maintaining) a single vm for a pet project is way more work than using the pre-made and pre-customized and curated toolkit.
This was at a 100+ developers organisation.
If you think you could easily get off with a single vm then you've never seen devops done right, i'm fairly sure.
EDIT: I probably fell for the bait, but the post i'm replying to really made me remember why we went on a killing spree in order to eradicate everything that was not k8s at my previous job and removing as much developer access to prod as possible. Some idiot developers think they know better, usually end up re-inventing a square wheel that breaks as soon as it's not running on their laptop anymore.