My company self hosts everything. They’re so bad at it. Something is always down and we waste so much time with our shorty tooling. We even had this amazing idea that we could implement our own version of GCP from scratch. The result is dismal, we dread using it because it’s very unreliable, and it costs double or triple what GCP costs. We’re not a small company, we have about 1k employees and our business is software.
I've come to appreciate the value of a really good, dedicated sysadmin. I used to think I was pretty good at it and that was fine, but I've come to realize that the skillset for an awesome sysadmin are quite a bit different than a developer, even though there is some overlap. And there is definitely a range.
Exactly! I'm a sysadmin (sometimes a Dev), but sysadmin is what i'm really good at. The most important thing is to see a solution from another perspective.
As a developer I completely agree. I have some knowledge of system administration, but I much prefer with dedicated professional where possible. It's partly just a different set of expertise, but I think it's partly also why it's best to have dedicated people doing QA work. The attitude required to do the work is completely different to dev work, and it's hard for one person to wear both hats.
Seconded. A sysadmin dedicates their professional life to deeply understanding systems, networks, and their interconnection. I am lucky if I can keep up with changes to my languages and frameworks as a programmer! Thank goodness there are sysadmins out there who can help us poor programmers out when our relatively basic understanding knowledge of linux fails us.
>Thank goodness there are sysadmins out there who can help us poor programmers out when our relatively basic understanding knowledge of linux fails us
Sometimes, and sometimes we have to ask the developer of that exact OS-subsystem/driver/firmware(AARGH!!) because it fails us too...and here we have a closed circle ;)
Unfortunately the market doesn't appreciate systems operations skills as much as software development, despite those skills being rarer and having a wider organizational impact in more industry verticals. Software developer salaries trend 20-30% higher at every single career stage than sysadmin/ops salaries, and at the top, there's usually management adjacent engineering track stages at larger firms for software developers like "principal", "distinguished", and "architect" which are not open to operations folks.
I was in ops for 13 years, and if you were to talk to any of my former coworkers they would bury you with praise for the quality of my work. Yet, I eventually chose to move into a management track because I had peaked my career about 7 years in and didn't realize it until later. There was nowhere I could go up, because I wasn't a software developer. Now I'm a manager that has technology understanding, which has a high value prop for many orgs all on its own, but I do sometimes miss "getting my hands dirty".
I've worked with a lot of software developers over the years, and while there are a handful who are really incredible, the majority of people are just mediocre. That's expected and okay. The same is true for Ops folks, as it happens, although generally it takes more competence to rise to "Senior" on the Ops side vs software. The thing is, "Senior" is as high as it goes for Ops folks. So you might meet really stellar Ops folks who are effectively titled and paid the same as a mediocre developer with 3 years of work experience. It's simply not sustainable, and the push towards moving everything to the cloud and off-premise is probably a symptom of this (not enough quality Ops folks to keep things on-prem) and exacerbates this (reducing need for quality Ops folks, driving down market demand, unless you want to work at a cloud provider).
Pretty much all of the other Ops people I've respected and admired over the years have moved into different career paths. I find the same is not true for software developers. So when younger people ask me about career paths, I always recommend software over Ops, if they are adamant they never want to go into management.
It's kind of sad, I suppose, but that's the way of it. I appreciate that there's a subthread on HN where folks recognize and respect the value of competent Ops folks, but I think you'll find that most are being pushed out of that career path.
Probably because of places like Netflix and Google where everyone is a software engineer they just happen to have different titles. If the industry wanted to counter the move to cloud raising the salaries of truly competent ops people would be a thing.
I see what you mean. If those 'value additions' can work on metadata, the separation of data and metadata might help some, but that will become complex very quickly.
> The result is dismal, we dread using it because it’s very unreliable, and it costs double or triple what GCP costs.
I'll be honest - you guys need to find a new technology partner then. Creating a private cloud that's reliable and offers the basic services that the major providers have is not difficult in 2020. Some of the aaS stuff can get tricky but is still entirely do-able.
Regardless, if they found a way to make your infrastructure MORE expensive than the public clouds they either have no idea how to negotiate or are really, really bad at their jobs. The public cloud is a lot of things - but cheap isn't one of them.
Presumably Kubernetes is the answer here, but you need to run a lot of things yourself that come for free with GKE. Also “just buy bare metal” ignores the massive effort involved in operating a data center.
I don’t think so. But knowing my company’s capabilities and proficiency, it was obvious it was never going to work. Would have been much more logical to buy from one of the many local providers who run their own data centers. It doesn’t really matter because in the end the solution is so bad that we try and use GCP anyway whenever we get the chance.
Depends on what is desired... theoretically using Proxmox and VMs containing CapRover hosting docker containers should get you everything you want... but the ability instantly scale upwards is difficult to do yourself.
> the ability instantly scale upwards is difficult to do yourself.
If your business involves any amount of low priority bulk compute, this gets much much easier. You simply let the low priority stuff fall behind while your order to Dell for new servers is being delivered...
Also, if you have compute that could be on-prem or could be in the cloud, you can set up a kubernetes cluster spanning both and let non-privacy-sensitive overflow to GCP as needed.
All of the above rarely comes out cost-effective though, because while the raw compute is cheaper to DIY-it, when you factor in the staff time to build, maintain, and deal with the shortcomings of your bodged-together on-site solution, it's going to come out much more expensive.
I use both MS and Google at various locations. The result is also dismal... But we can all be exasperated together and vent about how awful it is without upsetting anyone.
