
How to (Re)Build My Infrastructure Team? - ctoinneedofhelp
https://pastebin.com/55rJnHUy
======
raincom
You have identified some problems

\- k8s is not fit for your set up, unless you have 1000s of nodes serving an
app and tens of such apps.

\- Software engineers are not going to show any interest in infrastructure as
code, as they can make more on the engineering side than on the infra side.
Even at Google, I heard, it is hard to become SWE for an SRE, unless he joins
as SRE-SWE. This tells that not many want to become SRE-SWE.

\- What else can you teach to your sysadmin, when he can't learn "owing up to
mistakes" in a small shop like yours. Im Mega corps, there is a political cost
to "owning up to mistakes" right away. If he can't change, he is not going to
learn it. You need someone who wants to learn and show some progress.

\- Where is your company located?

~~~
ctoinneedofhelp
We are located in south east asia. In a place where it's pretty difficult to
find decent software engineers unfortunately.

~~~
raincom
You have better chances there for finding a software engineer who wants to do
infrastructure. Good Luck. If you are in singapore, you can find them easy.

~~~
ctoinneedofhelp
Thanks :)

------
mattbillenstein
What is your scale? I think a single engineer given sharp tools and a limited
amount of services to deploy can support probably hundreds of systems -- this
can be a vast amount of compute and storage given the state of today's
hardware.

~~~
ctoinneedofhelp
<100 servers to manage. <100TB of data. <50 technical users.

~~~
mattbillenstein
Cool, I hear you regarding choice of tech stack, cost constraints, etc -- I
manage systems with saltstack, Ubuntu, supervisord -- a very much 12 Factor
app architecture, but the new way of doing things is k8s and docker. A huge
amount of complexity imho.

I think you need to hire programmers and groom them more into devops. They
will think the product is the automation software while the artifact is the
systems. Traditional sysadmins think the systems are the product and the
automation software is the artifact, so they tweak the systems to fix things
rather than fixing the software and then letting that fix the systems.

So, it boils down to philosophy and what one group thinks is the source of
truth vs what the other one values.

And maybe you can reach into one of these communities - puppet, ansible,
saltstack - find a nerd in that space who shares your sensibilities who you'll
have an easier time working with.

~~~
ctoinneedofhelp
I understand how k8s seduces people: sysadmins feels like "let's deliver this
to developers then they'll be autonomous", developers feels like "oh cool, i'm
free to run my containers". That's cool for small web shops with a lot of web
services. But much less cool when you want to control/understand what's
running within those containers. Then you need a lot of CI engineering with
precise access control.

That said, I couldn't care less about all that container hype. Long running
web services are <0.01% of our workload.

You confirm my feeling that I should now target good software engineers and
teach them how to program the infrastructure. But how would you make them
interested in infrastructure? My understanding is that they want to be exposed
to the business, closer to where the money is, to get more exposure. And
that's normal, that's what smart people do to leverage their skills.

You're right I should target my tech stack open source communities. I thought
about it so many times. But I just don't know how to communicate what I want
without being classified as a spammer. So I never did it. ;)

~~~
mattbillenstein
Yeah, not an easy task - some of them have job boards or you can do the usual
digging keyword search on LinkedIn. I dislike LinkedIn, but everyone is on
there and being solicited about job opportunities is pretty much the norm.

~~~
ctoinneedofhelp
I did a lot of hunt on LinkedIn with keywords like Terraform, Python and
Puppet. I found that guy here. But years later, I have doubts about LinkedIn
efficiency, as most good SWE I found through head hunters did not have the
Python keyword on their profile even when they rock at it. Now I believe
LinkedIn is only useful to target people who are good at self-marketing.

------
ctoinneedofhelp
Sorry, I needed more than 2000 characters to give my question the right
context.

------
bryanrasmussen
as a general rule if I feel it is difficult to find the right mix of abilities
in one individual I want to hire 2 or 3 people and make them work together,
that is of course also expensive but might be necessary.

~~~
ctoinneedofhelp
Fully understand that. But beyond the cost, my main concern is communication
bandwidth. So productivity. Not even sure 3 people would be as productive as
the right guy.

