Ask HN: How did you get your first job as an sysadmin?

[_lnwk]

Heya HN, I'm thinking of my career prospect after university and I've came to conclusion I feel most drawn into administration as a field. I wanted to ask you for what your experience in getting job as an administrator after university, especially if it was your first job in IT that you did. What were the skills you had to have? How did you on about looking for the job, or internship? Have you had any certificates and so on?

[bravetraveler]

I feel like I got a relatively easy start, so I'm not sure how helpful I can be.

A friend I made (through Counter-Strike of all things) put in a recommendation/I applied.

The rest is history! I did SysAd for about 8 years and eventually moved onto SRE/Architecture

Prior to this I was a hobbyist, setting up LAMP stacks to power my feeble attempts at learning PHP. I had never done any IT work professionally.

I learned pretty much everything meaningful on the job. Expectations as an L1 were very low. Be aware of services, protocols, firewalls, things like that - and think twice before you act.

For someone looking to get started with Linux, I recommend the material for the 'Red Hat Certified System Administrator' course.

It does a very good job at teaching one to fish. Less rote procedure, more 'how to read the manuals'

Certifications in general haven't done much for me. With that said, more than once somebody has mentioned my Red Hat certs

[markus_zhang → bravetraveler]

Thanks! Judging from what you see, which one is slightly more useful, Red Hat or Debian/Ubuntu? I assume Debian also has those certifications?

I'm looking forward to transferring to a DevOps from a Data Engineer btw.

[bravetraveler → markus_zhang]

You'll Find Red Hat, Debian, and Ubuntu all over - depending on the needs/wants of the shop. General familiarity is good, but there's no harm in specializing.

Just realize that outside of package management (and some quirks that allows), Linux is Linux.

Distributions are all made of roughly the same pieces, they vary slightly in how/where they place them

For example, the things here tend to hold true - being one of many standards:

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

[markus_zhang → bravetraveler]

Thanks, guess I don't need to worry about it.

[_siis]

Claude Shannon huh?...

You generally don't get a first job in IT as a System Administrator, ever. Its one of those unicorn stories you hear about that almost never happens. That being said I'm one of those unicorn stories but then again I had a hell of a lot of unofficial experience backing me up. I've run system's in professional configurations since I was little. I caught a break in my 30s and transitioned internally from a assistant position to their IT System Administrator after their IT Contractor disappeared without notice with a pending SAM audit and left them holding a bag with a lot of over-provisioned licensing.

The accepted path is you get in via Help Desk Support. Several years as a T2/T3 may develop into additional responsibilities which overlap some with what a classical Sysadmin does which allows you to make the leap.

Certifications are a poor replacement for a degree. They show you are up to date sure, if you can pass them. Most of the time passing them is not about knowing the knowledge but about effectively dealing with issues that in any other business sector would be considered fraud and unfair and deceptive business practices, but they get away with it because it is wrapped up behind arbitration and non-disclosure.

Its done this way for profit motive and because you have no alternatives since only one place does the testing for these. Those can also be revoked for any number of reasons unrelated to the test itself.

In my opinion they are marketed fraudulently to employers as minimum qualification for a role while setting a very high bar; and inherently limit the number of people who receive them in a way where knowledge is only secondary to profit motive (repeat customers).

I've taught email server administration and had people I taught pass; and then when I tested with almost a decade of experience in that area, it was failed for issues not related to knowing the material that were outside my control. 1 month before they would even respond, 3 months of back and forth before they finally issued a full refund for the clear failures on their part and my pigheadedness about documenting every single conversation in writing (proper case-building). Unfortunately I can never share the specific details that are covered by the NDA so long as it is in place.

The parties involved also apparently have blanket liability protection which they received from some of their government contracts according to one of several lawyers I spoke with from my state's BAR referral line, when I was considering suing them. None would take the case without a significant retainer, and didn't have high hopes for a positive outcome.

HR also favors degrees.

If you have neither, many companies use this as a negotiation tactic to draw you in; and then low-ball you. You can have 10 years of directly applicable experience doing the same job and they will say "Since you don't have a degree or certification you aren't qualified, and we can't pay you what we would pay someone who is qualified; or We don't have that position available but we have another at .... whichver it is its <half pay or minimum wage doing the same skilled labor>. This is all done after you've invested time into the interview process for a position that clearly was never available to you, except they wasted your time so you would accept the low-ball as desperate people often do.

I still regularly see this happen with increasing frequency and stale certifications such as deprecated certs which you did pass; are the same as no certification. I keep a list of entities in my area that I will never apply to because they have shown they are bad actors. Incidentally, state run job board listings often have less of these because the companies involved have to certify the accuracy and availability of the jobs; often under penalty of perjury. Something to be aware of.

As for the most important skills you needed to do the job... Stress Management, Documentation, Risk Management, and Communication, followed shortly thereafter by a mindset of constant improvement.

If you lacked any of those; bad things happen.

Its not enough to write things down, you need to write them down in a way that at a glance you and anyone else get all the most important information tailored towards the job that will be working with it.

If you can't handle stress, you'll get high blood pressure, early onset diabetes, and potentially other medical conditions that will shorten your life. Nothing is worth your health.

Everything must go through a risk management paradigm. Before you run a command, you know it won't bring everything down, and have a contingency in place for if it does so you don't have to think. Its written, you follow each step mindlessly. You do everything you can to minimize risk to the business.

Communication, if you can't communicate in a professional tone and manner you will be crucified by holier than thou management. This includes non-verbal communication such as attire. They will walk all over you making your life hell, undercutting your authority; minimizing your advice, etc.

You know what you are talking about, but if you cannot communicate it along with the appropriate risks; you can't do this job.

You can find the bulk of most of the methodology you will need in two book volumes by Limoncelli Hogan and Chalup. TPOSNA.

[brezelgoring → _siis]

> Certifications are a poor replacement for a degree [...] HR also favors degrees [...] Since you don't have a degree/certs and we can't pay you what we would pay someone who is qualified

What a joke, I see it all the time. The worst part is that by the time this happens it's too late to do anything about it. What are you going to do? Let me get my 5 year degree real quick, brb. I'm almost 30, I can't go back to school and stop working.

I bet on work experience and self-learning as a substitution for paid education because I was dirt poor and it was the wrong bet, apparently. 10 years of work experience and I have little to show for it.

Good write up, than3, thank you for this.

[_siis → brezelgoring]

No problem, it has been my experience, and I try and pay it forward when I can. I'd have appreciated knowing this back when I was first starting out, but it wouldn't have changed much in terms of outcome.

As a young adult you don't often really have much experience dealing with duplicitous or deceitful people and processes.

Incidentally, I don't really go into it but there's almost as many if not more problems with getting a degree compared to certifications but it is the favored solution for the job market since you don't have people invalidating it (through deprecation) and its a one time fixed cost (after completion).

Without getting into detail, many core transfer requirements at the course level often don't provide any means for a student to control basic academic outcomes in bottlenecked courses and there's no effective due process to correct issues. You are at the professor's mercy and a lot of them simply don't want to teach but they still want to be paid for it. Its a form of fraud, but not one you can really go after them for without it costing more than the personal benefit.

That 4 year degree that is advertised may as well be a never-year degree if you are unable to somehow get around these arbitrary, unadvertised, but widespread impediments. Trying at another college or getting another teacher often isn't sufficient as these are widespread systemic issues. I've gone far afield while I was trying to get an engineering degree (early on), short of going to school outside the US , what you see with these things is what you get.

I've personally been slogging along for most of my young adult through mid-life trying to put myself through college for a degree while working. It may happen that I get a degree in business or underwater basket-weaving by the time I'm supposed to retire if I get a lucky break.

In the meantime I make a decent living by referral since the vast majority of most advertised positions are junk; and I'm a bit of a miracle worker, my reputations gotten around locally and my work pretty much speaks for itself; I've more than a few systems that are still running a decade after deployment with no major issues. Ransomware incidents have happened occasionally, but they've largely been inconsequential to the businesses involved as a result of preventative measures and policy recommendations.

Those junk positions are usually some data miner, or broker seeking to either elicit, obtain or sell your personal or proprietary information they get from you to someone with no real position being available. They've monetized the interview process and advertise year round, same positions.

I've had more than a few instances in what I initially thought were good-faith interviews, where I've had to end the interview prematurely because it became clear they were more interested in trying to solicit proprietary information that they were not entitled to. Sometimes you just have to walk away, any place that tries to do shady stuff during the interview process isn't a place I want to work and you are interviewing them just as much as they are interviewing you.

[markus_zhang → _siis]

Thanks for sharing. As a data engineer who dabs into docker and k8s and terraform but never heavily use them or build infra from scratch, what's my best shot to get into a DevOps position, if moving across team is not possible?

I do have a degree (math) and am really thinking about getting some serious certifications such as the Red Hat one the above commentor talked about, or some docker/k8s certification although I think they are a bit shallower than a Linux one. When I'm free, recently I began to dab into Linux system programming and a xv6 course but they are at best a long shot.

Programming language side I'm OK with Python, knows some C from system programming dab and C++ from another hobby of building small game engines.

[_siis → markus_zhang]

In many areas, DevOps means different things to different companies.

While I'm not in a DevOps position, nor have I been in one. The people I've met who were, primarily focused on providing value and automating repetitive work.

This usually took the form of creating tooling or instrumentation that made everyone else's jobs easier or reliable (eliminating human error).

While that is easier said than done, the tooling they created needed to be rock solid, and there are certain operation aspects that are often overlooked because Operations rarely has as much input as Developers in more classical teams. As long as it continues running without problems few people care.

Volume 2 of TPOSNA for Cloud breaks some of these operational requirements down. I'd also add, its important to understand and know the limitations of computation, and maybe a little compiler/automata theory on the types of problems and what leads to halting/undecidability.

In that kind of position, you will often be dealing with automation; and you may need to be able to verify requirements of computation such as if a output interface is injecting non-determinism into subsequent forked processes. Any automation after that will fail in unpredictable ways, and it happens in a lot of places (take a close look at ldd coreutils output sometime).

There are a few core requirements that programs assume are true, but then break down when those assumptions stop being true. Having an implicit knowledge of System's and Signals is very valuable with this in mind as those properties can be tested; and effective troubleshooting only works when certain properties are present allowing you to characterize problem classes quickly and allocate/provide estimates of time more effectively. Non-deterministic type problems take exponentially longer since you can only guess and check.

As for certifications, if you want a background in System Administration RHCSA was good; I haven't seen it since they were acquired by IBM.

You should look at the person who issues the certifications and see how long or how they validate those certifications. With deprecated certifications, they often stop validating whether its been issued so its as though you didn't have any certification.

[markus_zhang → _siis]

Thanks for the long reply. Your observation agrees with mine about the automation part and thats's exactly the reason I love the job. I did a lot of automation for the team.

But I'm missing a lot. I know absolutely nothing about network. I don't know how to build infra-as-a-service as our DevOps does -- I only use and dev on top of them. I'm especially wearing a blank face reading your "system and signal" paragraph.

I guess my best shot is to learn by earning a certification and go from there. I found all companies need sort of a senior DevOps who can immediately begin work. I need to fill at least some of the gaps.

[_siis → markus_zhang]

To elaborate, System's and Signals is a course usually reserved for EE majors, you can watch lectures about it at MIT OCW; there's a Textbook which is relatively cheap and while its very math heavy (probably requiring Calc3+a bit of abstract math for EE work), an intuitive understanding of what the properties are and how to test for their presence is all you really need to get some basic use from it as a System Admin/DevOps.

A barebones understanding of network is pretty simple (its an onion), it can get complicated as soon as you need to start worrying about time to convergence, multi-homed networks, and/or dealing with BGP and ASN's or if you need to reimplement network stacks. CCENT/CCNA/CCNP are certs that cover the material exceptionally well though its dry to put it mildly. Its also what I would expect of a Network Admin, not a System Admin/DevOps.

I've found System's and Signals provides a useful paradigm/filter for characterizing certain classes of operational problems (by whether the properties are present or not in computational contexts).

Computation for example in general require Step (usually associated with a clock signal), Time Invariance (given the same inputs provides same output regardless of time shifts), and Determinism (the fixed same input states going into function only provide a single unique output) systems properties to function and do work.

The latter property must hold and includes any series of intermediate steps as well, mainly to fulfill requirements of the theory of computation (afaik) regarding turing machines (finite state automata/automata theory with implications of halting and decidability when not present; this is mainly found in the dragon book or a compiler design course, pretty sure video courses for this are available on MIT OCW).

Time Invariance for example is affected by Memory properties and fails in the presence of Memory (such as a cache, which is why there must always be a way to clear it from an operational perspective to allow troubleshooting).

Troubleshooting interconnected systems relies heavily on Time Invariance. If you can demonstrate the property isn't there as a rule of thumb, you can make estimates that it will take much longer to resolve certain issues given the guess and check nature of the problem.

While these are mainly rules of thumb without formal verification (at least for me), they've been aligned from observations I've made and served me well as an additional tool. Most of what I know in this regard is self-taught aside from the initial exposure from going for an undergraduate in engineering where I took but withdrew from this course. I don't have a degree, though I know quite a bit about computer science and related theory at this point.

If you can verify a property is missing, you don't spin your wheels, also expectations are set appropriately upfront once you know the property is missing; well before investing a lot of resources to work the problem. Sometimes the problem isn't worth digging into in terms of cost, and recognizing it allows you to offer value in potential cost savings.

As for Infra, Scaling, and such. Volume 2 of TPOSNA will give you a solid foundation as its about Cloud. Volume 1 is all about in-house IT.

Edit: Being able to compose state graphs and a cursory amount of graph theory is also important, if you have to verify determinism. Most commonly people miscount the absence of something as a state, and it ends up borking the automation in weird edge-cases since the absence often gets mapped as two or more potential (non-unique) output nodes breaking determinism on a state graph.

[_lnwk → _siis]

Yeaaah, I didn't really thought much on the nick while creating account. I just picked first person I saw on Wikipedia page.

Okay, to be more serious - thank you