Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
The Linux Talent Draft is On (linux.com)
61 points by Tsiolkovsky on July 27, 2012 | hide | past | favorite | 54 comments


Furthermore, As someone who has been hiring every skilled developer I can come across for three years, I have found that good linux skills are one of the best markers of good development skills. There is something about people who aren't afraid to , and even enjoy , using linux that corresponds with an aptitude with programming, this is especially true of back end developers.


Also, though I have no sources on this other than personal experience, linux skills are frequently indicator of heavy prorastinator.


Realizes he's had a Clojure REPL open for days untouched.

I have no idea what you're talking about.

EDIT: Six minutes later sees this post and realizes he STILL hasn't gotten back to what he was working on.


Procrastination means absolutely nothing if there is good output. If you are not on the floor of some factory sweatshop, it isn't necessary to have continuous even output.

Procrastination can be managed. One can procrastinate on one task by carrying out another.

If I do not work for an hour and then make the perfect change, that's better than floundering for six hours.


Usually, when I do feel like not working on what I need to be working atm, I work on something else I DO feel like working on. Even when it costs me a deadline miss, I end up with something tangible in the end.

That's my best yet strategy against procrastination.


I always find the work you do that you are interested in usually has implications for the work you don't want to do, positive implications usually.


You are not alone. Consider the art of structured procrastination: http://www.structuredprocrastination.com/


The three virtues of a programmer are laziness, impatience, and hubris.

Thanks to Larry Wall.


That's a curious association I've never made. Can you say a bit more about that?


What draws people to Linux is the possibility of tinkering. It's not like tinkerers are always procrastinators but they prefer tasks that engage their "tinkerer sense": trying out a new API or maybe doing something unusual with an old one.

Doing something mundane with a known API is something they will avoid. Such tasks are probably in the majority of those found even in the most exciting jobs. This is what I believe to be the cause of the association.

Source: generalizing my own experience to the whole population of Earth. Btw, I gotta get back to work too.


Well, where there's much to play with, you end up playing much.

I frequently find myself playing in my .vimrc, .screenrc, .bashrc and other confs to not much obvious productive output.

Changing window managers, console terminals and anything basically, all in excuse of not working on job at hand.

On the upper hand, beats playing solitaire.


That's not procrastination, that's push-back. ;-)


What's missing in this thread is a description of what kinds of Linux jobs are going unfilled. I'm genuinely curious to know what the landscape looks like.


When I was hired for my job in information security, Linux skills weren't required or even spoken about. After I started, I automated some of my work with shell scripts and wrote other shell scripts to clean up my data for me. Now any new employee that comes in will see "Linux shell scripting recommended" on the application.

"Linux jobs" might be rare, but jobs where Linux skill comes in handy are abundant.


Senior linux system administrators are in very short supply. Companies have given up training people for that kind of work, schools don't teach it, and most of my sysadmin friends migrated to management or dev over the past few years. I've been looking around for a new gig in San Francisco (not south bay) the past month and sometimes it feels like I'm the only candidate and there's all these companies that can't find people.

It's nice feeling wanted but I won't do the work of five guys by myself and be the only one on call for years at a stretch, something will have to change soon. I wish Google would adopt that Lee Iacocca philosophy of training people specifically to spread their way of doing things across the industry, at least there'd be competent mid-levels around then, but they just churn those people on 1 year contracts like everyone else.


While this may be true it is important to note that the corollary is not necessarily true. That is to say, not having good linux skills doesn't make you bad developer. I've found plenty of people who don't know linux very well and are fantastic developers.

Personally, I'm having trouble living without grep (work is easier on a windows machine).


I'm sure you know, but there are many ways to get GNU grep or similar programs running on Windows.

http://stackoverflow.com/questions/87350/what-are-good-grep-...


Sure, there are ways to get various Linux-like tools on Windows, or Mac.

But inevitably -- and I've been using Windows offerings since the mid 1990s and Mac for the past 6-7 years -- these are nowhere near as complete as what you'll find in native Linux, they're missing key bits, polish and bugs are less than optimal, host integration isn't what it is in Linux, host features lack key Linux bits (especially /proc and /sys, and all that they enable), and if you want full coverage, you'll almost always end up with several different toolsets none of which offer quite what I'm looking for.

And you're missing soup-to-nuts, all-encompassing, sane, bug-reporting-integrated package management. On other systems, I manage bits of my Linux-like environment through better or worse (Cygwin -- don't get me started -- but look at apt-cygwin) package management. But most of my OS is outside of that. Inside Debian, I might manage a very small handful of non-packaged software items, mostly browser plugins. Oh, and lately Ruby gems, which are their own brand of hell, but whatever.

I first switched to Linux when I realized that I was spending more and more of my time trying to make Windows NT look like what I got out of the box with a $30 Linux distro CD (bundled with a book). Other than a few distro hops (last to Debian, in the late 1990s, though I continue to evaluate / work with others), I've never looked back.


Whenever I get stuck with a Windows box, one of the first things I do is load cygwin.

I even have it on a Win-VM running in OS X.

It's not a perfect solution, but it works well enough that I can live with it (which is rare).


Somewhat similar to Cygwin is UWin. Tried it recently.

http://google.com/search?hl=en&q=uwin

It was created by David Korn, author of the Korn shell (ksh).

You can find the UWin site link some hits down the Google results page. (First link is for its Wikipedia entry, then there are some unrelated hits for the same string "UWin", then the actual site).


What's preventing you running a VM to let you use it (and all the other helpful bits)?



Familiarity with Linux means people know the basics of UNIX, which is essential if you're doing anything on the web these days.


Don't tell .NET developers that.


Of course. If you love programming, an OS whose primary interface is a programmable shell is going to be very appealing. On top of that, the large number of small programming languages that come with typical installs of the OS (awk, sed, dc, bc, sh) make Linux a very comfortable place for programmers.


Do you mean bash/shell skills, familiarity with unix tools like grep? Or that deep understanding of an operating system is useful knowledge for a programmer?


It sounds like he is talking about the former.


> But 85 percent said finding Linux talent is difficult.

I couldn't agree more with this statement. I work in Chicago--which you think would have a rather large pool of Linux talent--but it's incredibly hard to find really top-notch Linux people at any price.

Like any technical job, there's no shortage of candidates that have "Linux System Administrator" on their CVs, but five minutes into a phone screen you can tell that nearly all of them are what I like to call "application drivers". They've been paid well to sit in front of an application GUI (vmware, backups, storage, etc.) for 10 years and push buttons. It's astounding to me that it's so hard to find people who can do the most basic things: build packages in the OS native format from source; write complicated shell scripts; write complicated applications in (python|perl|ruby); debug and port C/C++ programs; etc.

In my last job (Dept. of Energy lab), I used to think it was hard to find people because the pay wasn't good enough, but in my current gig in the financial industry, we're willing to pay basically whatever it takes to get the best talent, but the pool of great candidates is very, very small.

I think part of it is that in a shaky economy, the best people don't get laid off and they're hunkering down, unwilling to risk taking a new job, but I don't really know.


If they have all of that money for hiring qualified Linux talent, but they are unable to find it readily... why don't they start an apprenticeship style program? I know in the highly regulated nuclear operator field, companies setup schools, where students are paid to attend classes and if they can get their license, they have a paid position waiting. Why not something similar for Linux professionals?

I have been in IT for nearly 20 years and it is downright shameful how companies these days are willing to claim they will pay whatever it takes to get IT talent that doesn't actually exist, but would rather shoot themselves in the face before investing in training.


The talent does exist and is easy to find.

Problems these companies have are:

1. They are not really willing to compensate what it takes to attract the talent they really need. Most of these places start screaming bloody murder if you ask if they cover interview expenses or relocation, and trying to get them to actually state the range (which is no big deal to state if they really pay top rate since that's a huge recruiting advantage if true), is like pulling teeth. Which tells you they are not really paying top rate.

2. They aren't willing to do the work needed to recruit. This can include pushing hard from inside to make their company less unpleasant to work for, improving its reputation in industry, removing absurd contract requirements, treating workers fairly so Googling their name doesn't reveal the place is a nightmare and management are clueless. A lot of companies treat the employees as the enemy rather than an asset and in every case word gets out about this and good people stay away. A lot of companies are beseiged by politics so only the highly socially manipulative and dishonest do well. We see their representatives comment on boards and interviews on how it's impossible to find anyone, but going from their comment to a description of open positions that includes a clear statement of offered compensation is impossible. Companies are not reaching out to actual competent people and making offers. Instead they are hoping running an ad through a recruiter who posts ads on monster that often don't say what company it is, where it is, how much they pay, or clearly what it is they do. Why would anyone competent respond to these ads? Sure the desperate and incompetent and those that need visa sponsorship will respond, what about the people they really are trying to attract? Are qualified people really randomly searching monster these days? (Answer: no.)

3. Places with lots of mediocre and incompetent workers already in place simply can't attract talent at any price other than by the hour consultants who know they can leave at the end of the contract.


That's very true. We do actually do that with our production team. It's split into "ops", which is the day-to-day work and "engineering", which is the more complicated, more interesting work.

We hire junior people we think are smart into the ops roles and train them to eventually shift over into engineering. Some people can do it in a year or less, some a lot longer, and some just don't work out at all (or actually like doing the more routine work). Those people are definitely easier to find. We just talk to a lot of people and pick the ones we think are the smartest and will mesh the best and train them up on the rest.

In some cases, though, you just really need to get experienced people and can't wait the amount of time it would take to train.


What kind of experience are you looking for? Is it possible you are being too selective or misjudging the ability of experienced applicants?


That was definitely part of it, but we've changed our strategy a bit in an attempt to compensate. At first we were looking for people who had a lot of HPC-specific Linux experience. This, however, is a relatively small field because the cost-to-entry barriers are enormous if you're looking to do it at any great scale. That definitely restricts the set of potential candidates to a small subset of the set of "Linux people" (people from gov't labs, research universities, very large corporations, etc.)

Since we're willing to pay relocation expenses and buy people out of non-competes, we were doing a nation-wide search, but as a generalization, university and gov't employees are usually there because they like the relaxed environment and are willing to sacrifice some pay for it. (I have worked at both, so that generalization is drawn from only my experiences and those of my friends.)

So after getting few bites there, we rewrote the job description to just be very good with Linux in general, hoping to get a really smart person with skills that complement ours and then being able to train them in the HPC-specific stuff. That increased the number of applicants, but also dramatically decreased the quality-applicant to applicant ratio.

You're definitely right that I could be misjudging the ability of some of the applicants, but a great majority can't pass the Linux equivalent of FizzBuzz, even though they list 10+ years of experience.


What would be the Linux equivalent of FizzBuzz?


I'm thinking of it as a test for the simplest possible thing I can ask someone that will tell me immediately if they're worth talking to or not. I mentioned this in another post, the one I use that people bomb most often is: "I want to run this command on 100 files. Write me a for-loop in the shell of your choice to do it."


That's kind of terrifying that someone can't do that can say they are an SA. Did anyone you interviewed mention xargs as a BTW solution after presenting the for-do loop? I'm currently trying to figure out if one of the SA's at a current client actually does not know how to use tar, gzip, and scp, so I know where you are coming from, but still.

When you previously said "complicated applications in (python|perl|ruby)", did you mean someone who knew both front- and back-end? That implies someone who also knows Javascript, CSS, HTML, and a few databases, not to speak of one or more web frameworks, proxies, load balancers, etc.

When you say "debug and port C/C++ programs", did you mean someone who knows enough about Windows to port a program written there to run on Linux? Or someone who knows enough about BSD to port to Linux? Or someone who knows enough about [AIX|HP-UX|Solaris] to port to Linux? Port what, pure userland programs or stuff that messes around with ioctl calls? Debug from a hand-built in-memory circular log, core with no symbol table, using Dtrace, or just some judicious printfs or logging?

Depending upon how you specify what you want, you don't have to go far these days to stray a long distance from a "Linux SA" role, where even a "Senior Linux SA" job title won't cut it; you are moving closer to a Free Electron, and those people don't come cheap.


"Sorry, we don't have time for training."

"Sorry, we can't afford to send you to a week of training for $3000, you have to get this stuff done right now."

"Sorry, we're on a spending freeze."


They will pay whatever it takes* to get IT talent that does actually exist, at other companies.


There might be something you can do on your end to get better candidates. Usually problems like these stem from either bad (read: boring) work, or faulty recruitment practices.

Bad work can usually be overcome with cash and perks. Unless you're looking at the top 0.01% of devs, you can probably pay people enough that they will overlook bad work.

However, there probably isn't much you can do about changing bad work so double check to make sure that your recruitment practices are more or less in line. If you're hiring outside recruiters, make sure they aren't scaring away top talent. I've gotten emails from recruiters that try something new and crazy and throw up more red flags than they would with a standard "Hi, we would like to talk to you about a position we are trying to fill".

If you feel that both of these are in line, then there could very well be a shortage in your area and I wish you the best of luck.


Seriously? I'm a Linux sysadmin who prefers being able to fix problems from ssh. I mean, you can't copy and paste GUI commands. Am I that anomalous & out of touch?


The number of people who claim 10+ years experience with Linux and cannot write even the most basic of shell scripts is simply staggering and quite depressing.

(literally: "I want to run this command on 100 files. Write me a for-loop in the shell of your choice to do it.")


I dont think your for-loop example is not very fair. It took me about 5 mins of reading online to figure out how to do it right, but I certainly wouldnt be able to create a flawless working script offline because much of my work in Linux tech support is not of this nature. I spend more time collecting core dumps and analysing them, for example.

Don't you think you are prematurely excluding a lot of good candidates over a minor issue like this?


I've found there's two divergent breeds of linux sysadmin. They can be roughly divided along the lines of the ones that strongly need/want a GUI, and those that don't.

This is tied to their comfort in a terminal.


It's not entirely a problem with job applicants.

A lot of companies throw "Linux experience preferred" onto job descriptions and what they really means is that once in a blue moon you'll ssh into a Linux server to do a restart or deploy, but 99.9% of the time you'll have to use Windows. Maybe it's just my area, or bad luck on my part, but last time I was looking for a job that's what most "Linux experience preferred" requirements meant.

Does your job description explain exactly what type of Linux experience you want (shell scripting, coding, admin, ...)? Are you emphasizing that that they'll spend all/most of their time using Linux?

It's a real drag being a Linux user stuck on Windows, and if your job description gives the impression Linux is an afterthought, a lot of serious Linux people will pass it over.


Some (ok a lot) of comments from a non-application driver:

For a very long time (late 90's to late 2000's) almost all of the Chicago job postings specified local candidates only. While I was more than qualified and willing to relocate I had no desire to submit my resume when I knew it would be tossed aside once they got to the third line. I took a quick look on Dice and it seems that's no longer the case, but I and perhaps others have simply given up on Chicago due to this practice. I haven't casually browsed Chicago job listings for at least half decade because based on repeated past experiences it wasn't worth the mouse clicks.

Along those lines, I notice the pay isn't too hot. From what I understand Chicago is a very expensive, congested city and these gigs are offering the same as or lower than what I am currently making in a city that is less congested and has a lower cost of living. If a Chicago based company is truly willing to pay say 150/year (high, I know) for the best of the best then their postings need to trumpet it. I would take a chance and apply for a position that says "pays up to 150/year" knowing full well they probably won't pay that much, but unless it's THE dream job I don't have much desire to apply for something listed at 60/year and then have to haggle with the company to try to get them to pay me twice the posted rate. If a company is willing to pay top dollar then make it crystal clear because low salary job posting make me immediately click the back button, but something that pays well makes me want to read more - I won't see all those perks you offer if the posted salary is bad!

Another issue is separation of duties. While I can build packages and write shell scripts I, and probably 98% of others like me, can't write complex ruby/python/perl programs or debug C and C++ applications, let alone port them. As a sysadmin I can run traces for the programmers, help with things such as permissions and access issues, send them core files and dumps, ensure network communications, and check logs but actual C debugging is far beyond my skills. Without a vendor or in-house programmer I simply don't have the skills to solve the problem of why your daemon is crashing. I can monitor it, I can restart it, I can use my root access to assist the programmer/vendor, but I can't look at a core dump and figure out the whys. This is a fairly common gripe among sysadmins because it kinda boils down to us mumbling "damn programmers learn how to program because we aren't going to learn C so we can do your job for you, debug it yourself." The sysadmin+programmer all-in-one job posting is more common with smaller companies and if they are serious about growth they will need to break it into two distinct positions. Out of all the sysadmins I know there is only ONE guy that could do this and I doubt any amount of money or perks could get him to move away from his children and friends.

Cast a wide net when looking at people who are not currently employed. Some may be downsized because regardless of talent they have zero chance of keeping their position once management has made the decision to outsource/offshore/downsize/stop using technology X. I was let go from one gig when the company decided to cut their highest paid (and most talented) in a desperate bid to save money - they folded a year later. At another gig I quit because I wasn't too keen on the environment and getting a new (bad) boss pushed me over the edge. I understood that my skills are in high demand, finding something new wouldn't be a problem, and it wasn't worth the stress of searching while employed. I consider myself very lucky to be in a field where I can do this.

re: training.

Organizations do indeed need to step up and train when necessary. If they need an existing employee to learn something new they can't force them to study after hours, can't force the employee to buy classes, and can't forecast any type of timeline to implement the technologies they need. By paying to train an existing employee they can ensure they have instant access to a person with the skills they need, do not have to wait months/years to find a qualified person, AND get a huge ROI.

Generally a company's concerns with training are about the ROI and the risk of the person leaving after the training is complete. If a person is going to leave your company it's not the training that's going to make them do it, it's other factors. People aren't likely leave a job that has good pay and and good work environment after they are trained, but training a person who is underpaid and works in a bad environment may make them consider leaving with their new skill set. If this is the case, then the company must fix the environment and pay rates because these factors make everyone want to leave - trained or untrained.

If poaching is the concern then again, I feel this goes back to the pay, environment, opportunities, and perks that your company offers. Keep your people happy and they will have no reason to leave regardless of the pay and perks offered by other companies, period.

As for ROI, I'm not a numbers guy but I'm pretty sure paying $3k for training is a pretty good investment if implementing technology X is going to make or save the company 100x that amount. 100x is not an unrealistic number and could even be as big as 1000x or 5000x in a large organization. How much will the company lose while they attempt find a qualified candidate over the course of months or even years? Why not roll the dice and pay a few thousand dollars to train a bright, existing employee rather than trying to find a new person who may never come or won't be as qualified as their resume and interview make them out to be? If your business can't spare the relatively small amount to train an existing employee then you need to either re-check your ROI calculations, reconsider using technology X, or you have serious cash flow problems.

As a geek I consider annual training an essential perk because it allows me the opportunity to learn new things - and that's something we love to do by nature. Being forced to learn a new technology off-hours increases my job dissatisfaction and also increases the chance that my unstructured learning will cost the business more money down the road.

As far as I'm concerned training is frequently a win/win situation with minimal financial risk for the organization.


> In my last job (Dept. of Energy lab), I used to think it was hard to find people because the pay wasn't good enough, but in my current gig in the financial industry, we're willing to pay basically whatever it takes to get the best talent, but the pool of great candidates is very, very small.

I see lots of very high job requirements (postings at least) for comparatively low pay all the time. I am sure that is part of it. I would imagine at a financial institution (assuming postings actually mention decent pay) the HR gauntlet would turn away a fair share of well qualified candidates for various "check the box" reasons (missing a certificate of some kind, collegiate degree, etc, etc).


A lot of people claim to not be looking for "application drivers" yet ask for years of experience in multiple specific applications in their postings.

I've come to the conclusion that the only way to consistently work is to narrowly focus on a couple of specific, widely used applications, because general facility and a resume that covers a wide variety of areas for less than 2 years at a time are worthless. You get filtered out before the interview, even for pretty low paying positions.


> Eighty-one percent of the hiring managers surveyed for the report said that hiring Linux talent this year is a priority.

What is "Linux" in this context? Are they talking about doing actual kernel development or clicking around Gnome and using the GCC stack?


I assume they mean the typical Linux userland stack.


This article is very ambiguous on what kind of linux talent they see as being needed. Are they looking for sysadmins or developers? By linux do they mean using tools like gcc, make, etc? Or do they mean writing tools against a POSIX interface? Also, considering that basically every college uses a linux environment, I find it odd that they can't find devs who can work in linux.


I believe it's both.

Linux continues to grow so organizations need more admins to support more servers as well as more developers to write applications, scripts, and programs that run on those servers.

I don't see a glut of talent in either case.


This seems like a consequence of the rise of cloud computing.

My experience is probably biased by the fact that I've been working in the Seattle area for most of the last decade. I've worked with and hired excellent engineers with Linux backgrounds and also Windows backgrounds. Elsewhere it may be harder to find top Windows devs.

In the '90s, developers built their own Linux desktops and servers because there wasn't any IT infrastructure capable of supporting us. After the dot com crash and the nearly simultaneous death of big-iron Unix, Linux IT support started to become a thing. Medium and large companies staffed up on Linux sysadmins and support staff, so it was possible for devs to just code and remain fairly ignorant of the underlying OS - or at least confine their knowledge to the theoretical domain.

Now that you can just spin up server resources in the cloud, I can see devs with Linux experience having an edge over those without.


> This seems like a consequence of the rise of cloud computing

Could also be due to the rise of 'devops'. I see a fair number of places simply choosing to not hire (some I imagine may be having trouble hiring too) enough sysadmin staff, and instead just having developers do double duty and calling it 'devops'.


I wonder if it would make sense for companies looking for Linux talent to somehow target users of less 'user-friendly' distributions such as Gentoo or Arch. I don't know how well it would work, but I imagine that advertising on a project web page or, if allowed, making an announcement to a user's mailing list could be a way to find developers.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: