
Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job Posts - vthallam
https://letstalkalgorithms.com/analysis-of-2018-hacker-news-who-is-hiring-job-posts/
======
OisinMoran
I think there are some flaws in this analysis that can be confirmed when the
code is released.

I think the reason Scala seems surprisingly popular is that is was simple
string matching so a post that said "We are looking for someone who can build
scalable infrastructure" would count as one mention of "scala".

Could be wrong though.

Also see:
[https://news.ycombinator.com/reply?id=18725655&goto=item%3Fi...](https://news.ycombinator.com/reply?id=18725655&goto=item%3Fid%3D18725425%2318725655)

~~~
AznHisoka
This is why 90% of data science is the messy, grunt work of just simply
cleaning/normalizing the data. The analysis is the easy part.

~~~
screye
> grunt work

Honestly, it is an insult to call good data preprocessing skills grunt work.

It may not be as attractive, but requires a lot of understanding of the nature
of data and way in which patterns present themselves.

~~~
donaldknuth123
Eventually it will be split from actual Data Science work which is essentially
either Machine Learning or Statistics. Imagine making a master sculptor mine
his own Marble, it would be absolutely absurd similarly it would be silly to
make a PhD Machine Learning expert work on data cleaning. A massive waste of
talent on trivial work.

Edit: trivial isn't the right word, it's obviously an important step in the
process but quite different from what you are really paying a "Data
Scientist/ML Engineer/etc" to do.

~~~
jmalicki
It's quite different... I would hope the marble miner would know quite a bit
about geology, mining equipment, etc....

It's not that putting the sculptor on the task would be a waste, nearly as
much as that the sculptor doesn't possess any of the necessary skills.

How many people know how to find a source of marble, run a mine, _and_ sculpt
a great work of art? Roughly nil, but it's how many organizations seem to
hire.

------
vthallam
OP here. As some people were commenting, Scala and excel I guess have some
false positives which reflect the unusual popularity. I did try to limit the
search to exact word, but think that I overlooked that part. I will try fix
the code and update the skills chart soon. Thanks for pointing it out.

Edit: I fixed the false positives using a word boundary on regex
"\\\bword\\\b". The data now seems little sane, will keep looking for any
false positives. updated the data and regenerated the chart using new data.

~~~
ticmasta
I'd also expect some tough to catch exceptions for remote, as that's something
I always look at and despite the posting guidelines, posters can't stop
themselves from adding things like "sorry, no remote available at this time"
or "part-time remote might be an option for the right person in some roles".

I get that you're trying to be complete, but if it's not a truly remote
position, it's not remote. put nothing about remote in the position or at the
very most "REMOTE: NO". Working from home 10% of the time but needing to be in
a SF office the rest of the time is not a remote position; it's the
flexibility I'd expect from most jobs in this industry.

~~~
travisjungroth
Try searching craigslist for a shared apartment for a couple.

"Sorry, no couples!"

~~~
whyonearth
You have to get clever when searching CL. Plenty of places that allow cats
don't click the "Cats OK" box. Instead of using that search filter, include
the terms -"no cats" -"no pets". In your case, -"no couples". My CL searches
to filter the fake 1BRs get pretty crazy (-roommate -"room for rent" -jr
-junior -studio, etc...) but their search engine handles complex queries well.

------
dotdi
Take that all you people crying "Scala is dead" after Java 8+ comes out. /s

Anyway, I've personally seen (and interviewed for) jobs that were advertising
Scala (or plans to move to Scala, or that Scala experience is a plus) but
actually meant there was one proof-of-concept project in Scala, 18 months ago,
nobody ever touches it nowadays, the person who wrote it is long gone and by
the way, our legacy JDK5 application is on fire, do you mind fixing that for
the rest of your life?

~~~
markhalonen
I suspect there's false positives from "scalability" and "scalable"...

~~~
dotdi
I hope OP took that into account. Seems like pretty low hanging fruit when
cleaning your data.

~~~
quickthrower2
Yep just split on whitespace

~~~
imron
Or preferably, just split on word boundaries.

------
mountaineer
Cool to see that other folks find this data interesting. I've been analyzing
languages/frameworks/skills in the "Who is Hiring" posts for a few years
([https://www.hntrends.com/](https://www.hntrends.com/)) as well.

I know it's just one part of the analysis, but the skills list appears to be
be pretty far off of what I've been seeing. React gets over 200 a month by
itself. Are you capturing all pages of the postings in each month's thread?
Here's the data (counts) I have through November, broken down by month and
term -
[https://www.hntrends.com/data/data-20181101.js](https://www.hntrends.com/data/data-20181101.js)

------
eddd
The fact that Rust and Python are quite high in the rankings, makes me hopeful
about the future of programming again. The fact that remote work is not
growing is a bit concerning, but if people from the valley want to overpay -
it is none of my business. I hope eventually the market will correct it.

~~~
ptd
If you don’t mind sharing, why do you find the lack of growth in remote work
concerning?

~~~
TarpitCarnivore
Not OC, but for me it's just a real wonder why it's not better embraced at
this stage of the game. We have all the tools to accommodate collaboration
within remote teams and in (most) places the broadband to handle it. Add to
this the continued funneling of companies into these metro areas where COL is
high (NYC, SF, Seattle) and thus people may find themselves being forced into
higher commute times just to attain a better COL situation.

I personally am commuting close to 2 hours each way, so 4 hours total, because
the job market is much stronger in NYC then my immediate (30-45 minute) area.
If a job is open around here the salaries are almost 30-40% lower than NYC
despite our COL still being high.

~~~
ptd
We have tools to accommodate collaboration but I believe something gets lost
when you remove those water cooler conversations and physical interaction.

In regards to your commute, wouldn't a 40% lower salary be worth 4 extra hours
everyday? I guess it depends on the math but unless you're being compensated
for the commute(in which case it's just work on the train/metro) that seems
like an incredible sacrifice for a bigger paycheck. At what point is your time
worth it?

~~~
TarpitCarnivore
I think "water cooler" talk is over stated because many companies will have
different channels to discuss these kinds of things. You no doubt loose the
face to face human interaction, but there are other ways to account for this.
Getting out and socializing with people via hobbies can help greatly here.

As for the commute, I've worked out a flexible work schedule that lets me be
home a few days but as I stated I'm in a higher COL state (NJ). So a 30-40%
reduction in pay ends up being a fairly big change in QOL. I've learned to
make use of the time on the train by engaging in things I enjoy (video games,
podcasts) which may be harder at home. Moving closer also doesn't help because
A. it gets even more expensive B. QOL drops due to higher density areas which
makes it harder to ride bikes, garden, etc.

We've accepted the choice we make to be where we are because of what we get
from our life outside of work, but it doesn't mean I can't hope for better
remote possibilities.

~~~
shaklee3
It's not overstated. The only people I've heard say this are the ones that
prefer remote work. You're definitely more productive around where the action
is happening than in a house with kids screaming.

~~~
kbsletten
First off, forgive me if I don't take your word for it, but I just saw an
article recently saying that remote workers are more productive on average.
Whether that study was just an anomaly or there's some other explanation like
"only productive programmers can manage working remotely", it would hint that
your hot take represents the world as you imagine it, not necessarily the
world as it is.

Secondly there's something like a "no true Scotsman" vibe about the
implication that you can't trust people who prefer remote work to comment on
their productivity like you can people who are onsite. That may not be the
right fallacy, but there's a fallacy in there somewhere.

Thirdly, nobody should be working with kids crying. God invented doors for
just this reason. If you can't make a quieter space at home than you can at
work, you're either pulling in some serious perks on the job with your
private, sound insulated office or you don't have the fundamental amenities to
work at home, it's not the nature of working remote, it's showing up to work
unprepared that's holding you back.

------
lsiunsuex
So the next report / analysis needs to be:

Of everyone who asked for work or applied for work via these posts - how many
secured a position?

I've long given up on these posts trying to find freelance work or other
permanent work. There's to much competition (here) to stand out from the flood
of replies.

~~~
jedberg
Anecdotally, the one time I posted a job, I had about 80 replies for a remote
work junior engineering job where I specified the salary at $60k/yr in the
post (this was in 2015). Of those 80, about 10 were promising and I probably
would have made an offer to at least one or two if we hadn't shifted
priorities and stopped the hiring process.

------
vanrysss
Check your map, it looks like you may be matching "Remote, or XXX" with the
town of Remote, Or (Oregon).

------
sytelus
Some highlights I saw -

* SF is center of the tech world. Next biggest is just around half of its size. (not withstanding HN bias).

* Approx 25% jobs allow visa sponsorship

* HTML, Python, .Net overwhelmingly dominates everything else.

* Reduced interest in databases, SQL, Obj-C, Java

* No TensorFlow or PyTorch in demand

* 90% jobs in development/software engineering, 10% in management and misc.

~~~
dmode
It probably understates the SF market, TBH. As some of the largest employers
in SV who hire en masse rarely post jobs in HN. Such as Google, Facebook,
Apple, Salesforce, Uber etc. Also, Google, FB, Uber, Airbnb and others have a
boot camp system, where job postings do not correlate to hiring since one
posting is used to hire an entire bootcamp class

------
tinyvm
Wow , is .NET that popular in startups ?

I always assumed it was used heavily in enterprise but never that much it's
would somehow , it's also a popular stack in the Silicon Valley and NY. That
said London and UK are well know to be MS Stack users.

Can some Engineers from SF or NY or voice their opinions on this ?

~~~
zephyrfalcon
When you look at the job postings in your browser, and do a simple string
search for ".net" (so not a regex search), it doesn't show up all that often.
E.g. if I look at this month's posts (4 pages), I see it 31 times. By
contrast, Python shows up 251 times. I suspect the posts were searched with a
regex like /.net/i without escaping the ".".

------
xipho
The graph of top skills doesn't remotely correspond to the keyword values
(actual data). For example rails/ruby is nowhere on the graph, when both score
over 50.

------
corn_dog
As much as I like Scala I find that hard to believe. I actually look for Scala
in the job listings and I never felt it came up that much, compared to say
Python or Javascript. A dumb search on the page for scala mostly turns up
scalable ...

~~~
twoquestions
Ditto with Rust, most of the matches are part of the word "trust" when using
ctrl+f in Chrome.

~~~
vram22
This is one reason why we need regexes (to name just one point) in browser and
other search. I've said before in other places that Gmail could benefit from
an SQL-like query language, for example. So could many other web and non-web
apps. Google search too. One potential issue is excessive load on the servers
if it gets used a lot, though.

------
UltimateFloofy
Even though react and angular and vue are different frameworks, they are all
modern JS libraries and I would lump that data together instead of keeping
javascript so fragmented in the data set.

------
pbhowmic
A bit surprised, pleasantly I must admit, to see Django keep pace with Rails.
I have never been quite able to explain the startup world's love affair with
Rails. Nothing wrong with it but the money question is why?

~~~
albertgoeswoof
Everything in rails is figured out already, you don’t have to pick which ORM
or library to choose, and then find out half way down the line it’s not
working as expected. It just works consistently, reliably. There’s no other
reason to use another server side framework if you’re creating. CRUD app or
API (unless you’re specifically comfy in something else).

~~~
chpmrc
Doesn't answer parent's question. Django and Rails are both web frameworks.

~~~
lowercased
There's an implicit position that those qualities ("just works", etc) are more
prevalent in rails than django.

------
Southland

        "Engineering Manager": 271,
        "Principal Software Engineer": 25,
        "Product Designer": 243,
        "Product Manager": 425,
        "Program Manager": 35,
        "QA": 353,
        "Senior Software Engineer": 818,
        "Software Engineer": 2975,
        "Staff Software Engineer": 10
    

I think this section should have been ordered by volume and cleaned up a
little bit. 'Software Engineer' and 'Staff Software Engineer' are the same.

~~~
cialowicz
"Staff Software Engineer" is typically the level above senior, and below
principal. It's not commonly used at smaller companies. I agree about the
ordering/ranking.

------
dvaughan
You could make the argument that Go is underrepresented as the match is only
for golang and not go.

~~~
hu3
Yep. A quick glance at December's Who is Hiring shows at least 2 jobs that
aren't included by searching for Golang only:

[https://news.ycombinator.com/item?id=18662978](https://news.ycombinator.com/item?id=18662978)

[https://news.ycombinator.com/item?id=18590124](https://news.ycombinator.com/item?id=18590124)

There's probably more in December alone that the regex missed.

------
sephlietz
I wonder if some of the "scala" hits are actually from the word "scalable"...

~~~
typon
Really hope the OP did full word searches...otherwise he would be double
counting "react" and "reactjs" as well

------
howard941
I'm not sure where embedded or firmware would fit - developing for it's a
skill but it's not a language or framework - but I'm curious what % of posts
taps the skill. Going by memory it feels like roughly the same number of
firmware positions are on there every month, and it's the same guys looking to
fill them.

------
j_m_b
What no Clojure? I find that hard to believe.

~~~
fulafel
There are Clojure jobs posted every month, just an omission in the list of
language regexps s/he searched for.

------
ptd
Scala as the top skill for 2018. I thought it would be high, but I’m surprised
to see it at number 1...

~~~
RhodesianHunter
Keep in mind, HN will mostly have startup jobs, so you're more likely to see
languages like Scala represented here than in the job market at large, where I
would expect Java, PHP, C++ etc. To dominate.

------
cauchyschwartz
So if I'm understanding, raw text is fed into the geotext python library,
which parses out geographical locations and adds to the count.

I'm very skeptical that there are 18 new jobs on the island of Jinja, and that
instead the analysis is picking up the Jinja templating language

Edit: same thing for "cambridge" in the same location as Jinja Island.
Obviously probably refers to Cambridge MA, where Tufts/Harvard/MIT are all
nearby, so tons of companies will be there.

------
josh_carterPDX
This is great, but I would have liked to see more data around how many jobs
are entry level, management, executive, etc. Thanks for this! Really cool.

------
magissima
I was curious what the number would be for Washington, DC, but it seems that
the scraper couldn't distinguish between DC and Washington state.

------
startupdiscuss
So one simple way to solve the scala and excel issue is just to match for the
space at the end or punctuation. So search for "scala " and "scala." instead
of "scala". Same for "excel " and "excel." In fact you could just combine them
all into word + end where end can be [" ", ".", "!", "\n" etc]

------
GrumpyNl
Page doesn't load properly in chrome, like the rendering doesn't get finished,
its all blurred. Loads fine in firefox.

------
jimijazz
Good work. Besides fixing the false positives others have suggested, I would
really like to see the total numer of posts per category in comparison to the
total number of posts per month, i.e. a relative quantity to better explore
wether there is a trend or not.

------
dmode
I find it hard to believe only 5% jobs are willing to sponsor visa considering
most engineering teams are full of H1 visa holders. I think in this case it
may be worth looking at the negative text as the default assumption is that
the job sponsors visa transfer

~~~
shaklee3
You're comparing an HN post, which is typically smaller companies, to an
overall statistics where large companies hire from body shops.

------
musikele
I live in Italy, and the chart shows 24 jobs openings for Napoli on Hacker
News (64 in Rome...). I checked and I could not find a single text match for
the words Italy, Rome, Napoli, Neaples, Naples, etc.

Can you better express how you extracted the location data?

------
niutech
Have you seen this interactive map of HN job offers?
[https://whoishiring.io/search/0/0/2?source=hn](https://whoishiring.io/search/0/0/2?source=hn)

------
mrath
Strange Sydney has so many less jobs in HN than Melbourne. Whereas I think in
actual job count it is opposite. This could be because of more startup like
companies in Mellbourne which publish jobs on HN

~~~
nl
Melbourne, Florida has a surprising number of people advertising on HN.

Also I don't think this is de-duplicating, and many of the Melbourne FL ads
are repeated.

------
cgdub
Scala seems suspiciously high compared to the data on Hacker News Hiring
Trends: [https://www.hntrends.com/](https://www.hntrends.com/)

------
simonebrunozzi
Wow, 2nd most common skill is “excel”. I would have never guessed.

~~~
OisinMoran
I was also quite confused.

I think it's so high because "excel" is also a verb so a lot of job posts will
say things like "We are looking for people who excel at what they do" and this
was counted in the analysis.

Changing the search to be case sensitive might help here.

~~~
Kurtz79
Myabe the regex pattern is checking for the pattern but not as a separte word?

"scalable solution"

"excellent benefits"

------
LarryDarrell
It's not looking good for me, a remote C# developer.

~~~
Topgamer7
If you look in the data .net is up there in numbers. Seems like the author is
just doing straight regex matches not taking into account matches embedded in
other words. Scala and rust both are exceptionally high, which I wouldn't
expect.

------
kaustyap
It shows 18 jobs in Rennes, France. But couldn't find a single job post.
Wonder which word could produce false positive for "Rennes"?

------
edpichler
Scala is the most popular skill. Is there something I don't know about Scala
and the reason for its success?

PS: I use Kotlin/Java for web development.

~~~
PhasmaFelis
The OP didn't correctly exclude false positives, so anything with "scalable"
was registering for "Scala". It's fixed now.

------
chpmrc
There is "Oregon" on the map (with 54 jobs) but even if it was referring to
Oregon City that's way off the actual location.

------
aprdm
Weird to not see any devops role in your analysis.

------
g45y45
TIL Jinja is a place, its off the coast of Africa.

------
someguydave
So does hacker news consist of “engineering managers” who know html or is that
simply who everyone wants to hire?

------
sidcool
I am so thrilled to see Rust at #2.

------
zachguo
Sloppy regex.

------
jrochkind1
I think it's become indisputable that python seriously beats ruby in the job
market.

It is sad to me.

------
openbasic
Scala, huh?

~~~
sabas123
And julia over c++???

~~~
Kurtz79
This honestly seems just a mistake.

Skimming a few of the posts, there are maybe 15+ references to C++ for post,
and 1 for Julia at most.

The detection algorithm seems to be a bit iffy.

~~~
inetknght
I'd be willing to bet that the regex parser doesn't like the pluses in C++

~~~
Mikulas0
Definitely. I might be blind, but I don't see C++ at all in the list of
skills.

~~~
Kurtz79
It has changed compared to the initial version.

Pretty sure that a string like "c\\+\\+" was there before.

It seems that in the new version the old patterns have been commented out, the
search method was changed and c++ removed altogheter:

[https://github.com/venkateshthallam/hn-whoishiring-
analysis/...](https://github.com/venkateshthallam/hn-whoishiring-
analysis/blob/master/src/main.go)

------
quickthrower2
One of the maps shows a disposed event handler error on my iPhone 6 w. Safari

------
luord
The job posts for react are five times those of javascript :thinking:

------
csdreamer7
As a Rails Dev, why does Python score so high? Is Django or scraping that
important?

I know Python can be used in practically anything, but what do you keep seeing
companies ask for that is a recurring need with Python?

~~~
syndacks
I think you hit the nail on the head, "Python can be used in practically
anything."

Our Data Scientist was able to pickup Django pretty quickly because of his
experience with Python. And yea, actually someone at my company who was doing
a bit of scraping on the side was able to pick up Django pretty easily as
well. And last night, I chatted with my buddy who works at a big tech company
and they use Python for database intrusion analysis.

I don't know the Ruby ecosystem very well, but it seems the biggest
application of Ruby is Rails. I'm curious to hear what other ways people use
Ruby, so please share.

~~~
csdreamer7
Well I use Ruby also for Chef. I use it to keep my servers perfectly perfect
documented and tested in code. Since I am a solo founder I can stay away for
weeks from the actual configuration. When I forget something or something
breaks I look at my Chef recipe. It takes a little longer to get setup

I also use Chef to setup my local environment which is wonderful. When you are
learning how to really dig deep into your text editor and writing your own
custom snippets (and commands, and keybindings) for react. You don't how
important it is to have your local env, your dotfiles and atom config
automated and can be synced up with a single command no matter what computer
you are on.

This, with private git repos and syncthing is a wonderful thing.

------
dr_wolf
Why is Montreal missing from the map?

------
rplnt
Why the reverse scroll on the map?

