
a16z Programming Test - ghosh
https://www.hackerrank.com/tests/530697662bd88
======
dang
HN policy for years has been to allow two kinds of job ad: (1) job posts by YC
startups, and (2) the Who's Hiring threads. We kill job ads that are neither
of these.

(Incidentally, PG designed the /jobs posts to be the one privilege that YC
startups get on HN. Other than that, the site policy is to treat them the same
way as everybody else.)

It's not uncommon for posts to be in a grey area between job ad and story. A
programming test for an interview process doesn't seem all that grey, though,
so I've been killing these. Sorry to be a hard-ass about this, but I don't
want to be accused of enforcing the policy erratically.

Edit: I'll unkill the post (but otherwise leave it demoted) so the discussion
can remain open.

Edit 2: The argument has been put to me that this is newsworthy because A16Z
are doing it. Since that's corroborated by external sources (e.g. Twitter),
and the post has gotten a lot of upvotes even while it was buried, that's
reasonable—so we'll restore it as news rather than as an ad. Besides which, HN
is one of the few places where a programming test counts as interesting.

~~~
ig1
Can an actual list of policies be posted somewhere ? - I don't ever recall
there _ever_ being a discussion of such a policy before. Certainly there have
been highly voted non-HN job posts in the past (e.g. Khan Academy)

~~~
dang
I'm reluctant to mess with the informal way that HN has been run over the
years, so there isn't likely to be a complete list of rules. We do intend to
augment the HN guidelines, though—just not any time soon. The new team has to
get settled in first.

In the meantime, I've been posting feedback comments most places that issues
like this come up. You may need to look to the bottom of the threads to see
them, because we usually mark them off-topic.

And you can always get answers to questions by emailing hn@ycombinator.com.

~~~
ig1
The informal way things worked was by the community not upvoting posts or in
some cases flagging them; if the new HN position is that moderators will
manually intervene to enforce these informal policies then that's the major
change. Publishing the policies is minor in comparison.

~~~
dang
The informal way things worked before is exactly the same as how they work
now. There is no "new HN position". The only difference is that we're trying
to be more transparent, and that's not a policy change—it's that we have more
time to put in to answering questions, and users made clear it was something
they wanted.

------
abalone
I wonder if this could act as a negative filter for good programmers. Who
worth their salt is going to invest 3 hours of their time just to submit a
resume? Wouldn't you want to talk to the employer first to see if they're a
good fit?

In a tight labor market, people willing to do this are more desperate than
others. So it's going to filter for desperate developers who passed a 3 hour
basic skills test.

Old fashioned prior experience and education is probably still a better first
filter. This would probably make more sense specifically for junior hires
right out of school or with no experience.

~~~
balajis
Well, empirically we're getting quite a lot of people attempting it! :) Three
hours is pretty lenient (you could blaze through it in less time), though the
last question is challenging. We think the questions are kind of fun as well.

Without something like this we'd just have 100s-1000s of unstructured resumes
in different formats to rank against each other. This process is more fair and
(IMO) allows someone without a college degree, or from outside the US, to
compete on a level playing field.

~~~
7Figures2Commas
> Well, empirically we're getting quite a lot of people attempting it!

> Without something like this we'd just have 100s-1000s of unstructured
> resumes in different formats to rank against each other.

So instead of hundreds or thousands of unstructured resumes in different
formats, you have hundreds or thousands of completed programming tests. Even
assuming that you automatically filter out submissions that fail testing,
given that there are a lot of ways to solve a problem, you'd still be stuck
evaluating potentially hundreds of correct submissions.

More importantly, a test doesn't seem to be aligned with your top priority:

> ...but the most important requirement is significant independent programming
> experience as demonstrated by your GitHub account, personal projects,
> academic publications, or startup success.

Are you assuming that all of the folks most accomplished in the real world are
going to complete your programming test successfully?

Since you ultimately seem to be interested in folks who can build dashboards,
I would think you'd be far better off asking candidates to build a dashboard
prototype using sample data you provide. There are lot of really smart
developers out there with advanced degrees who could solve programming quizzes
all day long but couldn't design and implement a useful dashboard application.

~~~
dlss
> You'd still be stuck evaluating potentially hundreds of correct submissions.

I'm guessing the MVP of hackerrank was the web code editor +
lint/findbugs/whatever for filtering. A lot of people have spent a lot of time
writing open source projects that convert code to metrics, so making a
reasonably prioritized list is probably fairly easy.

> There are lot of really smart developers out there with advanced degrees who
> could solve programming quizzes all day long but couldn't design and
> implement a useful dashboard application.

Agree!

I think this is a great critique of HackerRank as a business idea -- while
they are new, it's just a hurdle that weeds the busier coders out of your
hiring pool... if they become established there will be books on cramming for
it, and will no longer separate the wheat from the chaff.

(ie
[https://web.archive.org/web/20130114163457/https://raganwald...](https://web.archive.org/web/20130114163457/https://raganwald.posterous.com/i-dont-
hire-unlucky-people) meets
[http://en.wikipedia.org/wiki/Goodhart's_law](http://en.wikipedia.org/wiki/Goodhart's_law))

~~~
lifeisstillgood
To be honest I think it's the other way round - if you can do programming
tests all day long you will succeed at most coding jobs. However there are a
lot of developers who can't tackle an interview that with a nights sleep and
no pressure can easily code a intuitive and effective architecture.

Sorry - programming tests raise far more false negatives than false positives
(and this from a person who considers himself a false negative in most tests)

~~~
dlss
I'm not sure what kind of coding work you do, but I assure you the vast
majority of web development work has little resemblance to coding questions.
You are almost never designing an algorithm: that was already done for you by
whoever designed the library you are using.

Majority of the tasks IME:

\- code reading & refactoring

\- figuring out why two libraries aren't playing nice

\- bug triangulation

\- optimization ( _not_ the algorithmic type, but rather the fix-the-auto-
generated-SQL-query / frontend asset packing kind)

\- repetitive/simple/obvious bits of jQuery

Maybe there's some kind of online test that could exercise those skills, but
if you ask me the main performance changers for modern web development are,
they're: really knowing the stack (problem anticipation, what to use, how to
use the various tools to quickly get search keywords), google fu, and most
importantly having done it a couple hundred times (muscle memory).

~~~
lifeisstillgood
That's terribly negative. If you are doing agency work maybe (same website,
same functionality, just different skin) but that vast vast open plains of
http-based development are around and going to be around joining parts of the
business, public world together in ways they do not know or imagine.

Yes, binary trees are pretty much coded for you, and you would be a fool to
roll your own hashing algorithm for production, but thats still not a reason
to keep your head down.

The next decade is going to be fun. Trust me

~~~
dlss
I don't understand this response. To be clear, I think web development is fun.
I very much enjoy it and don't resonate with "keeping my head down".

I guess to reiterate my point: once you've done web development for a couple
years, coding most websites is less about solving new problems and more about
applying known solutions.

To reiterate how this point relates to the question of coding interviews: in
as much as they don't test for lots of known solutions, and the ability to
apply known solutions quickly, they aren't filtering for productive coders.

How does your response relate to the points at hand?

~~~
lifeisstillgood
but web development is not about "making new websites"

Think micro-services supplying REST APIs over HTTP - that's going to be same-y
after a few years yes, but _behind_ that is a world of business apps,
integration challenges, complex algorithms and calculations - sorry I just see
"web development" as "anything that has of will have an http access point"
which I mentally count as "pretty much everything from here on in"

Programming tests do generally test for ability to apply known solutions
quickly - it's about all they do do. You just have to know the known solutions
(oh look calculate primes).

As for finding productive coders - well the amount of false negatives is
large, and the false positive rate is probably higher (plenty of people who
can waltz through these tests i find fairly poisonous to morale) so I am not a
big fan - but if you need to make a first cut this is a good start

------
numair
This doesn't quite make sense to me. Or, it makes sense, but not for the
reasons presented on the surface.

If you're Andreesen Horowitz, you have access to a nearly-unlimited supply of
capital. You also have incredible access to information and dealflow
throughout the industry. In situations in which you are not capital
constrained, the most efficient way to establish a workforce is to acquire it
from somewhere else; that's why acqui-hires make so much sense, because you've
basically had someone else do the hard (and often completely unpredictable)
work of figuring out if people can work together in a successful manner
(which, I should note, is different from determination of whether the thing
they're working on is successful). If I were A16Z, I would arbitrage my
information advantage to secure a high quality engineering team by acquiring a
company that had smart people but a lame product; going through a traditional
hiring process would be a waste of my time (and since everyone is throwing
money at my firm, it would also be a waste of time that could be better spent
on figuring out where to deploy my capital).

A16Z is rich enough to buy people, if it really wants people, for internal use
(whether it is for their own company needs, or for some idea that they want to
incubate inside of the firm). And even if they don't want to buy a whole set
of people, they have enough wisdom and experience to know that you want to
hire within a cohort of successful people rather than randoms (to see this in
practice, take a look at Don Melton's post about Eazel alumni ending up at
Apple on the front page right now). This tells me that their programming test
has more to do with some sort of evaluation of this Hacker Rank service ("tell
you what, we will run our own test on the service to see what happens!") or
some other sort of analytical adventure.

Or, you know, maybe it is really a programming test after all. Maybe they are
hoping to find some sort of diamond-in-the-rough engineers who are socially
and career-wise disconnected from an existing network. Some sort of "black
swan programmer" or something. But I highly doubt it.

So basically, if you go through this test, you're helping A16Z evaluate
something; that something, however, probably isn't you.

~~~
gcb0
Rich people are cheap.

Why acquihire someone who knows how much they can ask if you can pay close to
nothing to that test website (it's a genetic service) and get a clueless buy
competent nerd that will be glad to work alone on a project for peanuts?

That's why we're poor and they have "unlimited capital"

~~~
numair
There are a lot of stupid rich people in this world, it's true. It's also
somewhat of a dirty little secret that some of Silicon Valley's most
successful VCs are stupid rich people.

The team at Andreesen Horowitz, however, are not among these stupid rich
people. If anything, they have disrupted the standard operating procedure
among their peers by outsmarting them, and the nature of how they have
outsmarted their peers is strongly connected to an appetite for attaching high
valuations to small groups of really smart people. Their organizational M.O.
has been the exact opposite of bargain-bin talent hunting, which is why this
whole Hacker Rank thing comes off so weird to me.

I want to be clear that I don't think A16Z is up to anything evil. Just
something sneaky (which is fun!). These are smart people, and for them to
publicly engage with the hacker community in this manner is indicative of some
sort of really smart hypothesis being tested.

------
xedarius
An entire page detailing what you want from me, coupled with a 180 minute
test. Yet in all of this there are only 4 bullet point sentences describing
what you're offering me. That deal is not for me.

~~~
eternalban
They are establishing the tone of the relationship .. [& edit: it is a great
filter for the personality type they want.]

------
_pmf_
> machine learning (scatterplots & histograms, logistic/linear regression,
> R/Matlab)

So, someone taking Statistics 101 is considered to know machine learning in
the startup circus? (Not that this would really surprise me, given the
tendency to appropriate terms that had a certain meaning for purposes that
have only a passing similarity to the original meaning.)

~~~
balajis
We could certainly rattle off more terms :) Kernel density estimation? Deep
learning? Semi-supervised learning?

But it's amazing how far you can get with some simple scatterplots, histograms
and linear/logistic regressions.

a) Most of the time, more data beats better algorithms:
[http://anand.typepad.com/datawocky/2008/03/more-data-
usual.h...](http://anand.typepad.com/datawocky/2008/03/more-data-usual.html)

b) Even those seemingly simple things get complicated when you have a lot of
data: [http://komarix.org/ac/lr](http://komarix.org/ac/lr)

------
tpae
I seriously hope this doesn't become a norm. I hate timed tests.

~~~
7Figures2Commas
As an employer, I would seriously hope this _does_ become the norm.

If some employers want to implement recruiting processes that require
developers to complete tasks that don't reflect real-world development, it
makes it that much easier for other employers to differentiate themselves and
attract competent developers who aren't interested or desperate enough to play
these games.

Frankly, in today's market I feel sorry for any experienced developer who is
going to complete a "programming test" in which referencing a website or book
other than language documentation is forbidden. I reference websites and books
all the time, and I don't know a single developer who doesn't.

~~~
dlss
This was too awesome a comment to pass up, so I ended up browsing your thread
history / googling you. I found a lot of great posts as well as the hilarious
[https://news.ycombinator.com/item?id=5967671..](https://news.ycombinator.com/item?id=5967671..).
but sadly no real name / business name.

What company do you run? Are you in the Bay Area? Can I buy you coffee? I have
a feeling you could teach me things that are both true and that hackernews
would downvote... And that there's a reasonable chance I could return the
favor.

Background so you can filter me more easily: I'm a very good programmer (7
years experience coding on small teams made of up jQuery core, rails core, and
MIT AI Lab alums. Several offers from YC companies. 4 papers & patents.
Currently doing machine learning for the Scripps Institute of Oceanography).
I'm also an avid reader of psychology (cognitive biases mostly), marketing
(Joe Sugarman is amazing), and business books (I'm a raging Munger and Deming
fan).

So I'm guessing if we did meet for coffee you at least wouldn't find it
boring.

Email me? dls@lithp.org

------
nilsbunger
Cool, but... Why is A16z building an engineering team?

~~~
Oculus
My guess is to have an in-house team that can help out the companies they
fund. I know that Netflix's cloud architect, Adrian Cockcroft, left to join a
venture capital firm to advise on scaling issues for companies that firm
funded[0].

0 - [http://gigaom.com/2014/01/07/netflixs-cloud-architect-
adrian...](http://gigaom.com/2014/01/07/netflixs-cloud-architect-adrian-
cockcroft-is-leaving-to-join-battery-ventures/)

~~~
pmarca
"My guess is to have an in-house team that can help out the companies they
fund."

Nope.

~~~
seizethecheese
I'm really curious... a hint?

------
riffraff
I'd take the test to see what's inside but... 3 hours? Can I get a t-shirt if
I complete it at least? :)

------
buro9
Does this mean Mattermark [https://mattermark.com/](https://mattermark.com/)
have started a tech arms war amongst the larger VC firms to create their own
toolsets to differentiate their deal flow from others?

Mattermark has gone some way to provide proof that data can make a difference,
at least at highlighting opportunities that might not have been visible
before. I wondered whether this would result in herding investors towards some
companies, which means differentation and opportunity comes from spotting
things the others miss... finding the outliers that Mattermark miss.

Interesting that the test is "make a dashboard", well into the standard
skillset for most analyst programmers who work for financial institutions (I
built a lot of this stuff for the finance sector in London).

------
joncooper
Programmer == Labor; don't ever forget it.

The overwhelmingly largest share of returns accrue to capital. It's a good
idea to resist being cast in the role of simply "programmer" over time, for
this reason.

------
elnate
Doesn't accept my (NZ) mobile number as valid :(

~~~
rmmw
You can also email test-help AT a16z DOT com and we can send you a direct link
(where you won't need to receive credentials by phone)

~~~
pjreddie
I love that people do this. Do you really thing someone is going to write a
web scraper and NOT include obvious things like converting AT -> @ and DOT com
-> .com?

Also, who even worries about spam any more? I haven't seen actual spam in my
inbox in years.

~~~
cflee
Should we just go "AT AT AT" or something instead? Or perhaps adapt from
Mandarin speakers, who literally call the @ sign "the little mouse" ;)

Spam remains an issue if for whatever reason you're still using
rulesets/blacklists/greylisting and the like, instead of Bayesian filtering.

------
lifeisstillgood
My problem is I like taking these tests to find out what I don't know (quite a
lot usually). To be fair though code reviews usually tell me more than tests
these days.

The constraint that gets me is the honor code - I cannot use an IDE with code
completion?! but that leaves nano or notepad ...

~~~
nostromo
I think you misread that.

"You _may_ use reference language documentation or use an IDE that has code
completion features. "

~~~
lifeisstillgood
oooooooohhhhhhhhh

what's the first thing your teachers said ? Read the instructions carefully
:-)

------
thomaslangston
The test's restriction on consulting material from websites and books,
excluding the official reference documentation boggles my mind. Unofficial 3rd
party documentation is standard kit for a developer, especially when using
community driven programming languages.

------
jqm
I didn't upload a resume (just a note saying I didn't have one) but did take
the test.

Hopefully they send some feedback... I'm not looking for a job, just wanted to
know how I would do if I were:)

Besides, I love programming challenges.

~~~
jqm
I was actually disappointed when I didn't get feedback right away after doing
the test.

Something to be aware of if you are considering doing this... it looks like it
really is for employment and not a challenge.

I guess one of the first question "Are you ready to go to work in Menlo Park
CA" (No) should have probably clued me in:)

Oh well... they will "be in contact". Right.

~~~
jqm
And... my suspicions were confirmed. I still have no idea how I did on the
test and they have provided no feedback.

So, unless something changes, which I don't predict, it was a waste of time I
wouldn't recommend doing the test nor applying to work at a company like this.

------
6thSigma
Is anyone in their programming job given several distinct questions and only 3
hours to solve all of them?

A more realistic programming test would be a single, complicated problem with
an "end of day" time restraint.

------
dpweb
Have you guys thought about possibly employing smarterer.com rankings? I took
your test, I think smarterer may yield a better evaluation, although you can't
write code on a clock.

------
Oculus
1:30 am on the night before a physics test - why must this challenge be so
tempting! There goes that studying :)

------
minikomi
Is that last question typical of interviews in the States? What is it
selecting for? Aptitude in Maths?

------
ojbyrne
Random question - how are "scatterplots and histograms" machine learning?

~~~
graycat
This is a very difficult question and, thus, illustrates the transcendent
brilliance and astoundingly "deep domain knowledge" of A16Z. An answer is
available only on the bleeding edge of the most brilliant insights of research
in machine learning. Ordinary people can't expect to understand such depth!
All bow down to the incandescent brilliance of A16Z -- after all, one of their
founders used C++ to write a GUI application to read a text markup language
and display the formatted results! Maybe this was the first time, I mean after
Word, TeX, and some others?

------
argonaut
Is this just for full-timers, or will there ever be an option for internships?

------
peapod91
Will this be an ongoing opportunity or is there a deadline to submit this by?

~~~
balajis
It's an ongoing opportunity, but we will be hiring the first set of people
soon. So it will be to your advantage to apply sooner rather than later.

------
jheriko
aside from the fact that tests generally don't work i find this hilarious.

"the most important requirement is significant independent programming
experience as demonstrated by your GitHub account"

a great way to find newbie hipsters and not experienced and seasoned devs. or
at least limiting yourself to one community

~~~
balajis
Hmm. In all fairness, the full quote is:

    
    
      A BS/MS/PhD in Computer Science or the equivalent is nice 
      to have, but the most important requirement is significant 
      independent programming experience as demonstrated by your 
      GitHub account, personal projects, academic publications, 
      or startup success. Your accomplishments are much more 
      important than your paper credentials.

------
graycat
Looks to me like A16Z is blowing it. If Ben Horowitz prides himself on good
management of technical work as suggested by his blog posts, then he didn't
work very hard on the project of the OP.

Issues:

(1) Engineering School. Uh, guys, here in the US we have universities with
schools of engineering -- MIT, Johns Hopkins, CMU, Georgia Tech, etc. Running
a good engineering school is a lot of hard work. But this A16Z project has
A16Z being their own engineering school, that is, specifying 'course
material', testing for it, and deciding on pass/fail. Somehow it seems a bit
arrogant for A16Z to believe that they can do the work of an engineering
school.

A16Z gets really off the track with their,

> A BS/MS/PhD in Computer Science or the equivalent is nice to have, but the
> most important requirement is significant independent programming experience
> as demonstrated by your GitHub account, personal projects, academic
> publications, or startup success.

Wow! "PhD in Computer Science" ... "is nice to have, but"...! Gee, guys, just
shut down MIT, Johns Hopkins, CMU, Georgia Tech, etc. because now A16Z is here
with "the right stuff"!

Next, for their "as demonstrated by ... academic publications", sorry, A16Z,
"academic publications" are not at all about "programming experience".
Instead, the usual criteria are "new, correct, and significant" where
"programming" really does not count more than, say, just the word processing
for the paper.

(2) Machine Learning. Good grief! That "machine learning" is about
"scatterplots & histograms, logistic/linear regression, R/Matlab" fills a
"much needed gap in the literature" and "would be illuminating if ignited".
Their definition is for a 'statistical data assistant' mostly just for
'descriptive statistics'. Ever hear of J. Tukey? Ah, I'll be nice to you guys;
not fun to be too harsh on small children!

Then there is their unforgettable,

> If you also know enough data science/machine learning to design useful
> metrics for dashboards and workflows, that’s a major plus.

Uh, ever hear of reliability and validity? You have three hours to respond
here with a relevant description! You are not permitted to use Google or
Wikipedia!

(3) Ego. It appears that the A16Z specification was written by someone who has
enough ego to believe that their own 'skill set' is just "the right stuff".

(4) Linux. People programming on Microsoft's platform can have done well but
encountered only a small fraction of the topics in the A16Z document. The
document is mostly about Linux. Sorry, guys, I'm writing in Visual Basic .NET;
no, C# has too much of the deliberately 'idiosyncratic' syntax of C which I
find too error prone and hate.

(5) Three Letter Acronyms (TLAs). It appears that A16Z is using obscure TLAs
as code word filters. Hmm.

So, they have ORMs. Gee, what the heck is that? So, Google a little and get a
Wikipedia entry and discover ORM abbreviates 'object-relational mapping'.
Okay. Sure, if write the software with objects and keep the data in a
relational database, then need move data between objects and tables. Given the
huge variety of objects and tables, tough to believe that there is a
significant 'subject' here.

Then there is their ETL. Google again and discover that that TLA abbreviates
"extract, transform, and load". So that is essentially 'extracting' data from
whatever original sources, 'transforming' the data as needed by the work to be
done, and 'loading' the data into a data base. Okay. Sounds entirely routine.
I'm surprised they didn't also list (A) LRT, look at a screen, read it, and
type in what need, (B) PSS for put on shoes and socks, (C) SDP for start car,
drive to the office, and park the car, and especially the crucially important
(D) GSL for Google a question, select the promising results, and learn what
needed. One more: WTH, waste time on Hacker News!

Okay, A16Z, two can play this game: Think of a large server farm or internet
protocol network. Since this 'facility' is subject to errors of wide variety,
it is important to monitor the facility for unexpected problems of wide
variety, say, from software errors, hardware failures, system administration
errors, security breaches, etc. To do this collect some data, say, from
Microsoft's instrumentation means, HP's Mercury Interactive, what a relational
database management system or Cisco box can report, etc. So, maybe 100 times a
second, collect numerical data on each of 25 variables. Now we want a detector
of 'anomalies', including for problems never seen before, that is, 'zero-day'
problems. Such monitoring is essentially hypothesis testing. So, we want a
suitable hypothesis test. Due to the fact that the multi-dimensional
probability distribution of that 25 dimensional data is essentially impossible
to describe at all accurately, we need an hypothesis test that is
'distribution-free'. Since the data is 25 dimensional, our test must also be
'multi-dimensional'. We want false alarm rate known in advance and adjustable
in small steps over a wide range. And we want a good shot at doing nearly as
well as the Neyman-Pearson result but with much less data than that result
requires. Okay, A16Z, your mission, and I'd like to see you accept it, is to
say how to do that. Of course, since you guys are so good and brilliant, you
already know my published paper for the answer. But there is a typo in the
paper. Where is it? You have three hours to post your response here.

Also needed, find a fast way to do the specified computations. I found a way,
but that is not in the paper. You have 24 hours to post the answer to this
part! Don't report your answer in some silly way such a running code. Instead,
report your answer in clear English with any appropriate mathematics written
as high quality mathematical writing, e.g., W. Rudin. Go for it, guys!

Uh, you won't find the answer on Google or GitHub! And asking around Silicon
Valley won't help, either. You might want to rush off to the RAD lab project
of Patterson and Fox funded by Google, Microsoft, and Sun for when they worked
on such problems, but that won't help, either.

My view: Here A16Z showed that their understanding of computing is still down
at the level of a junior high self taught programmer.

