I suspect in addition to not having the "stomach" for an unlimited number of interviews, they don't have the vacation days to burn for them. Let's say you are working already and have 10 days a year vacation (pretty standard). With these ridiculous all-day interviews, you have the ability to waste that vacation on a maximum of 10 companies, and that's if you choose not to take any "real" vacation at all during the year.
In an environment like today, where every company is ostensibly hiring yet overly picky, a job candidate potentially will waste their entire PTO balance on interviewing with companies serious about interviewing but not serious about hiring.
After wasting about half of my vacation time on (multiple) rounds of interviews with each prospective employer (that ultimately lead nowhere -- most often, a potential employer who inexplicably stops communicating with me), I give up.
Then, when I need to find a new job (project/contract ending, etc.), I do the safe thing and apply to other defense companies, because getting an interview almost always results in an offer (and no multiple-rounds of white-board hazing either). After a while at the new job I'll start looking outside the industry again --- and promptly get pounded back into my pigeonhole :)
Heh. I'm going to squirrel that one away for later. It beautifully captures what's currently thought of as a best practice in tech interviewing.
there are two types of whiteboarding questions. The kind that's a perfectly reasonable thing to ask (i've had people ask me to develop some simple system... which I think is reasonable, because i'll do that as part of the job)
but then there's the solve x algorithm questions. Which ultimately is an exercise in memorization. At this point I've just accepted that i'll have to do that.
I mean, whether or not the company actually values the underlying non-memorization skill is another question (as someone who is very strong in that skill, I think that most companies/roles do not have a use for that, and should not test for that).
But the main argument against whiteboard coding applies equally to both kinds of questions. That argument is that it's a psychologically unrealistic environment, and that the main signal you get from such questions is whether or not the interviewee is psychologically comfortable under this particular artificial form of pressure. Does this generalize to other types of pressure? Does it generalize to the work environment at large?
I mean, me personally, I love whiteboard questions of either kind. Wheee. But I'm not convinced that the tech sector should use my hobby as the ultimate test of employability.
(PS: and for good engineers who are bad at whiteboarding in interviews, I'm available for coaching, heh)
I feel the same way about algorithm puzzle type interviews.
Do I think algorithm questions predict how well you do on a job? No. However I certainly don't think they are just memorization unless you mean memorizing run time of various data structure operations, but that's actually useful to know.
Personally I prefer (as both interviewer and interviewee) a weekend project style format -- pick your IDE, language, have time flexibility, etc.
We've all been in that interview with the one jerk engineer who seems to want to hear themselves talk more than honestly walk through a problem. (Hint: it tends to be the person who offers "Well, that's not the way I would have done it" as a negative and without further explanation)
The only thing I've found that does work is asking them about problems they've encountered/solved before, and then maybe have them write out something from them on the board. This gives pretty decent insight into their problem solving process.
Pedantic (but hey this is HN), but I don't think algorithms are a question of memorization all the time. Sure, "implement quicksort" is a bit silly, but "rotate this binary tree" is eminently reasonable as a shibboleth to comfort with data structures
Agree about it not being super useful for a lot of interviews though.
And all three jobs have been engineering positions.
Now, several years ago when I interviewed with a frickin' huge aerospace company who shall remain nameless, my experience was similar to yours. The interview was a panel-style discussion with the hiring manager and several engineers. They had to follow a script given to them by HR and ask every candidate exactly the same questions in exactly the same way. It was.... surreal. At one point I was asked something along the lines of (my memory is a bit hazy here) "Describe a time when workplace diversity significantly contributed to the accomplishment of a challenging project."
"Oookay. Uh... Well.... Uh... So... Hmm...," and so on and so forth.
And yes, I'm already in the exact scenario you say I shouldn't be able to handle.
I code for fun in Ada.
I left feeling very confused, but after I received the offer, I realized it was basically a grunt work position, despite what the original job listing said.
I once interviewed with Amazon and after flying out and doing the interview I'm told that they liked me but I wasn't a fit for the team and could interview with another team without a delay and they'd even help me figure out the team to interview with. But why would I waste my time again taking days off to fly out and either not pass the interview or interview with the "wrong team" again?
When I interviewed at Microsoft it was a pretty positive experience aside from one person and while I certainly wasn't upset about not getting the position I was told they had 100 people interviewing in person for a single position (supposedly almost 1,000 applications). Great odds for Microsoft but for me taking 3 days off work to do it? Great experience but I'm not sure I'd do it again so easily.
Universally I've found that by being a little bit picky you filter out the people who aren't serious... and a lot of people aren't "serious". They might seriously want someone but they have no capability to identify that person accurately when they have found them.
My Euro-biased perspective is for the job to be something that you don't actively hate doing. The fun stuff(hobbies, family, etc) can be done outside of that.
Learning to quit a job you hate and to save money are both vey useful skills on their own. In reality, it might just take a few weeks of intense effort to find a new job in a different field - but it's always good to have a buffer, especially if you want to re-skill to a completely different field.
Reply to below: Most companies have me do at least one non-technical system design and/or talk-about-a-project interview.
Seriously. I've been working for startups and starting companies for 26 years, and for half that time I've been building teams. I stopped asking people to code a long time ago. If they can't code it will be obvious by their answers to your questions (And I'm not talking about coding brain teasers either.)
Ask them about a project they worked on and to explain to you how it worked. That's sufficient.
I think the reason you whiteboard for google interviews because you will necessarily be a cog in a wheel, and the focus of the cog is to code.
At smaller companies / startups, you will potentially be expected to design, architect, gather requirements, code, etc.
(Interviewing is still somewhat silly.)
When I interview people, I start by asking the interviewee to tell me his/her background, then start conversation about things they just told me. If they mention Ansible I expect them to be able to answer a few technical questions regarding Ansible. If the mention they have Python experience, I expect them to be able to read some code and tell me if they spot a bug or if they knew a solution to the problem. I expect some side conversation like "actually there is a bunch thunder methods in Python, etc" (which is actually related to the question). If you just answer the question, that's boring. There is a skill in interview. You need to make the person enjoy talking to you at work. We are not robot.
I don't care if the person can implement heap or not. If you use a tool enough, you need to go beyond syntax. If you can't show me how you would debug the code, the interview would end there. print, dir with print, pdb, interpreter, whatever. Coding question can help eliminate candidates who are strong in communication but with weak technical skill.
Some candidates think they can code, but they cannot structure the code to make the code usable, and that's a red flag. For the position we want, we are not looking for scripting monkey.
If I am hiring a senior position for the team, I expect to have system design conversation. The last thing I ask is whether the interviewee has done any side projects. I don't penalize people for not having a github/bitbucket account, but would be a huge plus during the evaluation.
Anyway, mileage is different for different position.
Fizzbuzz is an absolutely necessary filter, or you will accidentally hire these people.
Most of the better interviewers work to understand how the candidate thinks and their aptitude to learn new things. Those can tell you a lot about how successful they will be in a job. And folks have shown that take home projects can show you what they can get done.
I found it interesting to compare the company's interviewer training at Google and at IBM. IBM is focused on finding highly qualified people compatible with their values, Google was more interested in finding people who were not intimidated by "impossible" questions. Both have their strengths.
Anyway, I disagree with you. Though I'm not saying your process is bad.
There's a reason I asked MCRed. He has a unique perspective on this that I value.
How would you screen such a person out?
If they don't make you write any code, do not work there: you might end up working with people who can't write code---because the application process doesn't filter those bozos out.
(Doesn't have to be `write code on a whiteboard'. Whiteboarding is actually pretty artificial. Just make sure they do make you write code at all.)
I suspect much of the initial bias and "hard line" preferences are due to a startup's unwillingness to spend inordinate amounts of time interviewing candidates to find that one diamond in the rough.
Imagine 4 of every 10 "academic programmer" candidates are a good fit, and 6 of 10 "product programmer" candidates are a good fit. Why waste time interviewing 10 academic programmers, for a max hit rate of 40%, when you could focus solely on product programmers and get a max hit rate of 60%?
This data certainly suggests that different founders have their own biases and preferences for what makes a "good programmer." Most of them probably realize they are pigeonholing and could be wrong, but they're willing to risk that for more efficient interviewing and a high hit rate.
Stick with what you know if it's "good enough." Optimally, as a founder you can already identify multiple potential early hires just from your own network. If you have the right background and connections, you can hire your first couple employees without doing a single interview.
Time is money. Interviews take time and cost money.
Imagine how mistaken that would sound if you substitute "startup founders" for "programmers."
The reason you feel company filters are anything except random noise is personal bias. It's disheartening that your article notices that company preferences are essentially random noise, and then you don't really internalize that fact or take it to its logical conclusion:
Interviews where programmers do anything except work-hire tests tuned to the signal that the companies want are doomed to fail.
And it's almost impossible to do a work-hire test in an interview. A real work-hire test. The type of test where you actually do some work and then show that yours is effective.
We're so backwards that people probably read this and think I'm saying whiteboard work or something stupid like that. No. Real, actual work. The problem you're working on can be fake, but the problem needs to be literally the same type of thing that the company does on a day-to-day basis.
Want a 100% success rate? Do that, and then don't pay attention to anything but the results. Including whether the candidate has an MIT degree, or any degree at all. If you actually set up a work-hire test, a real one, and then stop asking about all the other stuff that doesn't matter, then you will get a 100% rate. Again, you have to tune the test to what the company is looking for, but that's not hard. "Here is an iOS app. Find and fix these bugs, then add a new feature."
You are 100% correct that most research into interviewing has show it to be far less predictive than companies believe. However, I'll add that no one is doing the false negative studies that would really be required to show this. For example, Google has shown that among people who pass their interview GPA is not correlated with success. This does not at all mean that GPA is not correlated with programming ability. In fact, it almost certainly is. People with low GPA who pass google interviews have a lot of other things going for them. To really analyze this, you'd have to give a job to a random sample of applicants. No one has done this. My goal is for Triplebye to get large enough that I can.
Not for coding specifically, but there have been some studies correlating GPA with career success, and the correlation has been found to be very low. But the reason there are very few studies that look at this is that GPA A) isn't epistemologically meaningful B) isn't a measure in the mathematical sense.
"A grade can be regarded only as an inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite amount of material." -Paul Dressel
Why do you think the programmer from your article didn't get a job when you "sat back to watch him get a job"? You didn't tune your test to what the company was looking for.
Any YC founders who are reading this: Refuse to talk to Triplebyte until they set up a work hire test for your company. Force them to do it, and force them to work with you to tune the test to the work you need. You will get a 100% hit rate. And you'll notice something else: Your retention rate will go way, way up. Want to not deal with firing people? Get them to show you they can do the work. Nothing else matters.
The test needs to be set up so that the candidate can demonstrate they can do exactly the same things all other employees do on a day-to-day basis, or it's not a work-hire test.
If only it were that easy. I am not at all opposed to work-trial tests. They are almost certainly more accurate. But a high percentage of programmer will never apply to a company that requires one. Read any HN comment thread about the topic. It's very controversial. Lots of programmers are against it. Work-trials require MORE time upfront from the candidate, and of course still often lead to rejections (by design a hiring filter rejects most people). A failed work trial burns an entire week of vacation. An unemployed programmer (who is being paid the work-trial) may like it, but someone working another job may not.
sillysaurus3: "Spend a couple hours remotely fixing some bugs and adding a feature on a fake iOS app"
ammon: "A failed work trial burns an entire week of vacation."
You're not talking about the same thing!
sillysaurus3 is talking about doing a small amount of work that is similar to what the company does, under conditions similar to those you would have as an employee. You seem to be talking about short trial period contract-to-hire. Yes, the thing you're talking about sucks and is a huge turnoff to candidates, but (mostly speaking for myself here) what sillysaurus3 is suggesting is massively appealing.
But if your biggest objection is that it will take additional time, that depends entirely on the nature of the trial. I was given a work trial at my first job and it took about 40 minutes. It didn't go as smoothly as I'd hoped, but I would much much rather have done that than spent 40 minutes answering brain teasers in front of a white board.
If I'm already taking a day off, I don't care if the interview lasts three hours or five. I'd rather work on a collaborative task with my potential teammates to see if it's going to be worth radically altering my life to take a new job. I understand that there are many programmers who are opposed to work trials, and they can say "I'm opposed to work trials and would rather demonstrate my abilities in another way."
It seems like your research so far has been very comprehensive, why dismiss an effective technique because some people are opposed to it?
No vacation days burned. No money lost.
Ammon mentioned Weebly, above. They are one company that I've seen require an on-site week as a contractor. I know of a Weebly candidate who lost her current job because the employer considered the leave to be job abandonment. Thankfully, she got the job at Weebly. Perhaps Weebly's weeklong trial is effective for them at weeding out bad hires. But I suspect most good programmers would never consider giving up a week of their lives to an extended job interview.
And I wasn't suggesting job abandonment. Asking for a week of unpaid vacation is a different conversation than asking for a week of paid vacation.
Only if they don't need you and think you are not being productive. So be needed and be productive.
> This was a tough decision and one that we were on the fence about. We really appreciate you taking the time to work on the take home project. We're aware this requires a substantial time commitment and we are really grateful that you invested the time in completing it. We thought you wrote a great, very full featured [trial project]. It was especially impressive how much you dug into the academics behind [the project].
> However we made the decision because we felt that while going through the project together during the interview, we didn't see the fluency of programming when adding to it that we had hoped for. While we specifically designed the take home project track to help overcome the difficulties of coding under time pressure with someone watching, we do still need to see a certain level of programming during the interview. This didn't seem to be the case here, where making changes to the project seemed to be slower and more difficult than we'd have liked.
tptacek had some advice for hiring-through-a-work-test that I (from my armchair) agree with, which is that it's important to give everyone the same work test so your applicants are comparable. There's a little bit of tension (I don't think it's unresolvable) between that and "The test needs to be set up so that the candidate can demonstrate they can do exactly the same things all other employees do on a day-to-day basis, or it's not a work-hire test".
A friend of mine's company usually sends out a little canvas-based browser paint app with a bunch of features, and asks them to implement a few things, like flood fill. It's really enjoyable.
Unfortunately, they then flew me out to their office, whiteboard hazed me, and sent me home. Called me a few weeks later to tell me I didn't get the job, but that they sent me a hoodie in the mail (felt like a consolation prize).
It's a pretty comfy hoodie. They spent that VC money well.
I identify with the "product engineer" concept they outlined, I want to talk UX and user testing in an interview not prove I can understand someone else's code base. Honestly many companies will get far more value of a programmer who has user sympathy and writes user-friendly applications than one who writes a fully tested, bug free and technically sexy app which users don't want to use.
I'm not the best programmer, but I'm comfortable I'd slot in the top quintile, probably top ten percent, of people that walk through a startup's door. And I'd never even return your call. No work-sample test I've ever seen would take less than four hours. That's a $550 opportunity cost for me at my standard rates. What the hell makes you think you deserve that for free? You aren't Google and you aren't Facebook. You need me more than I need you. And almost everybody else who's like you--and given the way you are acting, probably you too--will forget about me or blow me off at some point in the process, wasting my time even further.
On the other hand, an in-person, discussion-based interview isn't work and isn't priced as work; not only does it tells me about the company and whether or not I actually want to work there, but I enjoy meeting new people in a professional setting (there are multiple companies where I've turned them down, but have become friends with people I've met through the process!).
Look at my Github and decide if I can hack if you want, that's why it's there, but I don't work on effing spec.
I'm very old and out of the loop, I have no idea what modern work-sample tests look like. I remember when I did one many years ago it was to the effect of "Write a script that checks to see if a URL returns a valid response code, and trigger an event if it does not". My prospective employer liked my coding style and I got hired. What was so wrong about that?
As I recall, they intended to modify the script and use it in production whether they hired me or not. I also don't care about that. It took less than an hour, and it seemed like a valid form of evaluation. Maybe it's a pride thing, but I had no problem with it and I think a lot of other people would have been ok with that practice as well.
If a company wanted to pay you for that work-sample test, that's different. But they don't, of course, because that costs them money instead of costing you. There's no need to let profit-seeking entities use you for nothing. You just don't need to be their monkey.
You made this exact case yourself in a related comment, in which you called meeting people business development. I agree there's a non-zero distinction here between work-hire-test and consulting-about-consulting, but claiming that the issue is that "you shouldn't work for free" is misleading.
Also, you suggest that if they pay, that costs them instead of costing you. It already costs them more to interview than it costs you. But sure, if making sure they don't get any benefit out of their relationship with you unless you get benefit too, tell them you'll only do a work-hire if they make a donation to the EFF in the value of your hourly rate.
Of course. That's the sit-down-and-chat "interview". But lawyers don't draft up a contract for you to demonstrate that they can write up a contract to your satisfaction. If you want a lawyer's expertise applied concretely to something of your direction, you pay. And while lawyers do have bar exams, they don't have a very detailed demonstration of their work up on Lawhub for your perusal!
> I agree there's a non-zero distinction here between work-hire-test and consulting-about-consulting, but claiming that the issue is that "you shouldn't work for free" is misleading.
I think it's misleading only if you consider the beneficiary of that work to be the same in both cases. I don't. The only beneficiary of every work-sample test I've ever been given was the company--it goes into some black hole and whether it was even good or not, to say nothing of any actionable feedback, has literally-literally never been forthcoming. On the other hand, I have very rarely not benefited in some way from sitting down and chatting with a technical leader at a company, whether it was directly about their problems (or their solutions!) or about tech in general. Both in a consulting-about-consulting capacity and an interview one.
> Also, you suggest that if they pay, that costs them instead of costing you. It already costs them more to interview than it costs you.
In an absolute sense, it definitely costs them more. In a relative one, it emphatically does not: they've got plenty of bodies that can parallelize. I can be interviewing with them, or I can be working, or I can be mopping my bathroom floor. I can't be doing all of those things at once.
*The primary beneficiary being you, if you're a top-10% programmer: It gives you a better chance to accurately show off your ability, which is probably worth thousands of dollars in salary.
But the alternative--it's not the whiteboard, it's the meeting people. Right now, I consult, but I do take interviews for "permanent positions". But we all know they aren't actually permanent positions. The last thirty years of corporatism have made this loud and clear: we are all expendable tools. and I go into them with that in mind: a W-2 doesn't mean I'm anything other than working for a client under a tax-advantageous status. Interviews are business development; I am effectively offering my exclusive services on retainer to a single company. So meeting people to discuss the gig (whether or not the whiteboard comes into play, the only interview I've been on in the last two years where I was both interested in the company and wrote code on the board was Google) benefits me, too: maybe it's not a fit, but maybe I meet somebody I'll remember as "hey, I'd like to work with them again." Or "hey, I don't want to work here, but I know somebody who will, so I can do them a solid and they can help me in the future."
So, yeah, it is work. It's just business development, which is a different, and valuable, kind of work. And it applies to everybody, not just consultants; it's the game we're all playing even if one doesn't realize they're playing it.
(Disclaimer: I strongly prefer to quit my current job before looking for new jobs. If that weren't the case, I'm sure I would be more sensitive to anything that takes time, but I think I would still feel this way.)
Look at the opportunity costs: what if you could spend the time they want you to give them, for free, building something not only personally creditable but maybe even generally useful on Github? If you're good enough to show off, as you suggest, then your time is valuable enough that it should be respected. (A work-sample test that's a useful, valuable problem and can be open-sourced? I'd be down for that. But that would be haaaaard.)
If companies were cold emailing me on LinkedIn asking me to do their two hour project, I would not do it, but neither would I bother going to interview with them. Given that I'm already cherry-picking which companies I care about, I don't mind investing some time to make them care about me.
EDIT: I agree that it would be nicer to spend time doing something really useful. I hope that if I handed them a FOSS thingy that was representative of my ability, that these same companies would respect that in lieu of their work sample -- although standardization is valuable for evaluation purposes, so I can understand if they wouldn't. (I just did the projects they suggested because I thought they sounded fun.)
(Right now, I'm helping out a few days a week over at a fairly large Boston startup, and while they have a general work-sample test that they roll out, I never took it. I probably wouldn't have followed up past the initial phone call if a monkey dance was necessary to get in the door.)
It was fun, and more profitable than a free boring discussion-based interview.
Do we care about equality, or not? I tried not mentionining this aspect, hoping people would realize on their own. But a remote work hire test is also mostly anonymous. It doesn't matter whether you're black, white, male, or female. All that matters is whether you can do the work.
On the flipside, what you're saying is that you genuinely want to spend a vacation day meeting a new company instead of with your family or working on your own projects.
And it's like, if you think you're a good dev, why wouldn't you leap at the opportunity to show it off? I get that it's a little annoying to spend a few hours on it, but the standard interview is literally random noise. Why subject your future to a random process?
I don't know. I respect your view. I'm going to bow out now. Have a good week.
And, FWIW, when salaried, I not once have taken a vacation day to interview. I've said "hey, I'll be in late," and because this industry is so deranged as to think that 50+-hour weeks are normal, no manager has had the temerity to get mad at me for taking a morning off because invariably it will be cashed in when I have to pull a sixteen-hour day to deal with a problem. (The days around Heartbleed earned me some goodwill at that gig, if you follow!)
The typical cost of hiring a programer is way higher than that, more like $30k I'd guess so they could always try paying you $550 to do something semi real? I'm not experienced but I've got a friend who hires remote workers and he says the only way to tell if they are good is to actually hire them to do a small job.
Not really. The best programmers have other stuff to do besides your make-work silliness.
"Get them to show you they can do the work. Nothing else matters."
This is even more silliness. Suppose you have a diverse team, and the a candidate comes up, and passes the test, but is Donald Trump. Are you then going to say that only the work matters?
All 3 ended up being very valuable and 2 of them stayed on well beyond their internship.
Oh, and these projects were done from home before we ever brought them in for an interview and we hired them before ever meeting them.
The effort required to administer an on-site work-hire test is non-zero, therefore I cannot administer such a test to every applicant.
I therefore need a way to determine who to bring on-site, in order to administer such a test. I cannot phone screen every single applicant due to the cost involved.
That process could also be a work-hire test of some sort (e.g. a remote coding project), but regardless of what has been said on this thread, many good applicants will drop off at this step. I know this empirically, because I've experienced it, many, many times.
Additionally, the people who tend to drop off because of this extra effort tend to be senior applicants, who often have multiple offers from multiple companies due to the competiveness of the current hiring market. It also disproportionately drives away passive candidates, who are often the best candidates, because the best candidates are often not looking for work (since they are good, people who have worked with them previously want to work with them again, and they get poached).
So I need some method of sourcing and filtering candidates down that is non-intrusive to both our development team AND the applicant. This is the reality. This simply has to happen in a startup's hiring pipeline.
Currently, most companies do this by looking at resumes. That is obviously sub-optimal.
Any suggestions for alternatives?
Try segmenting potential hires into different categories and using different strategies for each group. You should also clearly communicate the process, so they don't get the wrong idea.
>Additionally, the people who tend to drop off because of this extra effort tend to be senior applicants
Is this before or after the phone screen? If they think the work-hire test is going to be in addition to a phone screen and an on-site whiteboard session, they'll drop. If they're senior devs, you should be able to tell from their resumes. Do a phone screen, and if they pass, explain that they can either do a work-hire test or a whiteboarding session. Explain that the work-hire test should be doable in 4 hours, so they don't envision a week long project. Also explain that it's a substitute for the whiteboard portion of the on-site. People who tend to hate whiteboard coding will choose the work-hire test and vice versa.
If you're trying to find diamonds in the rough among junior talent, sending out a first-pass work-hire test might work. You'd still get better results if you explained that it was short and a substitute for the onsite whiteboard session.
Set up a system that can spin up a droplet for a remote candidate. Any time a candidate expresses interest in your company, spin up an instance and email them a link to it.
What does the link do? That depends on your company. Are you making an iOS app? Then the link takes them to where they can download source code for a fake, hypothetical iOS app. It says "X, Y, and Z bugs exist. Find them and fix them. Then add a feature: here is a clear description of what to add."
When the candidate is done doing this, they zip up their code and send it back to you.
If it sounds way more effective to look at that than to look at resumes, it is. If it sounds like it will repel candidates, well... Two things. First, if you're chasing a specific developer, then that isn't really the normal hiring process. You want them already. This pipeline is for everyone else. It makes no sense to subject them to a work hire test when you're actively seeking them out.
Here's the other point. The type of candidates you will find with this method will shock you. They will be so skilled that it won't matter whether they're called a senior or fresh out of college. You'll know immediately that you want them.
Everything I've described up to this point is a remote process. There is no on-site work hire test. By the time they come on site, you're mainly checking they can show up, and telling them about your company. You're no longer trying to filter them based on ability; they already demonstrated it.
Let's say your company's website is the primary focus, not an iOS app. Ok. The link will take the candidate to a hypothetical, fake website built with a similar framework. Again, it will have multiple bugs and a missing feature. Tell them what the bugs are, and tell them what the feature needs to do. Then have them send you their code when they're done.
I feel like at this point no one will even try to do this. You can think of so many reasons not to try: it takes too much work, it will scare too many people off, it will... Etc.
These reasons turn out to be largely fake or mistaken. Try it. Invest the resources to build this pipeline, tell HN when it's ready, and you win.
If this sounds prohibitive or unlikely, remember how counter-intuitive the most effective techniques in life are. Penicillin was discovered by accident. It sounds pretty unlikely that it would work. Same deal here.
I've explained this as clearly as I can. It's up to everyone else to either try it or to watch others win after they try it. Because the filter I've explained is the only way to let talent find you.
The type of people you'll discover will range from passive people who found the process amusing, to well-off senior developers who are demonstrating why you should pay them X equity or Y salary, to high school dropouts who turn out to be one of the most valuable people that join your team.
I'm not even going to touch the topic of what tech companies currently do. It doesn't matter. I've described what works, and if whoever reads this suppresses their instincts and builds this, they will discover it's practically the key to winning.
I don't hire programmers, but have been a hiring manager, running fairly technical business teams now for 10+ years, and have found that initiative is often the deciding factor in any given employee's success.
There are other ways to select for it, and clues you start to pick up on over the years, but once you've figured out how to gauge relative levels of initiative across different people, I've found it makes hiring/resourcing questions quite a bit easier.
We've evolved the same process in my own startup - in the last 2 months we've hired 5 remote programmers in Vietnam and India by giving them 2 rounds of remote work-sample tasks (a simple web app), then ending off with a text or voice interview. We ended up retaining 2 out of 5 of them (both Vietnamese, incidentally) who demonstrated a higher level of ability after they started.
The retention rate is low, so we've been tightening up the process. What we've found is that we probably needed to raise the bar a little by asking candidates to design and architect a feature (DB-backend-frontend) during the chat interview. That would have given us a better insight into their code structuring and teamwork/communication skills.
We've just brought on a 3rd hire (Vietnamese again) who passed our improved interview, and I have a good feeling that the process works very well to weed out candidates who don't have the needed level of ability and scrupulousness.
The best of our hires didn't attend college or have much relevant experience, but was clearly very capable and possessed great initiative. We're really glad to have managed to hire him. Another guy is finishing his 3rd year of college.
In summary, I think remote work-sample tests are fantastic, and I hope you're right that they're the key to winning :)
So this 2 hour work-hire test that narrowly focuses on specific tools and platform won't work for vast majority of programmers who haven't branded themselves in to one narrow area. Also remember that a lot of programmers at places like Google, FB and Microsoft work in highly proprietory environment with their own internal build system, platforms and tools. They are likely not familiar with everything that's latest and in fashion. However they are very smart people, can change gear easily and, most importantly, ramp up on demand very quickly.
Yet another big issue is that work-hire test usually fail to measure hardcore computer science problem solving abilities. Sure, you are good at fixing some iOS bugs but can you figure out how to do driving directions on maps or may be scale email system to 100 million users? Now I do agree that lot of jobs don't require hard core CS but many do. And for a startup, it's usually impossible to tell if your future 6 month down the line will hing on having someone with hard core CS skills.
As far as hardcore computer science abilities, a test still won't help, because what you are really testing is how far people are from college: Generalist cs is touched in college, but there are entire sections of computer science that someone with 10 years of experience might not have had to touch. I've had to do pretty specific CS stuff for work, but to get any of that done, I started with weeks of research, and then add my own spin to our specific situation top. You can't test the ability to do that by just checking if someone has memorized Raft, or black red trees, or any other random problem. I don't have a lot of CS algorithms memorized: that's why there's the internet, and why I have Knuth in my shelf.
You want to check for hard cs, ask the candidate to choose his favorite CS topic, and ask for am impromptu presentation on that. anything else is just going to fail at finding candidates that have different backgrounds than you, and that's PRECISELY the people you want to hire.
Isn't that pretty common among incubators and VCs? It's not the only factor, but for first-time founders, being from Stanford is a large plus when it comes to getting in the door. If you already have a track record then of course they just go by that. But when it's your first company, I don't think the "startup ecosystem" is any less reliant on these kinds of signalling credentials to filter the large volume of people who want funding.
I'm assuming if it was a fake problem, it wouldn't be something the other company could actually use.
One thing to realize is that your employer can't really retaliate. The hypothetical iOS app I mentioned is set up for one purpose: to let candidates show they can do work the company is looking for. Not only will the code be thrown away, but it wouldn't make any sense to use it.
Uh are you sure about that? Google's experience seems to be that academic achievement is not a good predictor. And they know a thing or two about hiring.
$20 buys you a copy of "The Who" book, which is a proven, scalable system with specific questions to ask at each step of the interview process....
Now I have a rule that I will interview with ONE company at a time. Consequently, I am very careful about picking which companies I will interview with.
Yikes, are people really doing this, or feel like they must?
If I'm interviewing, I expect to be asked about previous projects, maybe my Github portfolio, and so on. I also expect to be asked how I might approach a given problem within my area of work. I can't imagine studying that - I'm paid to do this stuff every day.
So are we talking about stuff that isn't related to my area of work, like brain-teasers and random algorithm questions?
It is not enough to be able to implement quicksort. One must have the algorithm thoroughly memorized so that it can be scrawled onto the whiteboard in one smooth motion of the dry-erase pen. If you pause to say "hmm..." while re-deriving quicksort on the spot, you will be branded as "slow" and obviously inferior to the candidate who spent three days bashing his/her head against an algorithms textbook.
I'm suspecting that this is a "Valley" thing, would that be wrong?
>It is not enough to be able to implement quicksort. One must have the algorithm thoroughly memorized so that it can be scrawled onto the whiteboard in one smooth motion of the dry-erase pen. If you pause to say "hmm..." while re-deriving quicksort on the spot, you will be branded as "slow" and obviously inferior to the candidate who spent three days bashing his/her head against an algorithms textbook.
If someone has years of experience in this field, and a strong portfolio, why would they need to remember how to do quicksort? They probably haven't done it since their university days. That shouldn't even remotely be the focus of an interview for someone who has done real work in this industry.
>If someone has years of experience in their field and a strong portfolio
Well, the problem is that quite a lot of programmers don't have portfolios outside of work. I certainly don't. I mean, I have a Github, true, but there's not really anything on there except for a bunch of half-finished experiments. Honestly, I don't have the energy to spend 8 hours a day immersed in programming, and then come home and spend one to two hours more building up my portfolio. Maybe that makes me a "bad" programmer, by some definitions.
The last time I had to write anything on a whiteboard in an interview, it was a data model relating to the work I'd be doing. I felt it was relevant to the process, at the time. If it weren't I'd feel like my valuable time (as well as theirs) was being wasted. But I suppose this comes down to what a company wants. If they just want people who are generally smart, and can be ramped up on the company's tech stack, fair enough. But some companies need to hire people who can hit the ground running. Much of my contracting has been done on such a basis.
>Well, the problem is that quite a lot of programmers don't have portfolios outside of work. I certainly don't. I mean, I have a Github, true, but there's not really anything on there except for a bunch of half-finished experiments. Honestly, I don't have the energy to spend 8 hours a day immersed in programming, and then come home and spend one to two hours more building up my portfolio. Maybe that makes me a "bad" programmer, by some definitions.
Your portfolio need not be only Github work. Barring an NDA, you can certainly discuss your professional work. In fact, I want to be asked about it during my interviews, as it forms the basis of my experience (which in theory is a big part of why I'm there in the interview room!).
Ask the candidate "Tell me about a project you're passionate about". Find out why. Get them to explain a hard problem they solved. Get them to teach it to you so you understand it.
Find the candidate that can do that and that gives you a good answer and you find a candidate who is passionate about something and can communicate it and understood it well enough to relate it in detail to you on the spot. That's the one to hire.
not the one who memorized quick sort... the former can implement a new quicsort.... the other one will spend all his time on stack overflow asking others to do his work.
I guess Google, Microsoft, Amazon, Facebook and Dropbox are all low quality companies, then.
One problem that seems pervasive on hacker news is Cargo Cultism. Just because one of those companies does something does not mean that it's a good thing. Some of those companies have really capricious and arbitrary hiring policies.
Don't get starry eyed. Don't confuse a household name for quality. And don't mindlessly copy the broken system of a fortune 500 company for your startup.
I'm an engineer, I'm not just pulling this out of my ass. I've been conducting thousands of interviews over the past decades and had to manage the people I hired with my process.
And yeah, again YMMV, but you definitely do need to largely solve the problem, so speed absolutely does matter here.
The really frustrating thing about all this is that if the experience causes you to go back, study algorithms, buy cracking the coding interview, and getting incredibly good at these questions, your next interview will involve… detailed questions about ruby syntax, followed by another interview with questions about how the JVM works, followed by another interview with questions about how MapReduce works.
Not saying this is good or bad, just that this is how it is, at least at the harder interviews.
Based on the final paragraph, it looks like triple byte does see the market opportunity here. The frustrating moving target of interview questions could correlate very well with what type of programmers a company is looking for. Considering that every interview requires a day (or more) off and a stressful and mentally taxing series of in-person exams, there's only so much one candidate can take before giving up and just staying in his or her job (assuming the candidate is currently employed). So if a company becomes aware of how to align candidate type with interview path, if you can line up with the algorithms style interviews, the person can slowly learn until getting a job offer, rather than experiencing a reset button. I wish them luck with this, because they'd be adding real value to a developer's job search, which isn't something devs typically expect from a recruiter.
If at an interview they asked/expected that of me, it would promptly disinterest me in working for them. And I'd probably tell them that right there.
Optimizing for memorization of specific algorithms (of sorting even?) instead of problem-solving and domain-knowledge is quite a weird/perverse incentive. I'm surprised it's gotten this far, if it is true. The places I've interviewed generally ask prior experience, talk about solutions and approaches, and theoretical OO-questions of the fizz-buzz level just to be sure.
I interviewed for jobs recently, and I studied my ass off for 6+ weeks, every night doing different coding questions, and spending weekends learning new things and coding algorithms for 6-8 hrs a a day.
It paid off, I got multiple job offers, and am pretty happy with the end result.
Indeed, I don't. I even asked, in one of my replies, if this was a "Valley" thing.
I'm ok with being asked to code or write things out during interviews - that is, if it is pertinent to the job I'll be performing. If I'm expected to do complex DB queries, for example, it is reasonable to see if I can write one. If I've been doing it for years, I should be able to handle such a task.
>I interviewed for jobs recently, and I studied my ass off for 6+ weeks, every night doing different coding questions, and spending weekends learning new things and coding algorithms for 6-8 hrs a a day.
But this... this seems more appropriate in the context of a university exam than a job in real-world software development. As I said in elsewhere in this thread, I can understand asking such questions of a prospective junior developer, as they won't have much experience, and the CS 101 material may be fresher in their mind.
>If I'm interviewing, I expect to be asked about previous projects, maybe my Github portfolio, and so on.
Oh. Well you are in for a big surprise if you interview at 90% of tech companies.
Not at all. For a junior position. But that's a difference of opinion with which we may have to live. For more experienced candidates, I'd be far more interested in what they've done to apply their knowledge in real-world scenarios. I need them to be able to do more than hypotheticals.
>I would never work for a company that didn't ask engineer candidates those questions.
And that's fine. Small point to add: I see at least one regional difference, with regards to the terms being used - I never see software developers called "engineers" unless they have an engineering degree. Maybe that's just my part of Canada, I don't know. Perhaps it is more of a protected term here.
For example, in 2001, Microsoft apparently agreed that in Canada, they would only use the acronym "MCSE", and not spell it out, because holding an MCSE does not make you an engineer.
I think it's amusing and cute that American developers call themselves engineers. It's not like they take on the ethical responsibilities that engineers do. But when in Rome, speak Romansh. Or however that goes.
Problem is there is such a wide variety of questions its hard to be prepared across the board ...
Yet it's amazing how many companies treat candidates' time as basically worthless, and infinitely replenishable.
High bars can really turn out to be self-sabotage.
Good God, is this actually a thing? I mean, a probationary period after hire is one thing, but if somebody actually had the gall to tell me I'd have to spend a week or two working for them before they decided whether or not I was actually working for them, I believe I'd walk out on the spot.
The company I'm at doesn't employ any magic voodoo in our hiring methods, but we somehow manage to find great people without being a revolving door for interviews. Many places are proud to say that they only make offers to 1% of the people they bring on-site, but I'm proud to say that our ratio is probably more like 25% or even 50%.
I've been watching a friend interview with their 'dream company' over the last month. It's been quite an exhaustive process. Four rounds of interviews: a phone-screen, a take-home test that took a weekend, an all-day in-person, and a half-or-full-day remote session. They've moved beyond interview step #2 and have secured a date for interview step #3.
The way you've phrased it here is exactly what I have internalized from watching this process unfold but have been unable to articulate so succinctly. Suffice it to say, I'm not looking forward to my next job hunt.
Is it really this bad in the States?
1) No paid vacation at all
2) 15-25 days of vacation a year
3) "Unlimited vacation," which is usually a hip facade for 1 or 2.
Really? 25 days is 5 weeks; I feel like the standard for PTO is 10-15 days, and often it's 10 days, but you get bumped up to 15 after you work at a place for a year or two.
Is that really true? I have very limited first-hand experience; only two of my employers have offered unlimited vacation. For one of them, it was as you describe. For my current company... well, by the end of this calendar year, I think it'll add up to a little over 5 weeks, which is fairly unusual by US standards.
As a response, many companies are switching to "unlimited" plans. These look good to the uninitiated. But the real impetus for an "unlimited" plan is that it frees the company from having to guarantee -- or pay out for -- a minimum number of days for each employee. It's not really about lifting the ceiling; it's about lowering the floor.
I'm sure there are companies out there who really do have, and make every effort to honor, "unlimited" vacation plans. But I believe these companies are the exception to the rule.
In reality, it was nowhere near "unlimited vacation" as everything still needed to be approved, and it was very common for vacation requests to not be approved for a variety of business reasons. She probably ended up with ~20-25 days off per year, including stat holidays, which is the same as you'd get at Google or Microsoft.
Since it's a young company, most employees are indespensible, and without any written guarantee saying "you have X days of vacation a year", everyone is too afraid to even ask for any time off.
I worked for one of those. I'm glad I got out. Unfortunately, I landed at a company that's ridiculously stingy with PTO (10 vacation days, 8 holidays, 7 sick days -- this is slightly mitigated by the fact that you can make up hours you miss without using PTO, so I've done things where I leave early one day but stay late the next), however because company policy requires that employees take all our vacation days every year, at least I know what their expectations are. I took a vacation this year because I knew I had the days for it and so I just filled out our standard form and gave it to my boss, knowing it would be accepted. On the contrary, I didn't take a single vacation during my time at my previous employer, and in fact I never even asked for one, because we were bouncing from crisis to crisis, and I could just imagine my boss saying "are you crazy?".
"Can I take three weeks off in July?"
"Absolutely! But we might penalize you on your performance review."
Usual disclaimers about not every company etc.
Just my experience, but yeah, what I've seen is a 15 day minimum, and I don't explicitly filter out places that offer less.
Perhaps it's the type of companies I go for? If I recall correctly, Google, FB, Amazon, MS etc. all offer 15+ days for new hires, and most other large companies try to at least match them.
Some people negotiate additional PTO as part of their compensation package but I don't need to tell HN that engineers are notoriously bad at doing this.
Not to mention that tech industry firms are reputed to be more generous than what is otherwise common, that sounds really low even outside of the tech industry; its lower than any company I've seen, and lower than what most sources I've seen indicate is common. 
 e.g., http://www.bls.gov/news.release/ebs.t05.htm and http://www.salary.com/time-off-paid-time-off-from-work/
I worked as a programmer at a big boring company which did this - 10 days of company holidays, 10 days of you-get-to-choose. It's a month off total, but you only have control over 2 weeks, so it's presented as 10 vacation days.
The first source I linked provides actual averages for paid holidays, paid vacation, and paid sick leave.
10 paid vacation days is dead average for professional, technical, and related employees -- with only one year of service, and low for such employees with more service.
I was at a start-up once where we got a new CEO who essentially canceled a week of "whole company time off" at Christmas. If you wanted that time off, it came out of your vacation. He didn't care . . . and soon after that, I didn't care, either :-)
[Didn't help that he was really scarce in the office that week...]
This pressure isn't universal, but I'd say the majority of the companies I've worked for hardly anyone ever took more than a week off in a go, and when they did it was the topic of uncomfortable conversation with managers kind of acting like the person taking the time off was doing something wrong (even if just kind of half-joking about it).
It really is an entirely weird situation, for all the talk of meritocracy and such the day-to-day reality is you are pressured not to take real PTO (paid time off) and instead just do the Office Space thing where you're at work almost every day even if there are periods of time where you aren't really doing anything productive for a large chunk of most days.
So you start with 22 working days off including holidays, plus generous sick time. Not perfect, but could be worse.
In 12 years I don't recall any jobs with fewer than 15 days. My current job is "unlimited", and I took nearly 25 days one year.
But also remember it's business days. So 10 days = 2 weeks.
You don't make an omelet without enslaving some chickens their entire lives, keeping them in barely one cubic foot of space from birth to death so they can push out eggs at maximal productivity, killing them as soon as they're no longer at peak performance, and of course, breaking the eggs.
In other words, you get what you pay for.
I agree with you that getting hired as a product manager isn't easy but, almost without exception (even at places like Amazon), the process is actually respectful, which makes a world of difference.
So it's an expensive, stressful, crappy experience. Weird I don't want to do it speculatively, eh?
If your job is just a vehicle to enable your lifestyle, like mine, then you'll be able to establish a set of criteria and only work with a recruiter that respects those criteria.
Good recruiters very much care about their reputation, because a sale isn't always a sale when either side ends up unhappy.
I think they are usually hired by the companies, or "loyal" to the companies, because that is were the money come from.
Would be a fun experiment to hire somebody to sell yourself. Then again, maybe that is what is usually called a "personal coach"?
I cycled through like three recruiters before finding one with a job I was interested in. If you really like working with a particular recruiter, what you might do is look for jobs yourself and then forward those listings on to the recruiter, so he can do his thing. It's not how most recruiters are used to operating, and it's not something I've done, but I think it could work based on my previous experiences.
In my experience, recruiters are far more 'loyal' to job hunters than they are to employers, at least in tech fields. Hunters have nothing but options in who they go through and how they conduct a search. Whereas the requirements of companies are usually fixed and often are completely irrational. It's often easier to work with and influence a job seeker than it is to talk sense into an employer.
Just the other week, I ran out for a meeting with a potential new employer in the middle of the day. All I said was I "had to run an errand." If I was less checked out, I would've made up a real excuse and told them I needed to get a blood test.
Most people are totally non-confrontational. You're not going to get called out on your little white lies. Obviously you can't use the same excuse 10 weeks in a row, so mix it up a bit.
EDIT: After few minutes with Google.
> USA: There is no statutory minimum. It is left to the employers to offer paid vacation days as part of the compensation and benefits package. According to one survey, 98% of employers offer at least some paid leave to their employees; full-time employees earn between 6 and 20 vacation days at 86% of employers surveyed. About 96% of surveyed employers give their employees paid time off during public holidays, typically 6 per year. Some employers offer no vacations at all. The average number of paid vacation days offered by private employers is 10 days after 1 year of service, 14 days after 5 years, 17 days after 10 years, and 19 days after 20 years
> European Union legislation mandates that all 28 member states must by law grant all employees a minimum of 4 weeks of paid vacation
> Sweden: Employees are entitled to 25 work days of annual leave. Sweden also has 11 public holidays and a few that may or may not be half days.
As far as unpaid time off, culturally, the amount of paid vacation time you get at a company is assumed to be the maximum amount of time off you can take in the normal course of matters without putting your job at risk. Normally people only ask for unpaid time off if there's some exigent event that they have to take care of (health issues, family issues, etc.).
a. Pre-screens candidates on off-hours since they are likely already employed with a focus on keeping it brief out of respect for their time.
b. Similarly match interviews at off-hours and perhaps moderate them to also not waste anybody's time.
I am sure there are many people that would not like this but if hiring is difficult and they want to make it rigorous I wonder if this could fit a niche. I personally am more than happy to go to an interview after work or on the weekend at a coffee shop so it doesn't cut into my job hours.
You can start shifting that extra day around for these occasions, leaving your vacation days for actual vacation ( which is 80% as is the pay ofcouse ).
The bar exam is three grueling days. 100 hours of study is two and a half weeks of full time work.
If you interview at three tech companies, I don't think it's at all over the top to say you might study for a few weeks in total, especially if you are rusty, and the interviews are often quite grueling as well and do last all day….
Ok, I'm stretching, I won't put one iteration of interviews at bar exam levels, but we are starting to get there. Now keep in mind, we do this over and over and over in our field. And while the bar is tough, at least you know roughly what will be tested - tech interviews are often a completely moving target. And while the bar does have continuing education requirements, you don't have to do the 3 day massive study thing over and over (unless, in some cases, you move to a new state).
I really do think this is a severe problem in our industry that causes higher levels of attrition than we realize. I suppose it could be one of those "tragedy of the commons" type things, where each company benefits from long and difficult interviews, but the cumulative result is either 1) people not wanting to enter the field, or 2) people giving up on interviewing for new jobs and staying with their existing employer even if they are burned out, or 3) people quitting and going into a different field or role entirely where they can escape this hazing.
I also think this is something hiring managers should realize before saying that there is a a shortage of programmers. Your own interview processes may be contributing substantially to this "shortage".
Oh, one last thing - excellent article, thank you for writing it! Absolutely fascinating, and it explains a great deal.
Maybe the issue is too risky to employ somebody? In my country a lot of laws protect employees, so it might be difficult to get rid of bad hires (although in the first 6 months I think it is easy, as they count as probation period).
>Almost no one passes all their programming interviews
But there is always someone who will, and many companies are more than happy to pass on all candidates until the find someone who days. I was super happy that my interviewer yesterday said we were skipping the whiteboard section.
What you say is true to a degree; if you have plenty of money saved up then you can spend a lot of time unemployed but this ultimately looks poorly on your resume at some companies and most people live closer to paycheck to paycheck than anything else.
But just to undermine that point, I've also learned that it's quite complex to try to juggle multiple job interview processes with a day job, especially if you want to be able to make a choice with multiple offers on the table. It ended up working out for me this past time, but next time I'm ready to make a move (hopefully not for years!), I may just resign first, so I'd be able to devote my full attention to making the right move.
Until my current gig, I was never once allowed vacation time.
I don't think you've ever experienced the passive aggressiveness that comes with using "unlimited" vacation time.
I'm not going to lay on my death bed and wish I had another couple lines of code committed.
It's a little humbling to realize that I had a price, after all.
Taking a perm role for a big salary isn't a bad thing, either. That money can be turned into more time off (as long as you don't die first, but that's the gamble, right?). I'd personally rather take it up front while I'm young, but the other way around does work too.
Facebook, Google, etc commonly use the contractor technique, where anyone you want to work for you but not give benefits becomes an independent contractor, or works for a vendor. Eg, the cooks in the Facebook kitchens.
Also there's no reason everyone has to receive the same amount. Indeed that would be very unusual.
Edit: And, you are supposed to use them. Not making sure your employees spend their holiday days can land at least the company if not the employee as well in legal hot water.
The real issue with unlimited PTO is that many people are so conflict-adverse (except on the internet) that they cannot initiate basic conversations about how much vacation time they think is reasonable. And instead of taking responsibility for communicating their expectations like adults, they stew in resentment against their employer for shifting that responsibility onto them.
If your employer balks at your idea of reasonable time off, then you might think about switching employers. But if you balk at the idea of having some input into your employer's expectations of you, then you need to take more responsibility, or move to a company with a less collaborative culture.
It is in all cases management's job to set these expectations, and their fault when they're not understood on both ends of things. That's why they're management. (And why "unlimited" policies are stupid.)
Poor choice of words on my part. Take it in this context: "Person X doesn't have a stomach for hard engineering problem Y".
"Doesn't have the stomach for" now can be interpreted as a synonym for:
* Is not capable of
* Is not smart enough for
* Is not hard working enough for (!)
In the current environment I definitely shouldn't have brought up gender.
You do, of course, have the natural right be offended by anything. That doesn't mean people have the responsibility to not offend you. In fact, in America, home of HN, we explicitly have the right to offend you.
Edit: Was originally some kind of nonsense about the expression "doesn't have the stomach for" being "a jab at masculinity" or something.