Hacker News new | past | comments | ask | show | jobs | submit | page 2 login
Announcing Starfighter (kalzumeus.com)
932 points by jsnell on Mar 9, 2015 | hide | past | web | favorite | 400 comments

Very cool! Signed up for an invite.

Couple questions:

1) Is this a Stripe-like CTF that happens over some caffeine-fueled weekend? Or is it more of an ongoing Project Euler-style drop-in-and-solve type process? I don't think I could handle the former, but the latter sounds quite enjoyable and something I might do in my free time just to learn. I did that with PE for a while since it was fun to earn completion points.

2) You mention "Let's Play" style videos. How do you make it so that the solutions aren't given away? Does each player have a customized CTF somehow?

I loved the Stripe CTF. We're not going to adopt the time-boxed model from them. Starfighter : our CTFs :: Blizzard :: Blizzard games. There's no reason for us to ever turn them off. We'll likely have a particular flagship property but we have the option of dropping a new game on the market any time we want.

You don't have to rush through Starfighter games. Some people will, of course, just like there is a metagame to be the first guild on the server to complete a new raid. Players are gonna play. But if you want to pick up our CTF a year after release and just casually spend a few hours learning a new skill, you'll be able to do that, and we'll fully support you in it.

How do you make it so that the solutions aren't given away?

The same way you do it with math problems. Mastery of the subject material is the easiest cheat code.

>The same way you do it with math problems. Mastery of the subject material is the easiest cheat code.

Elaborate? Math questions are the opposite of a good example, aren't they? Even with randomized values it's fairly easy to write generic solutions. People do this even for Project Euler. How much more so for something even cooler sounding?

(I'm very excited to see this project and it sounds like a lot of fun. I'd just be worried that it would end up being gamed, since the drive for cheating is so strong everywhere.)

I actually hope this happens, and that there's a community of code sharing that forms around it.

Wouldn't that just make cheating easier, then? I'm sure you have this sorted out somehow, it just seems like a rather large unaddressed question.

The complexity of the problem solver generally increases faster than the complexity of the problem generator.

For example, a general solver for "what is #{x} + #{y}?" is relatively easy. A general solver for "what is the derivative of #{random_equasion_with_diffuculty(3)}?" requires a full computer algebra system. Take one more step into advanced mathematics, and things get more dicy -- automated theorem provers, for example, require human guiding and hints.

> The same way you do it with math problems. Mastery of the subject material is the easiest cheat code.

This, at best, sidesteps the question. If a solution is a string of characters, that string of characters will end up on the Internet.

If a solution is a series of tasks---try this string of characters, observe what happens, then try this other string of characters---people will still share the strings and instructions on the Internet. All this still holds if the strings are personalized in some way; then the general patterns will be shared.

I spent a few years at university as a teaching assistant for a programming course using an interactive IDE where this kind of thing was required: Run the provided unit tests, ONLY THEN write some code, ONLY THEN run these other tests, ONLY THEN make the appropriate modifications etc. The idea was to make cheating more expensive than actually thinking about the issues, and to some extent this succeeded in the narrow sense that cheating WAS more expensive, but it did not succeed in the more general sense: people STILL cheated because they really didn't want to think things through.

The higher the stakes are, the more not-so-great people will be attracted to the system and will chug through based on someone else's solutions. This will be a fun game for those who see it as a game, but I don't see how the using-this-for-recruiting part will not implode almost instantly.

Finally, the whole thing about keeping even the general subject areas a secret is a bit silly. The work the best tech companies in the world need ranges from medical visualization to databases to systems programming to pentesting (one of you wrote this will not be a pentest game!) to static analysis and verification to distributed systems to scientific computing to drawing pretty pictures in web browsers to... You will only cover small sliver of this. Everybody knows this, why be an ass about it?

It is long-running and not timeboxed.

I am surprised that Patrick is moving onto yet a different project.

After leaving his job he works as a freelance online marketing expert. Then quits that despite implying making a lot of money. Instead wants to create online marketing courses to reach bigger audience, but takes forever to produce any content and is now abandoning that track. Creates AppointmentReminder with some good initial success but reading between the lines that is going to be sold/abandoned as well.

Now moving onto yet another project. Seems you have created several great opportunities for yourself but cannot stick and focus on any one thing?

I am curious about a biz dev kind of thing, and it may be entirely too early to answer, but here goes:

"Contingent recruiting" has an extremely negative connotation to me. Quite literally, I have to constrain my bias when designing hiring pipelines to not throw recruiter backed resumes away. I understand that you are only taking the payment model, and are trying to undermine the business model, but have you run into any problems associating with the industry?

Or, more interestingly, have you encountered anyone on the consumer side (HR, C-Suite management, etc) that has pointed out a non-obvious advantage to typical contingent recruiting. I for one would love to hear why companies keep going back to that obviously terrible well.

I'm excited to see what comes of this.

Another attempt to apply data analysis to produce a concrete number that represents the merits and talents of a fellow human being.

How utterly irrational.

It's ideas like this that make me question why I bother being a programmer for anything more than leisure these days... except that I have no idea what else I could do to keep a roof over my family's head this late in my life.

You call yourselves, "engineers," but I've yet to work for any company that treated you like one or even invested in your future. You're offered a salary well above average for most working adults. They lure you with frivolous perks and stock options. They never send you to school for training, offer you a pension to keep you secure in your golden years, and I've never seen any unions you could join to demand these things from your employers. Your career is probably not going to be on the line if you merge a patch that degrades performance or introduces a timing error. But don't think they'll care about you when it comes to the bottom line. Someone else seeks to profit off of your talents and abilities and nothing else. If they can hire someone to automate you away, they will.

And we're all the poorer for it I think.

You may love your job, but your job never loves you back. Don't remember where I heard this, but it's always good to remember. It applies (in varying measures) to anyone from the founders and CEO to the lowliest employee.

If you could tell us how to apply data analysis to produce a concrete number that represents the merits and talents of a fellow human being, I would love to hear it; it would make my job a lot easier. I didn't know that was possible. Is this some kind of linear algebra-y machine learning sort of thing?

Most likely it is possible. Take this "Alex" you frequently speak highly of. If you gave Alex a different work sample test, do you believe he'd have done poorly on it? If not, then there is some common factor (call it H) both tests are measuring.

(I can confidently state that you do believe he would have performed similarly on a different test - if he didn't, this whole idea would be fundamentally invalid.)

Now the important question is to determine, for any particular subtest, how closely a person's score correlates with the hidden factor H. This is, for example, how IQ tests are created. You can take a look at the literature surrounding them. Key starting points would be principal component analysis, clustering, hidden markov models and the like.

There is no simple recipe - every data analysis problem is different in it's own way. But there are general themes. I'm happy to discuss in more detail, feel free to write to me if you are interested.

This is a good answer.

I could tell if GP was being sarcastic so I deleted my original reply.

The softer side of the equation is that the approach of filtering candidates for businesses allows them to minimize (to some market minimum) the cost for highly skilled individuals. Projects such as this just triggered a bone I'm presently picking over.

Old-guard corporations that hire the legally-liable kind of engineer seemed to have taken a much different approach if the stories I've been told by a retired mechanical engineer who worked for Chrysler are true. A completely irrational approach. They hired people based on their potential and made them into the engineers they needed.

The copy on the announcement sounds like this game will quantify everything about the performance of a participant in the game in order to sell them to a curated list of potential employers. What about this system incentives employers to invest in the career development of these people and strengthens our collective bargaining power as the people who build this stuff?

To be more constructive I might suggest turning down the hyperbole and use fewer adjectives. Keep the pitch to employers for employers. Don't be patronizing to inexperienced developers: inspecting the assembly output of a compiler is not elite and not difficult to explain to someone given the right context and framing. I understand your game is about competition but the "winner/failure" schism it can create is a big turn off for a lot of otherwise intelligent, creative, and capable people. It doesn't have to be about being, "the best," in order to be fun and rewarding.

IME, IQ tests (like other standardized tests, SAT's, GRE's etc) are easily gamed. I did a couple practice ones and my score jumped up by 20+ points the second time. Where is the evidence that it measures some hidden factor? Seems to me that (like other standardized tests) it simply measures the subject's level of training and preparation.

It depends which test is used and there are limits in magnitude to practice effects, i.e. you can't coach anyone to a perfect score in a test with a large bank of test questions. Psychometricians are aware of these critiques. Tests like the SAT, GRE, GMAT etc., lightly disguised IQ tests all, are useful for predicting academic performance among other things.

>Effects of practice on the Wechsler Adult Intelligence Scale-IV across 3- and 6-month intervals. Estevis E1, Basso MR, Combs D.

A total of 54 participants (age M = 20.9; education M = 14.9; initial Full Scale IQ M = 111.6) were administered the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) at baseline and again either 3 or 6 months later. Scores on the Full Scale IQ, Verbal Comprehension, Working Memory, Perceptual Reasoning, Processing Speed, and General Ability Indices improved approximately 7, 5, 4, 5, 9, and 6 points, respectively, and increases were similar regardless of whether the re-examination occurred over 3- or 6-month intervals.

> Practice Effects for the Stanford–Binet Intelligence Scales The Stanford-Binet Intelligence Scales—Fifth Edition (SB5) is a widely used assessment tool for measuring intelligence (Roid, 2003). According to Roid, a key advantage of this intelligence test’s most recent revision is that it includes improved lowend items for better measurement of young children or adults having mental retardation. Sbordone, Saul, and Purisch (2007) report that the range of the SB5 was expanded to allow the assessment of very low and very high levels of cognitive ability. Roid and Barram (2004) indicate that the practice effects on the SB5 were smaller than expected. For example, the nonverbal IQ of the SB5 showed shifts of only 2 to 5 points as compared to the 4 to 13 points on the Performance IQ of the Wechsler scales (i.e., the WAIS-III and WISC-III). Roid and Barram add that the lower shift, and thus practice effect, is even more notable given that the retest period for the SB5 was 5 to 8 days versus 23 to 35 days on average for the Wechsler scales.

At an absolute minimum you should be collecting lots of statistics and using them to train a bayesian candidate filter to estimate hiring probability (the same architecture as a bayesian spam filter would work).

If you find yourself wanting something more advanced than that, random forests would be a good match here, as would clustering the candidates using DBNs or simply k-means and then using base rates.

Also, if you haven't looked at https://www.kaggle.com/competitions you really should -- they're similar to what you're doing, only for machine learning.

I was hoping the challenges would include machine learning. There is so much exciting overlap between ML and security. From detecting intrusion attempts to scraping and processing pastebins.

I think it is amazing what you guys did here: you got more points on a give me your e-mail, we are building something post, than Stripe got when announcing their company. Really speaks volume on Patio's writing.

Will there also be marketing challenges? Like: Every day patio11 clicks on one email. Your task is to write an engaging headline. Or: Here is some Google Analytics code. Make it show 50.000 visits any way you can. Very hard to cheat. (But so is getting Google security to file a bug report).

I really enjoyed the comments. Especially the ipod-hasnt-got-wifi criticism. You guys must know that what you are doing is terrible and you should feel bad.

Who knew decoding morse in some wave extracted from a corrupted image file in Cicada 3301 could also lead to a job. Will there be an ARG element to the CTF's? Like you pose as an agent from the NSA or something?

Well that's certainly one point of view.

So how you are working to make us all the richer? How can I join in?

I haven't yet cultivated any answers to my own questions to have fomented any clear ideas or call to action. I'm presently mired in Andrew Keene's The Internet Is Not The Answer, Alexis O'Hanian's Without Their Permission, and my own hunches and experiences from my lived experience and those of people I've met. Perhaps there's an essay in it somewhere.

Unfortunately there's no clean, quantifiable solution to human problems like labor relations that I can see. High-tech companies want to filter out potential hires to get the most highly-skilled candidates for the least amount of money they can get away with. They ultimately want to make more money with less capital and labor costs in order to maximize profits for their founders and investors. Which seems rather contrary to other engineering disciplines where, it has been related to me, they took in the hires they came across and made them into the engineers they needed.

Maybe it's a British thing, but the breathless enthusiasm starts to seem a little weird by about halfway down the page.

Hidden in the article is a real gem:

The science of hiring practices is settled: work-sample tests are the most effective way to assess skill in potential hires.

This is one of the most obvious, yet under-applied truisms in hiring.

If they can apply Starfighter beyond Games, or enable non-Game skills to shine while working on Games, it will be industry changing.

Makes me miss prop trading, where a tax return basically settled it. Now, job hunting ruins a weekend.

>Unfortunately, the technology industry is fundamentally unserious as to how it presently identifies and employs engineers.

Ok, so lets replace whiteboarding ancient algorithms and useless trick questions with a capture the flag game!! That'll make the industry seem less "unserious."

This is so unfathomably ridiculous. I can't believe you're marketing this as an interview replacement. If I get asked what my "Starfighter score" is before a job interview, I will be running as fast as I can in the other direction before they finish their sentence.

There will be people who become great at this game, and if (god help us) this gains momentum, they will get jobs.

I can only hope the rest who are developing applications and actually writing programs will come out on top, and employers will recognize that some people want to build things and not just improve their ranking in a game.

Nobody is going to ask you what your "Starfighter score" is.

Without opening up a pointless vein of second-guessing about Patrick's writing: I think the timing of this post has led people to believe that this right here is my answer to The Hiring Post.

No. The Hiring Post itself contains what I believe the answers to The Hiring Post are.

There are two halves of the recruiting problem: OUTREACH and QUALIFICATION.

Companies need to be smarter about qualification. We have a lot of thoughts on how they can be smarter and I am in full-on insane street preacher mode about those thoughts, which are not directly remunerative to our business.

Starfighter is about the OUTREACH side of the recruiting problem. Our goal is to be a credible pipeline of candidates who are not effectively identified by the resume/interview/github process. We are looking for buried talent, and to have fun doing it along the way.

I know HN all too well, and I think the instinct people will have reading this comment is to try to reconcile it with exactly the words Patrick chose to use in his blog post and then object to the perceived discrepancies. Please, no: that's a waste of all our time. Patrick speaks for Erin & I. I speak for Erin & Patrick.

Thanks for your well thought reply to my regrettably snarky comment. I agree with you on many points, particularly how companies need to be smarter about qualification.

But yes, this post does seem to directly correlate, in timing and in content, to Patrick's Hiring Post. I'm surprised to see you say this is not your response to that, especially considering that it addresses many of the questions we should ask ourselves as outlined in The Hiring Post (namely consistency, data, and the idea of one being a “natural”).

Also the post itself is linked in the statement:

>We’re not here to fix the technical interview: we’re here to destroy it, and create something new and better in its place.

I disagree with the fact that outreach is a problem in recruiting. There is an entire industry of companies and teams created for the very purpose of reaching out to software developers for their skills. How many of us reading this thread get multiple recruiter emails a week?

Are the recruiters effective? Maybe or maybe not, but I (who am apparently one of seemingly only few) just don't feel that a competitive game needs to add itself as another middleman in the realm of software recruiting with the intent of "destroying the technical interview."

Awesome, I've been waiting for something like this

That being said, I've got a couple questions

Will all the CTFs be security focused?

One of the most common problems coursera ran into was people copying the assignments of other people. How do you intend to prevent copying for the CTFs? I see that you mentioned you're not going to DMCA tutorials or Let's Plays and instead track their every iteration on the code. I guess you can get rid of outright copying that way but I'd love to hear your other strategies.

Other than that this seems like a really interesting way to do recruiting. I can't wait to see the actual CTFs.

Also you've got a tiny typo in the WHY IS STARFIGHTER THE RIGHT TEAM FOR THIS? section: "I have a folder in Gmail saving messages from geeks who used by career advice or salary negotiation tips to their advantage." I think the by should be my

I'm sure you've got your own ideas for your CTFs, but if you haven't heard of them before I would recommend you check out the Matasano Crypto Challenges (cryptopals.com). They ran them privately for awhile, you just emailed them and said you wanted in and they sent you the challenges, once you emailed the answers for one batch they gave you another, etc. I've tried various 'code gaming' things before, and the Matasano challenges were, in my opinion, the best ones I've come across. (I am exempting things like Stepic's bioinformatics challenges since their main goal is to expand your knowledge of bioinformatics rather than of programming, though they do get into some advanced data structures later on)

We wrote the Matasano Crypto Challenges.

Besides cryptography, what are some other programming problem domains you'd enjoy playing with? We're particularly interested in problem domains that are hard for ordinary programmers to pick up on a whim. We want to make it possible to seriously engage with interesting problems while in your couch with the Daily Show running in the background.

I guess what sort of domains do people like to play around with in their own time? For games there's 3d rendering, physics and AI which seem fun (at least to me) but I'm not sure how to necessarily make fun challenges out of them.

For non-direct-games things fancier AI like deep learning, audio synthesis, making compilers/parsers/transpilers, math stuff like project euler, puzzle/board game solvers/bots (https://gist.github.com/christopherhesse/51e9baf0e3d440d8aff...) come to mind.

Some of the stuff in the 3rd stripe CTF was pretty interesting like making a search engine or dealing with distributed system consensus problems (raft).

Maybe using the coolest new framework is fun for some people, like a challenge could involve angular.js so you could get some experience using it without having to go full-on side project or put it in at work without having any real idea if you should.

Implementing things that you are familiar with but that you don't understand the internals of has been fun for me in the past (for instance like https://www.mikeash.com/pyblog/friday-qa-2015-02-20-lets-bui...). Maybe you always wondered how this thing works, and now you can figure it out and solve this challenge at the same time.

The founders of Starfighter were the founders of Matasano.

That looks like fun. Thanks for pointing it out -- and thanks, tpacek et al for producing it.

Sounds very competitive, achiever-focused. Do all organizations want programmers who are viciously determined to win? Does that lead to society-efficient solutions?

(To elaborate a little, Richard Bartle noticed four types of players of MUD. Achievers are just one.)

We want the Explorers more than the Achievers, but we want to exploit Achiever impulses where they exist to keep the game interesting.

IIRC, The three other types are Explorers, Socialites, and Griefers

The 4th is killers, not griefers. Killers prefer playing specifically against other players. Killers are generallyl looking for the challenge/reward of fighting and winning against other players, rather than necessarily trying to grief someone. Though obviously getting killed in PVP can sometimes feel like griefing.

Very cool guys. As a long time HN reader, I have a lot of respect for what you guys are doing. I also spent the last two years of my life building CodeCombat, which started life with a very similar mission ("a game that finds developer hiring leads"). We were YC W14 and roundly failed at that business model. I just sent Patrick an email at his Kalzumeus email, but I'd be happy to chat with you guys about we learned and potentially connect you to others who can help you avoid the most common mistakes. My email is in my profile.

Hum, Thomas and Patrick working together... That by itself is worth keeping an eye on.

I didn't do so well on the microcorruption game. I'm curious to see what else they can come up with, hopefully both less specialized, more generic, but also more advanced...

May be interesting. But the claims made in the announcement are very hard to believe. The admission of not being ready to show that the claims have been achieved aren't helping my scepticism either.

Still... if it works, and is actually better than most other attempts at code-gaming this may be fun. Profitable? They will have to see.

The origin story of this company is the application of exactly this process in a real business, to great effect.

But it worked in a very specific context (security) and a specific company (Matasano), no? I know Stripe has done these CTFs in the past (and mostly security related), but they only spin them up on occasion. If the promise of such a technique is so effective, then why wouldn't Stripe invest the money (they've got plenty, let's be honest) in doing this all of the time? Especially given that a recruiter who would place someone at Stripe is probably taking home $20-40k on a placement.

Also, what are the plans outside of security related (if any)? Can you do a CTF style process for non-security?

The answer to the "why hasn't stripe done this" question is simple. They aren't in the business of doing it. You might as well ask why stripe doesn't sell database systems. I mean, they probably have some expertise in building those out as well.

Stripe is going to hire a couple of hundred developers over the next decade (assuming things go well). It sounds to me like Starfighter's goals don't translate to a couple of hundred X 20-40k. They seem a touch more ambitious.

I'm not looking for a job and am unlikely to be, hopefully ever again.

Will this be structured so I can use it as a proxy for a programming mentor? I think it would be awesome as a replacement for MOOCs, which I enjoy but find inefficient for the time invested. Books are efficient but not as much fun.

You might want to check out http://www.freecodecamp.com/ . I think they do pair programming

You might want to talk about the diversity benefits of this approach. In particular, it would be attractive to lots of companies if you can make a credible case for this attracting traditionally hard-to-reach demographics, and then of course assessing them more fairly.

I get that this is work sample testing - but it seems to be for an extremely limited work sample.

Most / much of the success of a valuable developer is not in their specific technical talent, but in areas like expounding a vision, documenting, running a team, navigating politics and fund raising. (Bit vague here I admit, personal experience of success is problem for me)

Patio11 made a name for himself saying "tech is a necessary but not sufficient condition for business success - now learn about basic marketing"

Something similar seems to be applied here - how can a CTF program ever handle politics or investor problems or documentation issues.

Getting good technical capable people is important - but not sufficient.

Will be interested in seeing the CTF program however ... Good luck.

> expounding a vision, documenting, running a team, navigating politics and fund raising

It sounds like you're mistaking "developer" for "tech startup founder". Someone looking for a developer needs at most 2 of those skills (documenting for sure, navigating politics in an organization full of bureaucracy).

This whole interview process is broken. Even with this mind of ctf game, one should not be expected to race through peers with different kinds of tools.

Developers are expected to build resumes at work via daily routines, then at home via github, hackerrank, side projects. System administrators are expected to do daily system administration routine, know OSes, tools, cloud/iaas/paas providers inside out and code side projects and build a coding portfolio.

With years of experience, one expects to be respected and not lose time on games, tests, unwanted side projects. Interview process is built towards young gals/guys.

What race/competition dynamics we are going to have are for the benefit of the participants. Microcorruption has been running for over a year; the first finisher finished a few days after we released it. I'd have paid very close attention to someone who finished Microcorruption yesterday, if I was still hiring for Matasano.

I understand what you are trying to do, but that's another competition in the roots. We already have too many.

GitHub, BitBucket, HackerRank, TrueAbility, Certifications, Education, Jobs. It's becoming of a norm that if someone puts more personal time to show off, they are better engineers. Also that's another wrong position in IT. You can not have personal life. I do not want to be a marketeer, I'm an engineer.

If you ever ask any of us around here, perhaps most will tell you a scary story about the competition and the time constraints the competition brings on people.

Eventually with every racing/gaming/ranking tool out there it's becoming to be harder to get a good position in IT without investing in more and more personal time.

I suspect that the Starfighter crew's answer around this is twofold:

1) Of course there will always be other ways to get technical work. You needn't feel compelled to use their product unless you think the time input is worth the reward.

2) If they haven't made the process fun, that is something you want to spend your valuable free time on, they have failed (at least in acquiring you as a product).

Perhaps you are right about the answer. But It is still another knot on the rope.

The next company expecting me do some obscure online automated coding test will need to pair me with an engineer inside.

That's not how this works. We don't host challenges for other companies. We're an outreach project. It costs companies money to find people through us; skipping us and going right to employers saves them money. I don't expect many employers to ask candidates to go to us to make them more expensive.

And, by the way, if you don't have a great resume and you're looking to work in a field directly related to the CTF we're running: making you more expensive is exactly what we're going to do. :)

So will there be profile scores? Will any company or person able to see how a participant is doing in StarFighter? If it is so, companies will use it as a differentiating factor in the hiring process. And that's what I'm talking about.

By the way nowadays shiny resumes are not considered relevant. It's the bottom line of screening. I have a great resume in my country and I am expensive here. But it would not matter for any SF startup.

I think you are misunderstanding the incentives that are setup via the Starfighter business model. You are thinking of them on the QUALIFICATION side of the pipeline. That is, you have entered a companies hiring pipeline and they are using Starfighter to determine whether you should stay in it or not.

The Starfighter team is trying to change the ACQUISITION side of the pipeline. They are stepping in to the space where traditional recruiters feed applicants into the pipeline. In marketing terms they are generating leads. Their value proposition to the employer is that the leads they generate will be of higher caliber than the traditional contingent recruiters. Their value proposition to you is that Starfighter is a) fun and/or b) at least better than dealing with a traditional recruiter. Your Starfighter experience replaces your resume/cover letter that gets you into the pipeline. Not your ability to navigate the pipeline once you get into it.

If your resume/traditional recruiters/responding to web ads/word of mouth is more valuable than Starfighter CTFs fun - hassle + employer payoff than you are quite simply not a good fit for the Starfigher Recruitment agency. Just like you wouldn't be a good fit for a DBA centric recruitment agency if you weren't a DBA.

If I'm disappointed about anything in the announcement, its that it is obvious to me anyway, that they are NOT trying to replace the current qualification steps in a hiring pipeline. Because to me, getting rid of that time wasting/unproductive process is the big value add, not shoveling more devs into the top of the pipeline (management at my company may feel differently).

No, performance and participant identities will be public only to the extent participants want them to be. Again back to our incentives: it does not in fact work to our benefit for any random employer in the world to be able to query our site to qualify a candidate. Those companies are free-riding off our work. :)

We need participants to make the game fun (for all of us). We can't do things that make participants wary of us.

This is fuzzy because I didn't let Patrick write about what the game was (that's entirely we're-about-to-ship-itis). It'll be less fuzzy soon.

Thank you.

I may seem to be against StarFighter, but no. I'm not. Actually I'm on the mailing list and waiting for the service to be offered. But I am not fully sure that I will be a good contender. I have not been a good gamer for a long long time.

I believe you fully understand my concerns. It's about the interviewing process, not the tools. Every tool has it's uses and quirks and every institution may use the tools in different ways, intended or unintended.

I really hope everyone doesn't feel like they have to be a "contender". It's not a competition to get our attention. It's much simpler: we have a basket of skills and concepts we want to let people play with. For people who engage with that stuff and find they really enjoy it and latch onto it, we happen to believe we'd have good ideas on how to match those people with jobs. That's it!

This "best of the best" stuff was meant to be empowering, but in a lot of ways it didn't come off that way.

I described our goals a little clearer here:


How about a game to discover and formally describe new problems? The ability to find new problems is just as important if not more important than the ability to solve already known problems (especially those that have been solved by others) Obviously, the ability to do both (find new problems and solve them) is the most important of all. Focusing on solving known problems is NOT that interesting... Can you imagine building an environment that allows programmers to discover and formally describe absolutely novel problems in computer science? Just curious.

Exciting stuff! It's ambitious but with the talented team behind it I'm confident they can pull it off.

> We come here not to serve technology recruiters, but instead to replace them with a small shell script.

Is there any protection planned against people who would be paid to solve the puzzles on someone else's behalf, i.e. against cheating by hiring someone to "train" your account?

Will you (patio11, tptacek, elptacek) apply to YC someday?

That would be uhm... i don't even know the right adjective.

Honestly they don't seem to be a good fit. I mean they are going to get in, no issue, but they already know how to run a business (charge more) and they can pretty easily get access to investors.

The more interesting question is if they will ever end up teaching at YC.

This looks fantastic. One possible wrinkle: I predict that, although top-performing Starfighters will be a diverse lot, they will lack "diversity". I hope the Starfighter founders have a plan to push back against the carping complainers a venture such as this is likely to attract. (My suggestion is to point out that CTFs are the ultimate expression of "Shut up and show us the code." [1])

[1]: http://esr.ibiblio.org/?p=6642

I don't understand your point. I've worked with mediocrity far too often in the programming world, and outright incompetence as well. These kinds of contests/games are designed to weed that out.

If you're talking about the "winners" having personality issues, then hopefully that will be caught in the interview process.

Perhaps I'm completely missing your point though.

I think they were suggesting that the distribution of the top performers might not be "diverse" (for some meaning of that term), which would lead to criticism (like how people criticize companies for not having a diverse workforce, or conferences for not having a diverse speakers list).

I am a beginner programmer, but I am curious what this is all about so I have signed up. Hopefully I can learn something from this even if I am just am just a fighter but no star.

This is neat, but is it really that hard for people to tell who has the skills they're looking for and who doesn't? Every developer I've interviewed it's been really obvious to me what they know and what they're bullshitting about.

I guess there's value is all in assembling a pre-qualified pool for employers to recruit from, though.

But in that way this is kind of like the 2015 version of CNE. Except it's free, which is good. And hopefully less miserable.

If you read http://sockpuppet.org/blog/2015/03/06/the-hiring-post/, you will see that this approach is looking to find folks that were not obviously good, as you say, but they found someone that is a stellar crypto breaker.

Many on HN, including myself, are very excited about Starfighter. However, many HN readers are probably already employed or can easily find employment if they desire. You guys and gal know that maximizing outreach will be critical to your success. You aim to find buried talent, but that talent must first know Starfighter exists and understand it well enough to be enticed to try. I assume you want to both find talent whose ability is not accurately represented in a typical technical interview, but also those who have aptitude for these skills but whose resume wouldn't even be considered.

I grew up near a small town surrounded by family farms. I've really enjoyed manipulating code and finding bugs in games and other software since middle school, and I suspected I was the only one when I was younger. There were very few "nerds" at our school who focused on computers, and even those that did almost never discussed the discovery or exploitation of bugs. Although I stumbled upon a few communities online that shared my interests, I hadn't found one that just clicked. I happened upon HN while learning more about startups, and it was just right for me. I'm grateful I found HN, but I wonder why it took me so long to find the community I was looking for.

I didn't even know about CTF games until tptacek's recent hiring post, and it was another "How could I have been oblivious to this awesome thing for so long?" moment. There are many people who are a great fit for your clients, and they won't know about Starfighter unless you cast a wide or deep enough net. How do you plan to address this challenge?

Good luck, and thank you for building this!

I actually began working on the exact same thing within the last month or so in my spare time.

Having competed in the CCDC for 6 years in college, I found it pretty insane when I saw that employers don't use real hardware during interviews.

Looks like I picked the wrong side project now this team is working on it. However, interviews are broken enough that there is plenty of room in the space =]. Can't wait to see what you launch.

interviews are broken enough that there is plenty of room in the space


Looks like I picked the wrong side project now this team is working on it.

Competition is a signal that you picked the right field. Just about everybody gets this backward.

The next step could be to crowdsource the design of challenges or puzzles. Like a 'masters tier', where you get special points for designing particularly clever puzzles. This could help remedy the constant demand for new puzzles due to writeup contamination.

BTW, I completed microcorruption, foobar and two stripe CTFs. It was lots of fun and I am really looking forward to Starfighter!

And then there were four.

This theme (test your coders, don't interview them) is spreading about, I keep getting spammed by them as a way to "only hire top talent!" its an interesting proposition and I can't wait to see a bit of history behind it. At the very least it will weed out people who think they are programmers when they are not.

It might employ programmers by people who are not.

I'm excited; tptacek seems to have put a lot of thought into the hiring process and approaches to improve it. I'm in.

Congratulations patio11, you sound excited.

I am curious to see how this will work for those like yourself that make CRUD apps. I didn't go to school for programming, but self-taught. I'm afraid it'd be over my head?

The best line: We haven’t done any consulting in a while. We will continue doing no consulting, to the best of our inability.

I think that anyone motivated to research and work on these problems, will likely be a decent hire in most organisations.

Having done other CTFs and puzzles like Matsano / Stripe's, I have found the research part is quite enjoyable. I have little to no knowledge of crypto but researching and solving some of the Matsano challenges was extremely rewarding.

With that said I found the information is pretty thinly spread and it is often hard to find relevant information to the problem domain, particularly with regards to crypto, so I hope they include good starting points for problem research.

tptacek: ever heard of Cicada 3301? It's this mysterious group that has anonymously posted very involved cryptopuzzles. Not clear why, but one theory has been for recruiting people into intelligence agencies.


I love what you're doing but I agree with other commenters that presenting it as a game to programmers but a recruiting channel to companies might not align well. What do you do if a lot of your top players aren't interested in getting a job? There are probably exceptions, but do you think people who are good enough to beat your CTF have that much trouble getting a job?

---> VERY YES. <---

I'm not sure how this solves the topcoder problem. That problem being is it attracts only certain kinds of software developers and while it is used as a recruiting tool it basically changes nothing when it comes to the interview and hiring process.

Even if you are recruited on topcoder you sometimes still need to submit your resume to some online resume eater "to get you in the system". And then you go on the same time wasting interviews as everyone else.

Personally, I don't play games of any sort but I like to write software and solve puzzles so I find topcoder SRMs more appealing than a programming game, but I'll definitely check it out.

Topcoder challenges were not at all relevant to the kinds of programming problems I needed solved, and were not credible enough for me to consider whether they were predictive.

Don't get me wrong, I would love for the current system to go away, since every interview I had got me one step closer to an imminent hear attack. But, I think the most we will get out of Starfighter is that it will become just another step in the process, MAYBE replacing the phone interview. Nothing is preventing a group of people from sitting in front of the computer, collaborating on a solution, there is absolutely no way to catch that. Also, none of the companies that are most in demand will abandon their dreaded on-site interview any time soon. It works for them, they need to weed out as many people as possible.

Reminds me of a local company who recruits employees exclusively by organizing coding contests: http://contest.catalysts.cc/en/

It works great for them, but I believe you attract a certain kind of developers with these contests; you attract people who enjoy puzzles, and people who enjoy measuring their abilities against others. It could be that those are the traits that make good employees; but I myself would never take part in such a competition.

Then again, I don't actually want to work for anyone else at all.

So, we launched a long running CTF platform a few days back: https://backdoor.sdslabs.co. I guess more in the space is merrier.

My prediction is that Starfighter will become something akin to HackerRank where companies host their own CTFs to recruit people. While it sounds fine, making CTF problems is significantly harder than making algo problems. I'm sure the team is right for this, so it might just work.

I hope they do release some work in the open (such as their sandbox environment).

What skillset do participants have to have to play this game?

Is it purely security stuff?

Enjoy programming in any language.

Be fearlessly willing to pick up programming in new languages.

No security knowledge is required.

Security is a backdrop for what we're doing, but not the only backdrop; it is one of two problem domains we're starting out with.

We use security mostly as a venue for systems programming.

This has piqued my interest more than any other similar challenge/game I've seen in the past.

What's the second problem domain you're starting out with?

I hope it's to do with graphics/parallelized matrix math sort of stuff.

We'll send a reading list out to the subscriber list, sometime soon. It'll be pretty obvious from that.

I wonder if the authors are fans of Stargate Universe. The show's premise involves a key character being identified by success on a puzzle embedded in a video game.

I presume it is based on the film "The Last Starfighter" http://www.imdb.com/title/tt0087597/ - in which a teen perfects an arcade game, and is then selected as a pilot for the actual spacecraft.

I'm sure the name is from a much earlier movie (mid-80s), The Last Starfighter.


From the blog post it sounds like you primarily plan to be used as a "funnel" for employers that engage you, where you are like a super-recruiter bringing candidates that you have qualified yourself.

Do you plan for there to be an option for employers that aren't asking you to refer candidates to still see some view of the performance of a player? Either by paying some smaller fee or by allowing players to have public profiles with some sort of scores?

There is a long-term PVP element to what we're doing, so we want as many players as we can get. We'll play with a number of different ways of matching talent to very smart employers, but our #1 overriding dominating design goal has to be to make players comfortable, so the game works.

patio11: Why name the company Starfighter, while the domain is startfighterS(.io)? Why not name the business Startfighters? (I understand the other domain was taken).

I like this idea, but I also like the one proposed here http://ortask.com/a-better-way-to-hire-developers-and-tester...

I fear that starfighter will overlook goood candidates simply because they are not good at gaming, which makes the second approach by ortask more interesting and maybe balanced.

What do you expect low 700's rank as of now on micro corruption to equate in the new venture? Salary estimates?

I feel like this is an implementation of my idea here: https://medium.com/@sargun/interviews-as-a-service-dfa6d4a03... -- Or at least a competing alternative to traditional interviews.

Love the concept, can't wait to see it. I don't normally play computer games, but this sounds intriguing. One minor point: "Alt-tab over to your email and click the confirm link in the email we just sent you." If you use web-based email, that's probably Ctrl-tab :)

There's a small typo that caught my eye. In the 'Why is the Starfighter the Right Team' section, 4th paragraph, 2nd sentence "I have a folder in Gmail saving messages from geeks who used by career advice or salary negotiation ..." It should be my instead of by.

- Do you think there will be some threshold where people get past a certain stage and then start getting offers, with nothing being offered below that? (my only experience is basic OverTheWire challenges, so not sure if this will have similar progression of 'levels')

- Is it going to hurt?

Based on my experience with the crypto challenges and Microcorruption: there will be a lot of players, and most of them will just be there to twiddle the knobs and see how things work.

There will be a subset of participants for whom it's worth their time and ours to have a conversation. Our job will be to identify that subset and start that conversation.

Will the CTFs be security focused or more non-traditional, like Stripe's distributed systems CTF?

Starfighter is not a security company. We're going to test for a wide range of skill sets. These may include web app security but will not be limited to that.

You can reasonably assume "What skills does the market want to hire for?" is a good proxy for what we'll be assessing for. I love security, but that's 0.01% of the software market. We have... grander ambitions.

This sounds like the sort of thing that teenagers would come up with to "solve" recruiting.

I'm going to go ahead and take that as a compliment.

Yes, that makes sense.

Would like to try this as an employer, particularly targeted towards developers here in South Africa.

Is it targeted to devs looking for full-time work only or will it make sense for freelancers as well?

If the scores/achievements can be public accessed then you can use it as part of your portfolio when bidding on contracts.

Is the second problem domain web perhaps? So people can learn that <script> tags can not be self-closing, and so they won't break the JS includes so bootstrap.min.js doesn't load and their burger menu in mobile-view actually works!



This sounds insanely fun & reading this got me really excited, just subscribed a moment ago!

Wow, what an idea. And I couldn't have more respect for the people who started it. Nice!

Part of the current interviewing process is Github and side-projects. I truly understand the reasons for Starfighter, but are you putting any real value behind it? I like CTF, but it's not quite the same as contributing to open source projects.

You're quite right that a CTF is not the same as open source projects... and I thought the point of this was exactly that. People who contribute to open source projects prove they have a certain set of skills. What if you need people with a different set? What if instead of being able to learn a large codebase, it is more important that they are able to look at a problem and design an algorithm to solve it in the first place? Rarely in an open source project will you be going in and radically changing the fundamentals.

Makes sense. From that angle, we'd even see better contributions.

So this site is about helping recruiters narrow down candidates more effectively by presenting real coding challenges.

I want to point out trueability.com that does same thing with Linux. I find it a very good and fair source for judging Linux admin candidates.

The Last Starfighter is by far the most underrated scifi movie of all time.

"You will use real technology. You will build real systems. You will face the real problems faced by the world’s best programmers building the world’s most important pieces of software. You will conquer those problems. You will prove yourself equal to the very best. Becoming a top Starfighter player is a direct path to receiving lucrative job offers from the best tech companies in the world, because you’ll have proven beyond a shadow of a doubt that you can do the work these companies need done."

What an absolute waste of time. You can do all of those things as hobby projects which are yours and actually do something useful in the world besides exist as an achievement to be unlocked.

Show your future employer your github, not your starfighter score.

Isn't it about putting in the time ? How would kalzumeus know if candidate is doing it for the first time vs after 500 hrs of experience. I presume the latter would have a huge advantage

What sort of range of engineers will this be aiming to recruit? Only the very best (and most expensive)? Or a range including more junior / less well-paid positions?

This looks very innovative; I hope it's a great success.

So, apart from GitHub, we now have another opportunity for coding slaves to prove themselves worthy.

Spartacus, open your mouth, I want to see your teeth.

You buried the lede here: you're selling BCC! Somehow HN won't feel right without regular updates on BCC's progress.


What is going to happen with Appointment Reminder now?

The answer is at the bottom of the announcement: "I have no announcement to make about my involvement with Appointment Reminder at this time. We will, naturally, continue keeping all commitments to our customers."

Thanks - I just found it myself and came back to edit my question.

That's a non-answer though. Let's see how the answer about the fate of Appointment Reminder would evolve over time.

In any case, that's an extra proof that being solo-founder while possible - is a disadvantage.

Is it significantly different from topcoder?

Yes. I don't know how to describe how until you've actually played it. You know how World of Warcraft and Tetris are both games? Starfighter is a game, too. We're very much not Tetris. Or Topcoder.

The sign up form isn't accepting my email address: _@gmail.com

Edit: rather, (9 characters long) @gmail.com

This is awesome. Sign me up.

I wonder if this was inspired in part by the recent Alan Turing movie :)

How so?

It has a scene in it where recruits are approached using a cross-word puzzle in a newspaper followed up by a much tougher test.

There's a chuck of the movie where they're recruiting potential code breakers by putting a difficult puzzle in the newspaper and telling readers if they can solve it in under 5 minutes to apply to the job. Same sort of premise here.

Ah, cool. I haven’t yet seen the movie. Only a few weeks left till the video release :)

Awesome news, best of luck Tom!


"You will use real technology. You will build real systems. You will face the real problems faced by the world’s best programmers building the world’s most important pieces of software."

I'm wondering, what is your business model? Are the users doing free bug fixes and work for the employers while they are proving their skills?

No sane employer would want free bug fixes. Think of what that would do to their IP assignment issues alone.

Our business model is explained in the post, under the heading How Will Starfighter Make Money. Briefly: we'll find the best engineers in the world, then broker their introduction to companies who want to hire them. Companies will pay us a substantial amount of money for this service.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact