Hacker News new | past | comments | ask | show | jobs | submit login

I'm all, involved? and stuff? You can, like, ask us stuff, I guess.

Let me answer the most frequent questions we're getting right here:

"CTF" stands for "Capture The Flag". Conventionally, it's a contest with a collection of "flags" each of which is guarded by a programming puzzle; teams of people compete to collect flags. What we're doing is not a conventional CTF, but if you want to get the flavor of what we're doing (without the whole game dynamic), check out MICROCORRUPTION.COM, which is a more conventional CTF we ran last year.

Security is one of two problem domains we're starting with. But this isn't a "security recruiting" service, and our take on security uses it as a venue for systems and network programming, not for the minutia of SQL query quoting rules.




Thank you for running a long-term CTF ! I've always been interested in those "improve your coding skills by playing" challenges but from my inexistent experience most of them fall in a combination of:

- They are timeboxed

- They are mostly (if not only) about cracking security

- They are targeted towards low-level languages

You've already stated that it won't be timeboxed (which totally makes sense as a hiring "middleman": you're interested in applicants at all times). You already said that security will not be the single domain. Now, will I be able to use Starfighter as an excuse to finally stop procrastinating and learn that shiny new language I've had on my to-learn list for far too long ? (Patrick more or less hinted it shouldn't be the case, but I'd like to know)

I'm really looking forward to this. On the overall point of breaking the interview standard we have, I'd like to say a huge "Thank you". The points you've made in your blog post really resonate with what I can see (the interview process is a joke if you want to hire actual programming engineers). I really hope we can move towards a model where applicants can show skills through a portfolio, of which Starfighter should be a part if I understand things correctly.


If you haven't seen it before, check out https://projecteuler.net/ which is a series of (very) loosely connected programming challenges.

The challenges start mostly bite-sized and typical programming problems (think fizzbuzz or "what is the 100th prime") and grow into some really interesting areas. You use whatever language(s) you wish, with no time limit, and are free to skip around to whichever questions interest you (or just do them in order... whatever you like)

I find it fun. You might too.


Euler is more like math problems that can be solved with programming than programming challenges. Very neat, but decreasingly useful for programming practice past the first 20ish.


I always ask for the easier Project Euler tests (or give them a couple) to do; the reason they are good test (IMO) is that they are too small a task to leverage frameworks and can demonstrate things like good approaches to engineering and programming practice (even if the questions are mathematical)


It is possible to demonstrate good approaches to engineering with your solution. But if you're cranking out PE problems, you really are better off just throwing out a bunch of one-off unmaintainable scripts. It is a good test of math and cleverness. It is not a good exercise for maintainable software.

See http://bentilly.blogspot.com/2010/01/solving-project-euler-p... as evidence that I am not just talking out of my ass about PE.


If you are interested in a long-running CTF, we launched https://backdoor.sdslabs.co a few days back (with >30 challenges as of now). We also tend to have more beginner level challenges.

I'm also really looking forward to this, especially from the pov of Hiring. Ours is an academic initiative, with no plans to monetize ever and its cool to see patrick and tptacek picking up the mantle on such a task.


Your app is not accepting email with the new longer domain extensions. eg: .systems


Thanks. I'll take a look at the email validation code.


The only truly reliable way to validate an email address is to try to send email to it.


My collegues like http://www.codingame.com/ which is more game oriented than security and you can use various languages (Haskell,bash,Clojure,C,...)


As someone who plays in a lot of "conventional" CTFs (e.g. DEFCON, Codegate, GitS, etc.), this sounds very cool, even if my participation in those already gets me some job offers :)

One question: for a less "just for fun" site like this, with no time barriers, how do you plan on dealing with people just following write-ups? You can randomly generate the flags, but people could still follow the steps. You can say "don't write write-ups," but write-ups will still be written ;)


This is (a) a real challenge, (b) something we're addressing in part by not letting the content get stale, (c) something we were surprised by with the crypto challenges --- if you ask people to follow the honor system, they tend to do it! --- and (d) something that the game model will also somewhat address.

It's hard for me to go into more detail on (d) without revealing a whole bunch of stuff about the game I don't want to talk about yet; I should be more comfortable talking about it, but until we announce it officially I have a lot more leeway to slip rev1 features. :)

The shortest simplest answer though is: we're a firm whose whole purpose is to make fun, interesting CTF-style games (well, one game; we're the Blizzard of CTFs, and we're building our WoW), so we can address a lot of these kinds of problems with brute force, because this isn't a spare-time thing.

If it helps to understand where we're coming from:

Chris Eagle, the author of The IDA Pro book, published an IDA Pro plugin for the bizarro-MSP430 that Microcorruption (our last CTF) emulates. There are tools with "microcorruption mode" in them because of all the little ways we broke MSP430. Someone wrote a symbolic execution engine to solve the Hollywood level on Microcorruption and posted it to Github. There is still a #uctf channel on Freenode for Microcorruption.

This. Is. Awesome. It is my favorite thing about Microcorruption.

We did practically nothing at all to foster a community for Microcorruption, apart from Erin starting the IRC channel. That's not an opportunity we are going to miss this time; in fact, doing better on the community and sharing side is part of the thesis of the company.

I reallllllllllllly hope people share code and tools and stuff to make progress in the game. How cool will that be?


> "... if you ask people to follow the honor system, they tend to do it!"

This is great but I think the stakes are somewhat different if it's ostensibly about jobs/hiring. I don't recall the crypto challenges being promoted as a hiring mechanism (though they may have been useful for that).


The crypto challenges were how we (for instance) hired Alex:

http://sockpuppet.org/blog/2015/03/06/the-hiring-post/

I think we plan on making minimal demands of our users, and none of them involve grooming them for prospective employers.

Look, the reality is, most of the people who participate aren't going to be looking for a job when they do. So all our incentives are to make the experience itself rewarding to participants.

I'm weird about typing those words because very very soon we're going to actually ship the first rev and levels of this thing, and as anyone who ships software knows: right now, at this point in the release calendar, my instincts are to be LOWERING the bar, not raising it. :)


> The crypto challenges were how we (for instance) hired Alex

But that's the use from your (or Matasano's) perspective. I joined the crypto-challenges not at all because I want a job in security, but because I continuously heard people be super-enthusiastic about it (both the participants, as well as yourself, tptacek :) ), because it reminded me of the old Malattia+ 3564020356 puzzles (level 6!), because it seemed more fun than the Euler Project puzzles (which I did enjoy, but you can only solve so many palindrome prime puzzles before it gets tedious) and of course because I would learn things about practical crypto.

Unfortunately I only got halfway the first set of the Matasano challenges, but that was more because I did it in Python and at some point got frustrated by its lack of speed :) (even using NumPy) I did make a rather elegant English-text MLE detector using a log-probability frequency table of only 256 bytes :) I thought that was pretty cool. I might have another go at it and this time use Java instead.


>I think we plan on making minimal demands of our users, and none of them involve grooming them for prospective employers.

>most of the people who participate aren't going to be looking for a job when they do.

This is weird, and I'm sensing some miscommunication between you and patio.

I'm only hearing about this as a tool for proving my worth. Your major marketing (as far as I'll likely ever be aware) has CAREER CAREER CAREER stamped all over it.

People won't follow the honor system if the stakes are at the 'career' level. You're losing the 'fun and free' culture of Microcorruption that makes people spend their free time building fun tools.

Your understanding of the incentives differ from the incentives that have been communicated to me about Starfighter.


Patrick.

You're fired.

Anonymous Throwaway Account? Yes, you over there. I'm looking RIGHT AT YOU. Yes you.

You're the new CEO.

Get to work. Explain this to everyone else on HN. The clock's ticking!


Just giving feedback as someone that's theoretically right square in the middle of your target audience. I really want to take this 'college course' and take advantage of what I see as your opportunity as a security expert to make your knowledge replicable.

But I'll play along. I'll be taking Starfighter in a new direction. Most notably, we'll be reorganizing how we react to online discussion. Anonymous critics will be summarily executed unless we cannot identify them, in which case they shall merely be barred from Starfighter for life, which, if you believe our marketing department (AND AS CEO I DO IN FACT I'M SORRY I IMPLIED IT WAS POSSIBLE TO DOUBT THEIR CLAIMS), will make it very difficult for them to find work in the hiring utopia that is the post-Starfighter process.


While this might indeed help employers find better employees, I am concerned about hidden (or not so hidden) bias. Games like this (and the way this one is described confirms it) select for competitive people. Other talented programmers might prefer expressing their creativity and talent in less competitive ways. While competitiveness might be a desirable quality, sometimes it is not. It might also appeal to men more than women, and so might be inadvertently sexist. I think you should very carefully analyze player demographics to make sure you're not making matters worse in terms of diversity.

When you design a test -- any test -- you should make sure that 1/ score correlates with success (i.e. full bivariate correlation), and 2/ if the test is voluntary, that there are no prior biases which would select candidates prior to taking the test.


Best answer I can give was that we've heard that concern from a bunch of different sources, we share it, and we're designing against it to some extent.

As a reminder: this is an outreach strategy we deployed at Matasano to enormous success. We had an English professor finish the crypto challenges. We didn't solve all our diversity problems, but we made a palpable dent in them, and we did that by coming up with something that surfaced aptitude that wasn't held hostage to the biases of random human interviewers.

There is, to my mind, no hope for the "random human interviewer" hiring strategy. It's hard to make the problem we have now worse. But we're watching out for it.

I would love to hear more thoughts on how we can address this at the level of game design! We're at a "finishing touches on infrastructure, just starting with level design" place in our design right now.


Somewhat relatedly, as a person who's discussed these issues with you in the past, it sounds like you'll be gathering new evidence. How will your beliefs change?

For example, suppose $GROUP is over/underrepresented among top starfighter performers. What will be your new belief about $GROUP's technical talent and the causes of over/underrepresentation in tech in general?


I don't think my beliefs about the facts are any different from yours; we disagree on the causes of the facts and their mutability. Therefore, my beliefs will change when the dynamics are tested. But this kind of data provides evidence for our disagreement no more than a still image of a flying arrow can settle the one between Aristotle's and Newton's theories of motion.

Somewhat relatedly I'd like to add that while the result of an experiment in dynamics will obviously change my theory, it will in no way change my values[1]. Privilege based on an immutable characteristic such as intelligence (assuming such a result were to be obtained) is no more arbitrary than one based on bloodline. A smart person, though she will obtain it, deserves no more power than the average bloke, just as a nobleman, though he will obtain it, deserves no more power than a commoner. This is one reason the original meaning of "meritocracy" is satirical[2], as it does not change anything other than for the worse, by making the wielders of power believe that they actually deserve it (like the nobility in ancient times but unlike more recent ruling classes such as the American WASPs).

So far I have found that people who believe that the variance of some traits such as intelligence between population groups is dominantly the result of genetics do so because they think it provides a moral justification to the social order: things are as they ought to be because nature dictates so. I see no connection between the two. Nature (if it is, in fact, at play) has little bearing on ethics[3], and thus can, at most, explain but never justify an unfair distribution of power.

[1]: http://en.wikipedia.org/wiki/Fact%E2%80%93value_distinction

[2]: http://www.theguardian.com/politics/2001/jun/29/comment

[3]: http://en.wikipedia.org/wiki/Naturalistic_fallacy


Tom and I disagree on how much of the underrepresentation of women in tech is due to skill deficits. His new platform can answer that question quite effectively.

My disagreement with you stems from the fact that I don't believe in any privileged population groups - I only believe in individual rights. I may or may not disagree with you about "power" but so far you've yet to provide a clear definition of it. (I did read your wikipedia links, but they provided multiple disparate and unclear definitions.)


> His new platform can answer that question quite effectively.

Assuming his game tests for skill and skill alone. Though even if so, it wouldn't explain the difference -- just report it.

> I don't believe in any privileged population groups - I only believe in individual rights

I don't understand. One of these things is normative (individual rights) and the other is positive (privileged population groups). The existence of privileged population groups is a matter of fact[1] -- no one thinks they should (normatively) exist. As to individual rights -- everybody believes in them, too. The question is what would be their nature. For example, I believe that if the wealthy were allowed to wield their power (money) over the poor unhindered, then the poor should be allowed to wield their power (numbers) unhindered over the rich as well. The point is that power, by definition (see next paragraph), means restricting in some way the freedom of others, so to obtain freedom you must either restrict all power or unloose all power.

> they provided multiple disparate and unclear definitions.

Perhaps, but not different enough or unclear enough to preclude study or reasoning. The gist of it is, power = the ability to bend (or sway) others to your will. Power is measured by how many people you can sway, and to what degree you can sway them.

[1]: For example, that white men are more privileged in America than black men is a fact.


The existence of privileged population groups is a matter of fact[1] -- no one thinks they should (normatively) exist.

As a person who cares only about individual rights, I don't care if membership in some particular group is correlated with lack of privilege.

If you want to argue that lack of privilege is an individual injustice, fine - but then you need to stop discussing race since there are plenty of privileged blacks/women and underprivileged asians/males.

...not different enough or unclear enough to preclude study or reasoning.

You assert that white men are more privileged than black men. The definitions you've provided are insufficient for me to concretely state a test we could run to to disprove that.

For example, being black will sway college admissions officers for you but police against you. How do any of these definitions allow me to say that on balance, these things are negative? At what magnitude would the balance become positive?

Anyway, this is completely tangential to starfighter.


> I don't care if membership in some particular group is correlated with lack of privilege.

And what if lack of privilege is caused by association with the group?

> If you want to argue that lack of privilege is an individual injustice

You may believe in individual rights, but you can't deny group injustice. Blacks were made slaves not due to any individual selection.

> For example, being black will sway college admissions officers for you but police against you. How do any of these definitions allow me to say that on balance, these things are negative? At what magnitude would the balance become positive?

You can't possibly be serious. But just in case you are, there are clear tests to measure power: money, positions of control in the private and public sectors, positions of control in the media. QED


I don't deny group injustice. I don't believe injustice based on group membership is worse than any other kind - slavery is an injustice even if it's equal opportunity. Insofar as having bad schools or bad parents is not an injustice, it's also not an injustice if some group has more of them than another.

And insofar as such a thing is an injustice, the fact that it's correlated with some group is irrelevant to me.

If the only way power can be measured is via outcomes (in this case, I guess a high developer salary?), then I'm not sure why we need a new word to describe it. I also don't get what your point is. I guess you are arguing that smart people don't "deserve" the money/control that comes with a developer job any more than dumb people, and therefore starfighter is a bad thing?


Well, group injustice is just individual injustice done to many individuals. And, I think, you would not consider it unfair to demand a robber to pay back the money he's stolen. The problem with underprivileged groups is not the reality of the offense against them, nor the identity of the victim, but the identity of the perpetrator, and this is where study and of power comes in. Because the reality is that human society, like gas molecules, performs many acts not as individuals but as a group (even collective property preceded the invention the first private property), and therefore the perpetrator in the offenses of underprivilege is society as a whole. The actions under discussion are not as acts of nature but acts like for men, acting as a group (and collective action is a positive fact). Assigning individual responsibility is as futile as assigning individual responsibilty to specific gas molecules in expanding a balloon; doing so is just an ineffective model. And what society has stolen as a group (from individuals!) it must pay back as a group. Identifying the victims as a group is just a matter of statistical convenience as well as an aid in the description of the dynamics.

That an electron is measured through its effect does not mean that we don't need to describe the electron itself. In fact, it is crucial that we do in order to understand its action. Same thing for power: you almost never observe it directly, but its study, and the term itself, are required in order to understand the workings of society.

And as to this game, I don't think it's inherently bad at all, but if its role in society is not studied it might become an unwilling accomplice to injustice. And, if, one day it is somehow discovered that abilities that convey power are dominantly genetic, I do not think that we should give certain jobs to people unsuited to them, but our society is judged not by the achievements of those born to privilege, but by how it takes care of those who lack it. Exactly how this moral obligation should be carried out is a complex matter in itself, and far beyond what I can write here.


That an electron is measured through its effect does not mean that we don't need to describe the electron itself.

We certainly do. So look at what physicists did. First they came up with a clear definition - "a discrete and indivisible negatively charged particle". Second, they went out of their way to distinguish concepts like an electron from charge as a fluid and other models. They didn't simply declare "well, electric current proves electrons exist", they went to crazy lengths like Millikan's oil drops to distinguish these concepts.

When they can't actually distinguish these concepts intrinsically (as was the case with Maxwell's equations vs Aether), they tend to drop the more complex theory.

In contrast, you seem quite resistant to doing any of these things. I don't quite understand why.


Resistant to what? I just gave you a clear and concise definition of power (the ability to bend others to your will). The social sciences did all of the required work as well, I'm just unable to write a whole sociology book in HN comments. There have been countless studies in sociology, history, anthropology and psychology examining the different forms of power and how it works. But bear in mind that these sciences are much more complicated than physics; in fact they're intractable. The simplest social mechanism is more complicated than the gravitational interactions of 1 million bodies.

Actually the example I’ve give of the electron is a rather weak one, as the electron is a very specific thing, while power is the most fundamental concept in all the social sciences except psychology (i.e. history, sociology, anthropology and political science). A better example — the obvious one, in fact — is energy. Energy can also only be measured through its effects, and yet it is a very fundamental concept. Describing it not as one thing but as disparate manifestations would take away its explaining power and many of our most useful models (I should really write a book called “social science for physicists”).

I’ve also given more thought to your focus on individual rights. Individual rights are an obvious “good thing”, but here’s where they get complicated: When I think of individual rights I imagine a universe composed of mass but no force. Such a universe will be no more than a cloud of plasma. But in our universe, mass gives rise to force, and force creates the interesting interactions that have, in turn, created our world. Likewise, human society is not made of humans, but of human interaction, interaction gives rise to power, and power, by definition, restricts freedom. Now, this is not a bad thing necessarily, as, if you think about it, all cooperation is basically the voluntary yielding of freedom in order to concentrate power for some common goal that wouldn’t have otherwise been achievable. But that doesn’t change the fact that whether they like it or not, just as mass-ful particles induce force, humans induce power, and both force and power take away freedom from others. So saying something like “everyone should be free” makes little sense, as that is only possible in a plasma cloud. In order to grant freedom, freedom has to be compromised — based on some values — and then managed somehow. Obviously, different people will prefer different compromises. Personally, I’d either like to see all power restricted and controlled, or all power unrestricted (including physical violence).

This is why I think that American libertarianism is either hypocritical or ignorant. It is either ignorant of the fact that there is no freedom without power (and hence, coercion), or hypocritical in calling for unrestrained use of certain forms of power alone (money) and not others (physical violence, preferably mediated by a democratic government that restrains the use of money).

I was therefore delighted (intellectually, that is) to learn recently of a fringe Silicon Valley movement called neoreactionism or “Dark Enlightenment”, funded by Peter Thiel. These guys (few women would join that openly sexist, openly autistic movement) are probably all former libertarians that have discovered that there is no such thing as freedom from power, and now openly call for a tyranny. As someone who’d studied medieval history in graduate school (though I have never obtained my doctorate) I was delighted to see the movement’s leader, a programmer by the name of Curtis Yarvin, analyze some historical document and call for the return of feudalism (he complains that those documents are not studied by historians, which is true for the simple reason that they are false accounts).

The problem with the return to feudalism — even if you were to believe the false accounts of how life was good then (it by no means was) — is that the power structure back then was at least held in check by technology, that is, lack thereof. With the invention of mass media and fast transportation, power can be more concentrated than ever before, which is why the greatest invention of Western civilization was the central government, which rose to contain and manage power (of course, this only made conflicts among those governments more violent than ever before, but that fear of annihilation reduced the number of conflicts considerably). BTW, the modern academic definition of politics is, not surprisingly, the management of power in society.

Feudalism combined with modern technology has only been tried — to the best of my knowledge — once, in nineteenth century America. The US at the time had a very weak federal government with almost no regulation. The result was a period of extremely fast economic expansion but at great social cost: a large portion of the American population was enslaved in all by name by a very small number of slave-owners-in-all-by-name known as the Robber Barons (Rockefeller, Carnegie, Mellon, Stanford, JP Morgan, Frick, et al.). All options were taken from them — they couldn’t migrate (they were sometimes paid in company-issued currency, that was useless anywhere else) and couldn’t organize to concentrate power to improve their lot (in fact, they did organize, but the robber barons had private armies that killed the rabble rousers and intimidated everyone else). The people cried for help, and Theodore Roosevelt rescued them by creating federal regulation.


Energy is also well defined - for any particular case you'll be able to write down a precise formula defining it. For example, a system of colliding particles has E=sum(p^2/2m) (p is momentum, m is mass).

If you want to make a claim that one system has more energy than another, testing that claim will be easy - just apply the formula. If you wanted me to take it seriously, you should have applied the power formula to figure out whether influencing college admissions officers > influencing police - all you did was scoff.

I assert that your use of the word "power" is pointless. It transmits no information about the world, much like how you use privilege. Why do you insist on engaging in long discussions advocating the use of words that mean merely "any probabilistic cause of social outcomes"?


Information can most certainly be conveyed outside the use of precise formulas, lest you think that entire fields of human endeavor are bunk (merely "long discussions"): art; music; aspects of philosophy, anthropology and sociology; etc. Nonetheless, I would posit that a number of quantifiable metrics contribute to pron's definition of power (the ability to bend others to your will), such as net worth, yearly earnings, spending patterns, education levels, employment statistics, and social network models, especially when these are considered over time.

These "long discussions" that you show disdain for are attempting to introduce a semantic base on which "well defined" terms can be further developed and evaluated in light of new social experiences (e.g., the latest claims of * -ism in SV). pron has set out an extended metaphor, say, which is akin to describing how one would perform an oil drop experiment. That is the opposite of "resistance". Your assertion that "power" and "privilege" are semantically empty is completely without grounding. Just because the metaphoric equivalent of "applying the power formula" has been left as an exercise to the reader, does not mean that the terms of the equation are void of meaning.


Power is also well defined (I have provided a definition which is not "any probabilistic cause of social outcomes") and in cases like this -- easy to measure. You may assert the last forty years of research are pointless. That it's definition applies in many cases (though not all -- an earthquake leveling a city isn't directly related to power) does not make it any less useful (just like energy is always involved in any interaction of particles). If you apply the definition, you will see that influencing college admissions officers << influencing police in this case.

The fact that intractable interactions lend themselves less easily to formulations (let alone closed expressions) does not take away their reality or invalidate the model. Much of the work in applied mathematics (non-linear equations) is qualitative, as well. I think it is you who are resisting to admit that the past decades of research have taught us a lot about how society works.


If your definition is not simply equivalent to social outcomes, there should be an experiment that can potentially measure power absent social outcomes. If so, what is it?

I.e., suppose a group has good outcomes but low power or vice versa. How can I find out?


You can't measure power without its effects just like you can't measure energy (at the very least, it will have to affect your measurement device), but you can often tell whether the effect was due to power or not. Sometimes -- like with the case of potential energy -- you might be able to deduce its existence once you've learned how it works for a while. But, you can often see powerful people fail (say, a billionaire dies in a plane crash, or loses all his money when the market crashes), and powerless people thrive (a homeless man wins the lottery). But power, like force in physics, is the main thing driving human interaction. It is certainly the only mechanism of any interesting social dynamics, but it is not trivial, as power -- like energy -- takes many forms.

So, to me, your question sounds a bit like, "how do we know when the planets in the solar system move by gravity, and when they move by something else?", to which the answer is that the planets almost always move by gravity, except very rarely, when, say, hit by a particularly large asteroid; how doe we know when that happens? We look. Same here, if a group has good outcomes and low power and vice versa -- while very rare (as power is at the core of the mechanism), you can either study the case carefully (which is what historians do), or compare it with power's known outcomes to see if it's one of those flukes. But, you'll say, I can isolate gravity and test it in a lab to make sure I'm certain this is how the planets move. Well, experiments like that are harder in the social sciences, but they are done quite regularly. Two very famous experiments in power are the Milgram experiment (testing authority power) and the Stanford prison experiment (testing authority power as well as its effect on those who have it). Many dictator games are experiments in other forms of power.

Besides, I don't see what exactly you're driving at. Thousands of studies have uncovered some mechanisms at the very core of human society. The mechanisms behave similarly enough to warrant a name (kinetic energy, potential energy etc.), and that concept seems to be at the heart of what drives most of society. Not only that, it induces a quantifiable (if sometimes only roughly, or even in theory) property. That mechanism, along with its quantifiable trait is called power. It was found to be roughly "the ability to bend others to your will", and has produced interesting, useful models (qualitative -- not quantitative). You want to give it another name? Fine, call it X. But 100 years ago we did not know about X as much as we do now. If you want to identify X with something that you think has been known for a long time -- you'll be wrong; if you want to identify X with something you think is still a complete mystery -- you'll be wrong again. You want to argue with scientists about the names they choose and then quote someone who says arguing about names is futile -- great. What is it that you're saying?

The concept of power conveys a lot of knowledge that has been gathered over decades. Your responses seem to be like those of someone who's just heard of energy, and says, "If energy is what moving things possess, why not just call it speed? Oh, a ball at the top of the hill also has energy, why not just call it height? Oh, fire has energy too? So energy is everywhere, and if it's everywhere then it doesn't mean anything!" Either that someone decides to learn basic physics, or decides to stay ignorant. But if he decides to stay ignorant, I think you would agree it would be foolish of him to continue arguing.


Actually, since you keep talking about energy, you brought to my mind a critique of some textbooks by Feynman: http://www.textbookleague.org/103feyn.htm

I turned the page. The answer was, for the wind-up toy, "Energy makes it go." And for the boy on the bicycle, "Energy makes it go." For everything, "Energy makes it go." Now that doesn't mean anything. What they should have done is to look at the wind-up toy, see that there are springs inside, learn about springs, learn about wheels, and never mind "energy."...Now that doesn't mean anything. Suppose it's "Wakalixes." That's the general principle: "Wakalixes makes it go."...It's also not even true that "energy makes it go," because if it stops, you could say, "energy makes it stop" just as well.

I claim that this critique applies equally well to your use of the word "power".

And your comparisons to real sciences are quite inapt - again, as I've pointed out to you before, a discussion with tptacek on crypto or kasey_junk on high speed trading results in the aforementioned posters being very specific while their critics are vague. Kind of the opposite of what is happening here.

Note that you still haven't actually provided an experiment or measurement that could identify a successful yet powerless group (if such a thing existed), or vice versa.


You keep saying "your use of the word power" as if power is not a well known, well studied concept. It is not "my use of the word power" but simply power. You did the same thing when we discussed sexism, which is an academic term invented relatively recently by feminist scholars, which you insisted on treating as some obscure, ill-defined notion. This tone shows a misplaced contempt to a vast scholarly endeavor.

In any case, I don't see how that critique applies to power at all, because, yet again, some of the mechanics involving power are well known and well documented. Nobody says "power makes it so". It's just that explaining the power dynamics of racial neighborhood segregation or sexism in tech would take dozens of pages.

It is not simply that "power is what drives women participation in tech down". I can trace a process -- some documented and some hypothesized -- starting with "classical" gender roles, through the massive transition in gender roles and general separation between the sexes that occurred in Victorian times (they had rooms in houses meant to serve men and rooms for women) and shapes society to this day, through the history of women in computing (starting with the transition of switchboards from being seen as a job for women to one for men), with the more general association of which jobs are for men and for women. That would take me about 50 pages, I guess. But power is the central mechanism. I'm not saying "power did it"; I can show how. Just not here.

> And your comparisons to real sciences are quite inapt

Well, I've been using metaphors, naturally. The intractable sciences are much more complex than physics, chemistry and even biology. There are no closed-form formulas in the social sciences; at least not yet.

> a discussion with tptacek on crypto or kasey_junk on high speed trading

Maybe they're just better communicators than me, and maybe HFT is more amenable to discussion in HN comments than the history of gender roles and the evolution of power in human society. However, if you have specific questions (and they would have to be more specific than "how come there are fewer women in tech") I could try to answer succinctly if it is at all possible. The problem is that these are things that are never even taught to first-year social sciences students (some are only taught in grad school), and unlike with HFT, I don't think you even have the basics.

For example, I don't know if you're at all familiar with the techniques used to study history or sociology, how historical documents are analyzed, how different societies are compared etc., and I really can't lay out an intro to social studies here (BTW, that Curtis Yarvin guy I told you about suffers from the same problem, except he considers himself knowledgable for some reason. His writings read like an Aristotelian scholar discussing quantum mechanics; he's completely ill equipped to handle the materials he's using, which is why he draws such ridiculous conclusions. Of the months spent teaching students simply how to approach reading documents, he doesn't even apply the very first lesson: classifying the genre of the document and identifying the intended audience and purpose)

Now, I'm sure that there are some introductory materials to gender studies that skip the basics of social science, but I doubt you'll find them convincing if you're not familiar with the methodology. If you are interested, I could try to find some online course in history or sociology that seems good, but my guess is that they won't get to gender roles in an intro course (and if they do, it will be by skipping the groundwork, which, again, will make it seem less convincing).

> Note that you still haven't actually provided an experiment or measurement that could identify a successful yet powerless group (if such a thing existed), or vice versa.

You still haven't provided an example of a planetary system whose planets revolve around a star due to a force other than gravity! Gravity is what makes planets revolve around a star, and power is the mechanism by which groups (and individuals) obtain success. Once in a while there are aberrations, to which I have provided examples: winning the lottery. Or, if California is covered by the ocean, then the very powerful people who live their might become extremely unsuccessful. Of course because that population is powerful, various disasters would probably be addressed by the government faster and with more rigor than in other parts of the US, but that may still happen.


You still haven't provided an example of a planetary system whose planets revolve around a star due to force other than gravity!

I asked for a measurement which could identify such a group, not a measurement that would. I can easily tell you experiments to test this in physics - solve Newton's law of motion and find a celestial body with motion that doesn't agree with it.

If I were advocating for the invisible roller coaster track theory of celestial motion, I couldn't provide such an experiment. The invisible roller coaster tracks are observable only by celestial motion - whichever way the moon moves, that's where the track is.

The only way to refute the theory would be via an alternate method of observing the position of the tracks and then observing whether the moon actually followed that track. If someone didn't provide that alternate method, I'd say he was not even wrong.

However, if you have specific questions...

Besides the one I repeatedly ask, you mean?


Well, there are numerous examples of groups that had little power that achieved success, but remember that there's a feedback loop here, as once you achieve success you obtain power. But everywhere you see social mobility, those are cases where people with little power slowly obtained success, which then turned to power.

Examples from the middle ages include grants of knighthood[1] as payment for some unusual service. While usually a knight would only come from wealthy or noble families (or at least a family with good connections) -- hence, from a position of some power -- sometimes knighthood was granted to brave foot soldiers -- i.e. people with little power. Sometimes, the title came with land (and the serfs that worked it, of course).

In non feudal societies, social mobility was usually achieved through money, although some classes were barred from obtaining any money whatsoever (slaves). You can see groups of immigrants, provided the host society did not block their steps too much, slowly gain money, and later recognition and connections. This process would often take several generations.

Analyzing those processes is helped by the fact that often you can observe power directly. Money and nobility titles are very conspicuous forms of power, easily measurable directly. More hidden forms of power such as connections can also be traced directly (a boy of low background would be taken to the home of a merchant as a gift to his parents in recognition of some service; this lets you trace connections across classes); charisma (which in the middle ages was a great way to attain power in religious circles) could be seen in some extraordinary ascetic acts[2] or visions[3]. The latter was one of the few ways women could rise to positions of power in medieval societies (see Joan of Arc), although others would be marrying, and surviving, a man of power. While it was often expected of widows to remarry, some medieval societies were surprisingly relatively accepting widows, recognized their independence, and allowed them to transact on their own.

[1]: http://www.lordsandladies.org/steps-to-knighthood.htm

[2]: http://en.wikipedia.org/wiki/Simeon_Stylites

[3]: http://en.wikipedia.org/wiki/Lucy_Brocadelli


>I was therefore delighted (intellectually, that is) to learn recently of a fringe Silicon Valley movement called neoreactionism or “Dark Enlightenment”, funded by Peter Thiel.

AFAIK Thiel's sole, extremely tenuous connection to the Dark Enlightenment is thinking libertarianism and democracy are incompatible. He certainly hasn't funded any of its leading lights. This is less accurate than the belief that the Koch brothers control the Tea Party. (They did have a large impact on its early growth.)

>These guys (few women would join that openly sexist, openly autistic movement)

Ableist. If you want an example of a woman who's been involved (more than periphally but not as an identified adherent) look up Justine Tunney. If you're transphobic then she doesn't count as a woman, ableist.

>are probably all former libertarians that have discovered that there is no such thing as freedom from power, and now openly call for a tyranny.

That's one branch of the trichotomy in case you're interested.

http://www.xenosystems.net/wp-content/uploads/2013/05/reacti...

http://www.xenosystems.net/trichotomy/

>As someone who’d studied medieval history in graduate school (though I have never obtained my doctorate) I was delighted to see the movement’s leader, a programmer by the name of Curtis Yarvin, analyze some historical document and call for the return of feudalism (he complains that those documents are not studied by historians, which is true for the simple reason that they are false accounts).

This is a massive misreading of Yarvin, a.ka. Mencius Moldbug. He's got a hard on for absolute monarchy, not for feudalism. The two are very, very different. Feudalism was basically a Western European phenomenom, Ottoman depotism, Russian autocracy or France during the reign of the Sun King are more his thing.

What false accounts are you referring to? Could you provide some links to the deceptive documents among Yarvin's output?

>The problem with the return to feudalism — even if you were to believe the false accounts of how life was good then (it by no means was) — is that the power structure back then was at least held in check by technology, that is, lack thereof. With the invention of mass media and fast transportation, power can be more concentrated than ever before, which is why the greatest invention of Western civilization was the central government, which rose to contain and manage power (of course, this only made conflicts among those governments more violent than ever before, but that fear of annihilation reduced the number of conflicts considerably).

Autocracy, not feudalism.

> Feudalism combined with modern technology has only been tried — to the best of my knowledge — once, in nineteenth century America.

This is so ridiculous that it makes me question your claims of having studied history at a graduate level. Have you ever heard of the Bolsheviks? They had a successful revolution in the Russian Empire in 1917 and founded and ruled the Soviet Union until its dissolution. They arose in a state that attempted to combine autocratic government with modern technology.

>The US at the time had a very weak federal government with almost no regulation. The result was a period of extremely fast economic expansion but at great social cost: a large portion of the American population was enslaved in all by name by a very small number of slave-owners-in-all-by-name known as the Robber Barons (Rockefeller, Carnegie, Mellon, Stanford, JP Morgan, Frick, et al.).

Are you aware that the USA has been among the richest societies on Earth for its entire existence? People were poor because there was so little to go around, not because people were hoarding for the sake of it. You occasionally say lucid and intelligent things but the US economy was growing insanely fast by any historic standards more or less from settlement by Europeans to around 1970. It's still growing insanely fast but the trend has slowed down. economic growth in North America was labour limited for a long, long time. Things were much, much better in North America than anywhere else on Earth, all this while having large inflows of migrants from much poorer nations, i.e. the entire rest of the planet.

>All options were taken from them — they couldn’t migrate (they were sometimes paid in company-issued currency, that was useless anywhere else) and couldn’t organize to concentrate power to improve their lot (in fact, they did organize, but the robber barons had private armies that killed the rabble rousers and intimidated everyone else). The people cried for help, and Theodore Roosevelt rescued them by creating federal regulation.

Did you study under Howard Zinn[0] or something? Everywhere else was worse. The Pinkertons were awful but the USA has never been a weak enough state to allow private armies anywhere on its territory. It was probably the friendliest country on Earth for labour organising for the period in question.

[0]http://en.wikipedia.org/wiki/A_People%27s_History_of_the_Uni...


My knowledge of the Dark Enlightenment is very limited at the moment, as I've just discovered it during preliminary research for a study of fringe movements in California related to technology. But during a cursory reading of Yarvin's I've found a few mentions of feudalism as a desired goal (obviously, under some weak autocracy), as well as mentions of traditional gender roles and a defense of slavery. I also believe that Peter Thiel funds Yarvin through a startup company of his. I will not list the false accounts he mentions (I noted down a few so far), because I don't want to discuss the subject here. A cursory glance, though, reveals the Dark Enlightenment to be a fascinating Californian fringe movement, with some truly novel interpretations of familiar ideologies (mostly fascism, but not quite, as fascism was directly influenced by Romanticism while DE rejects Romanticism except that restricted Ayn-Randian form of it, and fascism was very much nationalistic -- again, through romantic influence -- while DE isn't).

As to your description of the Gilded Age, I am not sure what our points of disagreement are. Yes, the Pinkertons weren't actual armies, nor were they entirely ignored by the government. My description was greatly simplified for brevity. As to the conditions of workers at the time, comparisons to other societies are irrelevant, because that economic growth (which was the result of immigration, land expansions and new resources) was not at all contingent on the exploitation that was taking place.

In general, comparisons are often less useful when the framing discussion (in this case, questions of policy) is normative. If a tyrant comes to an island where the population is starving, and feeds them a loaf of bread a day but forces them to do backbreaking work for him, then their position is better than the alternative, but no one would suggest that this is in any way desirable or even ethical. The same would be true for another tyrant who feeds his enslaved population two loaves of bread a day.

Comparisons are useful if the claim is made that no other policy would have been possible, which I don't think anyone is making. Sure, growth would have been slower (concentrated power is always a lot more efficient, as time and resources are not required to achieve compromise and accommodate other stakeholders), but preference of efficiency over other human goals is purely a matter of value.

The Bolsheviks instituted an autocratic form of communism, which is pretty much the complete opposite of feudalism (which is usually the result of a free market, although the terms are anachronistic as no one used the term "free market", when real feudalism was actually in place), and in any case, feudalism is certainly anathema to any ideology promoting equality (like communism).


"What will be your new belief about $GROUP's technical talent"

Hopefully the most reasonable one: that there's no such thing about the technical talent of "$GROUP" per se, though our society might presently be so organized that it's easier for members of $GROUP to excel at certain kinds of tasks.


Of course there is such a thing - cook up a tech talent score based on these test results (see my other post on that topic) and then measure a group's mean, sigma or credible intervals.

How society is organized may be a cause for a group having higher/lower technical talent (so might "intrinsic" features, i.e. things not mutable by "society"), but it doesn't mean that talent doesn't exist.


The thing is that the effect of such biases is hard to measure directly. Employers will get better employees than before, but may also miss more good employees than before, yet notice the former much sooner than the latter (which will have a delayed effect).

The best way to address it is to first measure for biases (run a demographic analysis on players), and second actively design for unselected populations with focus groups. I am sure some of this information has already been collected by game studios.


It is a bias, but it is similar to the following bias: http://www.theonion.com/video/in-the-know-are-tests-biased-a...

How do you test for noncompetitive but still productive people using a competitive test?


My first thought when reading the blurb on your site was 'oh great... they're going to make a CTF where you have to participate in a team, and all other members of your 'team' are actually AI programmed to be as distracting and abrasive as possible and you will have to work on a 25 year old PL/SQL system designed by a prima donna so you can show your ability to work in hostile environments.' I am sure that's not what you're doing, but that's what comes to mind when I hear 'real systems' and 'real challenges'...


You will not have to participate in a team or work on my 25 year old PL/SQL system; I made the PL/SQL optional.


Sad. There should definitely be a challenge involving table functions all the way down - combined with a Call of Cthulhu esque sanity system...


It looks like he clearly implies the challenge is there :)


On your "About Us" page, you say "We're especially interested in talking to you if you will be hiring 10+ engineers in the next year, as we may be able to help you out with that. Get in touch with any of us."

Does a company need to be hiring 10 or more engineers to make it worth it to talk to you?

I'm at a small company, that has been trying to hire engineers for the last couple of years, and not having much success; we've hired a couple, had a couple take other offers, lost a few to attrition.

We would love to be able to get some candidates who have already passed a technical screen; less time for us, scheduling an hour with a bunch of candidates who don't pan out, less time for them, only focusing on one more in-depth set of problems rather than a whole bunch of different phone screens.

But we're not hiring 10+ developers any time soon; that would double the worldwide size of our dev team. We're looking for maybe 3 or so at this point.

Would you consider working with smaller companies as well?


I obviously can't speak for the founders, but I suspect that if you're viewing this as a replacement for a technical phone screen, you might not be viewing it as intended.

Did you read Thomas' "The Hiring Post" which was at the top of HN a couple days ago?


Yes, I read that.

I would love to improve our hiring process, and include more realistic work samples as part of it.

However, I don't feel that a full-fledged work-sample test is realistic, especially if every employer started requiring it. Imagine applying to a few different jobs, and each requiring you to do 20 hours of learning of their material followed by some open ended realistic problem, that could take many hours to complete. If you were looking around at a few different employers, this would be a full-time job itself; some candidates would skip it because they could find a job which required less effort up front easily enough, some would simply not have the time because they are working a full-time job while doing it.

On the other hand, if there is a shared work-sample test like this one, that will be used by a number of employers, it means that it can be more in depth than any single employer could do, while simultaneously wasting less of the candidate's time as they do the one test, rather than one per potential employer.

So yes, in a way I am viewing this as a replacement for a technical phone screen, because I am interested in improvements to that process but haven't come across any potential improvements which seem viable, but this one sounds like it could have potential.

The "read the resume, read their existing code samples, and do a phone screen" approach covers a lot of ground in a little time, so while there are a large number of problems with it, it's pretty hard to improve on without spending a lot of the candidate's time to get up to speed on a more realistically sized problem, and a lot of good candidates aren't willing to devote that much time unless they really, really want to work for that particular employer. We're a small company, working in a niche business, so it's hard to get that kind of dedication from a single candidate.



I'm totally happy to talk to anyone who hires engineers, is interested in what we're doing and/or wants to know more about how we made this work at my last gig. Please don't hesitate to reach out; you can't waste my time.


They just say, "especially," and I think a charitable reading would tell us that it's a call to get a good base of early adopters from which more-easily generalizable results can be gleaned.


No questions, just a hearty "Congratulations!" and "Good luck!"

I'm excited to see this develop.


Disclaimer: I have only skim read the article, so this is for my laziness and the benefit of all

Apart from the gameplay mechanism with code challenges, how is this better / different to HackerRank for a) "programmers" b) companies?



This looks great. Looking forward to trying it. Good on all of you for trying to change the tech hiring status quo.

I know this whole thing is still in flux, but does it look like there are going to be any remote work opportunities in the early days? I'm probably in the minority but given my personal circumstances (living in a non-tech-rated US city and not considering moving) I'm really only available for remote.


Is this a reference to "The Last Starfighter"?


"Greetings, Starfighter! You have been recruited by the Star League to defend the Frontier against Xur and the Ko-Dan Armada."


This was how Thomas announced to me that Starfighter would exist and I would be CEO of it. I was confused as heck until I actually saw the movie.


lol!


What do we do??

-We die.

Fantastic name for this product.


Back to sleep, Louis, or I'm telling Mom about your Playboys!


Congrats on this new venture, looks fun.

What do you expect will be the effect of this on hiring women? Do you have any relevant data from microcorruption (what % players were women)? It'd be great if Erin could also chime in.


We do not have access to the µc data, any longer. That belongs to Matasano. I am now sad that I did not ask myself the same question and pay closer attention while I did.

However, this is what I am hoping and dreaming for: that work-sample testing will completely level the playing field. Regardless of gender, age, or origin, if you can do the work, you should get the job. I believe in this so strongly that I am dedicating the rest of my career to work sample testing and, eventually, training.

Getting women to play, though? I'm not so great at marketing, so still working on that. I am writing a blog post to go up later in the week that addresses some of the issues in hiring women, based on my personal experiences and those of my female peers. The tone of the post is as politically neutral as it can be. My hope is to draw out some productive dialog rather than piling on stink for flies. Specifically, vast majority of the people in my professional network, male, female or in between, are genuinely interested in addressing the "women-hiring problem," but they avoid any discussion of it because ... well, it's always covered in flies.

The challenge will be to make just enough of a stink that it draws attention and not flies.


> However, this is what I am hoping and dreaming for: that work-sample testing will completely level the playing field. Regardless of gender, age, or origin, if you can do the work, you should get the job.

That's a nice thought, though it rather presumes that the playing field is level outside of the immediate neighborhood surrounding the hiring process.

I think that neighborhood, though, is probably, while not without gender/race/etc. bias, overrated in terms of the proportion of that bias on the course between birth and getting a software job that it is perceived to contribute.


Believe me, that has crossed my mind. This is where the segue into training comes. We've discussed it. As a woman, my one experience of being allowed to play outside and beyond the hiring process is pretty consistent with the anecdotal evidence given by my peers. But correlation is not causation. For every one candidate, there are an uncounted number of pivots to consider, and mapping them all out is impossible.

I agree that bias is overrated as a contributing factor to this so-called problem. But it is one we focus on because we also believe that behaviors can be controlled for and habits can be rehabilitated. There are a lot of narratives where the circumstances are very different but the dynamic is the same. Someone whose parents want them to be a doctor or lawyer who aggressively derail them from pursuing the arts. Some very intelligent kid growing up in poverty with no access to resources who starts acting out and ends up with a criminal record. A woman who is forced to take HomeEc in high school instead of trigonometry (me).

Taken in isolation, these narratives can be explained away. As examples of a larger pattern, they become flaws in a system that can be engineered away. There's a meta-bias, and I'm still searching for a way to describe it that doesn't trigger an emotional response. It seems that until we can have that rational, engineering-focused discourse to identify the not level playing fields of the world, cataloging their characteristics, finding what things can be controlled for and/or eliminated,... sigh I don't know how to finish that sentence.

You say it's a nice thought. I believe it's a nice thought. More people hope for it than do not. What exactly is keeping all of us who genuinely believe that it's at least worth a try to apply a dialectic method to the problem from banding together and attempting it?

I don't know.


I think the desire to to figure out where a level playing field can be built within a neighborhood of aptitudes. And to be fair to a process of refinement for that goal, some problems can't be solved before you have them. Like a business plan, the initial draft is just a guess, and I think we can all agree that it's a heretofore unsolved problem, the gender thing.


Also true for large coding projects. At some point, you have to identify the components that will need to be built and portion them off into approachable tasks. In this case, the first task is agreeing on the 'heretofore unsolved problem' -- that hasn't happened yet.

Discerning invested participants from trolls might help.


>The tone of the post is as politically neutral as it can be. My hope is to draw out some productive dialog rather than piling on stink for flies.

I am sorry, but that is a line with a negative width: if you so much as hint that this may in part be the fault of males (as a whole or, even worse, individuals) pitchforks will be out. If you do not hint strongly enough that this is the fault of males (as individuals and as a group) pitchforks will be out on the other side. As a more personal point the sheer toxicity of those debates continues to shock me, even after spending more than a decade on the internet. Vi vs Emacs debates don't tend to end in treats of rape and/or genocide but "why aren't there more women in computing" almost always do.

This isn't to say that we can't draw any benefit from your writing, just be aware that of the things you can't say (http://www.paulgraham.com/say.html) this is the thing you can't say most.


Hi, I've taken over from Tom for hiring at Matasano. There's a couple of things that you need for "diverse" recruiting (e.g. hiring women in tech): 1) A way of evaluating candidates that avoids mirrortocracy style institutional -isms like sexism and ageism. 2) A way of convincing people who have bounced off the field due to -isms to even try.

BTW, Tom was great at Matasano on #1, and only moderately successful at #2. We get tons of people who aren't in the industry, but they tend to self-select to be young men. For an individual company, #2 is way harder. It's "easy" to fix yourself, but hard to fight the larger culture.

Starfighter looks to me like something that supports #1 directly (by allowing objective metrics), and enables #2. It's a way for people who aren't welcome into a field to dip their toe in without some roomful of young white men asking them illegal questions about their child-rearing plans. On the other hand, there's still tremendous pressure pushing people away, and it's a deep problem. Still, part of the puzzle and I (personally) highly endorse.

As to the numbers, I've been meaning to dig into the data, but Matasano's hiring as fast as we need to, so it's been hard to motivate myself to do so. Maybe once graduation season ends...


I got completely stuck on the first level of Microcorruption. Should I even attempt this new stuff?


Yes. And as someone who also was stuck for many moons on later levels of that challenge: keep going. Research. Ask questions -- people will help. Honestly, I had no expectation of getting past the first microcorruption level but goddamn if it doesn't feel good when (not if) it happens.


\o/


if you're monitoring everything players do, doesn't that create an incentive for them to write their own tests locally and only interact with your servers when they have stuff already working? so they look like a genius instead of a guy who made a bunch of mistakes. "look i never make any syntax errors, hire me". but seriously whatever you monitor, players will have incentive to try to figure out and game it. and that incentive messes up the regular experience, it's an incentive to do something other than beat the level.


First, no part of our model involves monitoring how many "mistakes" someone made, and second, if you see what challenges like this are like (again, see Microcorruption), you know that there's no realistic way to do them that doesn't involve making zillions of little mistakes.


How are you planning on combating cheating? For sure people will post solutions online to your CTFs.


I bet that Thomas and Patrick could do something like have dynamically created tests, where there are multiple versions of an exploit, so that they test the same skill but in different ways. You might have a challenge that tests whether you can do SQL injection, but the tables may differ. Overrunning a buffer (as Patrick mentioned in his post about Microcorruption) could depend on different payloads.

I naiively believe that things could be varied enough that you would need to understand the concepts rather than paste answers, in which case "cheating" would mean "learning", and is basically something they say they want you to do. I'm not sure how hard it would be to get to that point rather than being vulnerable to Bob the Super Coder posting walkthrough screencasts, but I trust that the founders are already savvy enough to have been thinking about that.


Unfortunately, I suspect you overestimate the obfuscating power of dynamic content when the number of users is sufficiently large.

A certain MMO I play recently had a limited-time event built around figuring out the meaning of different clues (locations to go to for the actual meat of the event), and despite a fairly large number of variations, people had collectively figured out just about every possible clue->location mapping within a matter of hours.

That's not to say you can't prevent cheating, but that even with relatively little incentive (that whole clues thing gave only a single cosmetic item, and anecdotally I've seen very few people actually use theirs) users can and most likely will outpace any attempt to prevent it by means of varying the problem.


This is an important problem. As part of interviewing processes, I have several times tried offering "take home work challenges". I had to stop after, having run the experiment several times, I detected plagiarism in about 30-40% of the cases. The risk is of plagiarism is real for any problem that's well known enough that the problem and solution can appear online, and detecting plagiarism is not always easy.


Wait, why'd you stop? Isn't this a great way to filter out those 30--40% of people?


I like this response. I think that one of the best things about a take-home challenge is that during the interview, you can then say, "The way you implemented this function was interesting. Let's talk about why you did it that way." If he knows what he's doing (or made a weird solution), you get a great glimpse into his thought process. If he plagiarized, then he's garbage.

English teachers do the same thing for detecting plagiarism. Bring the kid into your office and talk about the essay. If he's completely clueless, someone else wrote it for him.


It's a neat idea, but ultimately, worrying about cheating or whether someone has cheated feels like a distraction. It suggests a test that isn't repeatable. I'm only confident in my ability to detect obvious plagiarism. Subtle plagiarism can exist in varying forms, such as reading an analysis of the problem and solutions. Some people who plagiarize will pass a thorough Q&A about their code, because they fully understood the explanation of the solution, but are getting an unfair advantage over candidates who worked out a solution themselves from scratch.

Overall, I get more value from tests that are constructed so that I can learn positive things about the candidate - as many opportunities as possible for the candidate to distinguish themselves. If the only reason for a particular approach was to provide the opportunity for immoral candidates to weed themselves out by committing obvious plagiarism (negative data), then there's probably a better approach that tells me more about the candidate per unit time.

If I was going to continue, I would use a problem that is (1) more representative of the actual work being done by the team; less of a puzzle (2) custom designed for the team or company; not a preexisting or well known problem. (Even candidates who don't cheat can have an unfair advantage on well-known problems if they have coincidentally encountered it before! Another reason to use unique questions.)


> Some people who plagiarize will pass a thorough Q&A about their code, because they fully understood the explanation of the solution, but are getting an unfair advantage over candidates who worked out a solution themselves from scratch.

This reminds me of the debate over whether performance-enhancing drugs should be allowed in mathematics. Why do you think it's so important for the candidate to personally invent every aspect of their solution? What if you just told people that it's ok to use external resources to solve the problem?

A class might give exams in any of these ways:

- exams only happen in class, where everyone can notionally be supervised

- exams are take-home, but you can't read the textbook while you're taking one

- exams are take-home, and you're free to read the textbook

There's cheating under all of those models, including the first one which takes the form that it does specifically to prevent cheating. The implicit goal (for the students) of model 1 is to make sure they've internalized whatever is being taught. The implicit goal of model 3 is to make sure that, even if they haven't internalized the material, they're capable of applying it. The implicit goal of model 2 is to make sure they'll comply with arbitrary, unenforceable demands (in this context, usually called "the Honor Code"). That might make sense if you're hiring a cashier -- but is it really your first priority?


At Princeton university, faculty members are not allowed to proctor in-class exams. (See the top of page 2 of https://registrar.princeton.edu/faculty-services/Conduct_of_... for a reference.)

Do you feel similarly that the implicit goal of this model is "to make sure they'll comply with arbitrary, unenforceable demands" and still not to test internalization of the material?


This is to my (1) as my (2) is to my (3). The applicable standard is even called "the Honor Code". I don't see why you think I'll see a difference. It's quite clear that making sure (or emphasizing that) the students are The Right Sort Of People is an explicit goal of the Princeton policy; see the final sentence of the relevant section of the document you linked. ("STUDENTS MUST WRITE AND SIGN THE HONOR PLEDGE IN FULL ON THE COMPLETED EXAMINATION PAPERS", caps original.)


It's less important than it sounds for a bunch of reasons having both to do with the actual "game" we are building and with our business model. It's not like we spit out a number for every player, and everyone with a number better than X gets a job offer.


Hey, tptacek.

Your last post on hiring definitely found its legs and I am sure will cited for some time to come. You mentioned that you used to send a sampling of key text books to potential Matasano candidates and was wondering if you could share some references. I imagine Applied Crypto might be on there. Any any other important resources for mastering CTFs you might pass along would be obliged ;)


Definitely don't read Applied Cryptography!

http://sockpuppet.org/blog/2013/07/22/applied-practical-cryp...


These days we mostly send The Web Application's Hacker's Handbook and a link to microcorruption. (We do somehow get candidates which haven't heard of microcorruption.) Generally, we continue to endorse Tom's Amazon reading list: http://www.amazon.com/An-Application-Security-Reading-List/l...


Can you talk at all about the pricing structure for companies that will use your app for recruiting?


Patrick can, at patrick-at-starfighters.io.


Is it ok for a group of us who to complete the challenge as a team for fun? Or must we agree to only work individually?


Fun is the point.

I don't know exactly what the "rules" are going to be, but what rules we have will have one purpose: not to screw the game up for anyone else.


Do you have a wishlist of all the problem domains you'd like to tackle? (I'm thinking web, mobile, and desktop applications above the system level when I ask this, but any other areas are wonderful. Databases. etc.)


We prefer to keep things a pleasant surprise, but we'll be emailing anyone who signs up with a suggested reading list if they want to start brushing up on old skills or acquiring new ones to get ready for launch day. The reading list will give a good deal of direction.


Right now I'm wondering, "Does this have any overlap with tptacek's Amazon reading list?" but I suppose I'll find out with the email arrives.

Thanks :)


Sounds interesting. What's the ETA?


Optimistically, weeks-not-months.


EDIT: Ill need to lookup other CTFs like this... as clearly I am not familiar with how they work.


Solve a puzzle and then move on to next puzzle. It's fairly straight forward.


Thanks, my original question was what actually is a flag. Is it an atomic object that can actually be captured.would be great if it was something unique like say Nixon's are.


A Frickin' awesome idea.

My best wishes in your success!!!

I look forward to the results.


Will you get me above £100k in London?


Will you be allowing anonymous participation?

By this I mean, the article uses some creepy language as only a company enthusiastic about its power to broker reputation can.

>>We can tell you exactly what happened when your candidates tried to implement a REST API.

What, are you going to keep a record of the time I fiddled with it for an afternoon and then use that to disqualify me from a job?

You say elsewhere about MicroCorruption:

>Microcorruption player identities were totally private. There's no way for a recruiter to look someone up, unless they used a very-identifiable username.

Sounds like a good starting point, but what if I don't want my report card stored and used to compare with other people? What if I don't want to feel like my performance is owned by you?

>>We assess for skill first, passively as players play our games and then actively. Our founders — talented technologists — personally reconstruct candidates’ solutions and evaluate them.

If I don't have any control over how and when my information is going to be monitored, I'm forced into an ugly position where I have to treat the whole thing as part of my career and perform competitively. The sense of relaxed exploration is killed, the idea of treating it like a college course or project isn't viable, and honestly I begin to resent the project.

Alternatively, I play on an anonymous account, and then if I decide to use my information about me in a professional setting, I'm best served by creating a new account and just redoing everything.

Edit: I should say that the idea really excites me and I'd love to learn in this format.


Will you be allowing anonymous participation?

Yes. You can participate anonymously or pseudonymously. I think we say in three places that we only give out contact details if candidates ask us to.

What, are you going to keep a record of the time I fiddled with it for an afternoon and then use that to disqualify me from a job?

Our incentive is to find reasons why you're hireable, not reasons why you're not.

What if I don't want to feel like my performance is owned by you?

Your performance is owned by you -- we won't have copyright to your code or anything. Signal we gather, though? Owned by us. That's the trade: we give you a really fun game to enjoy, but we control the universe you play in and, inside that universe, we control everything and see everything. If you do not like this trade, that's your prerogative and I respect your opinion. Nobody will force you to play Starfighter.


Edit: The following is a neutral appraisal, not a criticism of either the implicit or explicit variety.

If you feel that it is an accusation or a criticism, I would suggest that indicates something unrealistic in how you view business negotiations.

As patio said, this is a trade between interested parties. Generosity is therefore expected to be limited.

---

Fair enough. I just wanted it on record that this is a business transaction for you, not a generous educational endeavor.

Some people want to pretend that they're in it to make the world a better place when that's not truly their absolute top priority. They're probably asking "Why can't it be both?" when I say that this is business, not generosity.

One simply can't have multiple top priorities, particularly with profit and generosity; generosity is practically defined as something that doesn't profit. Something like one of your (patio's) blog posts are given generously in that there is no formal expectation/obligation on a reader to 'give back' some asset, for instance.

Thank you for your clarification.


If your top priority is providing a service to the community (e.g., education), you need a sustainable support model, whether its as a business or through someone else's charitable donations.

If your top priority is a profit-making business, you need to provide a product that is (or, at a minimum, people perceive as) valuable.

Perhaps you can't have two top priorities (I actually am not sure I agree that this is the case, I don't see that it is actually impossible for two priorities to be equal in priority and above all others), but even if you can't, its quite possible for "business" to be a means and "education" to be the goal, and vice versa. They aren't incompatible.


Exactly. Tesla Motors comes to mind as a company for which "electrification of the world transportation" is the goal, and "profitable business" is only means to an end.

I understand the fear though; most of the companies we interact with (and I'd wager, 90% of startups we see here) are what I call toilet-paper companies - they'd gladly switch from whatever it is they're doing to manufacturing toilet paper if that would render more profit or increase chances of getting acquihired; their top priority is business, not the goal. I hope that Starfighter isn't such toilet-paper endeavour.


As a public company isn't it Tesla's obligation to try to run a (eventually) profitable business?


It is. And Elon Musk stated repeatedly that it's a secondary goal, means to an end. There's a lot of wiggle room between "profitable business" and a "toilet paper company".


Sure. Did 'TeMPOraL say anything contrary to that?


Which is why I was careful to prepend "generous" to every instance of "education" in my response. Starfighter isn't generosity of knowledge, it's barter of it.

Perhaps what you're saying is true in that universities have a financing division, but my point is that this isn't a case of university. The focus is on how they're going to make money as middlemen (and dwarves will sing about their riches), not how their business is subservient to their idealistic educational aims.


Even if you are being generous with your time and effort, you can't carry out an effort with any kind of reach without some support model. Financing it through its own operations may not be the only choice, but its not a choice incompatible with generous motivation.


The thing is, making money off providing a useful service that benefits society as a whole isn't immoral, so I don't see why it's that big of a deal. Nobody is forcing anyone to participate.

EDIT: In fact, socially rewarding companies that do good things incentivizes others to adopt ethical strategies and might do more good than a vow-of-poverty educational service provider.


> The thing is, making money off providing a useful service that benefits society as a whole isn't immoral, so I don't see why it's that big of a deal. Nobody is forcing anyone to participate.

Of course it's not immoral but still, knowing whether "a useful service" or "making money off it" is a top priotity for the company is important. Most of the companies you and I interact with are of the second type, and I guess this is at the root of throwawaymaroon's worry.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: