
Launch HN: CSPA (YC S18) – SAT for Software Engineers - cspa
Hi HN!<p>We&#x27;re James and Angel, the founders of CSPA (<a href="https:&#x2F;&#x2F;cspa.io" rel="nofollow">https:&#x2F;&#x2F;cspa.io</a>).<p>The Computer Science Proficiency Assessment (CSPA) is a standardized assessment for software engineers.  We find software engineers to take our exam, and then share the assessment results with employers, who can then hire these candidates for free.<p>James was the founder of Crunchyroll and oversaw product and engineering.  Angel also worked at Crunchyroll.  We decided to start CSPA because:<p>1. There are many self-taught programmers out there who are really really good, but don&#x27;t get a fair chance because they don’t have a degree.  We want to give these people an equal opportunity to pursue their career.  That&#x27;s why there are no eligibility requirements to take the CSPA.  You can come from any background, and if you get a high CSPA score, you’ll stand out.<p>2. As hiring managers at Crunchyroll, it was challenging to find great software developers.  We wanted a service that <i>quantitatively</i> compares job applicants in an <i>unbiased</i> way, and ALSO did that assessment for us.  In short, we wanted candidates to come to us pre-assessed.  It would have saved us and the applicant a TON of time.<p>3. As software engineers, we hated applying for jobs.  We did the same phone screens, the same coding challenges, and same whiteboard exercises over and over again, company after company.  It&#x27;s a huge waste of time.  There has to be a better way.  Why couldn&#x27;t we just do it once, and then reuse the results at every company we apply to?  Keep it DRY, right?<p>If you want to take the CSPA, you can register here: <a href="https:&#x2F;&#x2F;cspa.io&#x2F;winter-2018&#x2F;register" rel="nofollow">https:&#x2F;&#x2F;cspa.io&#x2F;winter-2018&#x2F;register</a>.  The first 100 people to sign up can use code &quot;LAUNCHHN&quot; to waive the exam fee.<p>If you&#x27;re an employer looking to hire a pre-assessed candidate, sign up here: <a href="https:&#x2F;&#x2F;cspa.io&#x2F;add-employer" rel="nofollow">https:&#x2F;&#x2F;cspa.io&#x2F;add-employer</a>.  We don&#x27;t charge a recruiting fee.<p>If you&#x27;re interested in joining the Technical Steering Committee, apply here: <a href="https:&#x2F;&#x2F;cspa.io&#x2F;tsc&#x2F;apply" rel="nofollow">https:&#x2F;&#x2F;cspa.io&#x2F;tsc&#x2F;apply</a>
======
tptacek
I scanned through the "General" questions in the practice test linked on the
thread, paying particular attention to the "Security" questions.

Not the best.

There are two questions regarding denial-of-service attacks. One of them I
found hard to answer (had a plausible answer and a plausible "none of the
above"). The second one was clearer but still ambiguous. Weirdly, the
ambiguity in both questions stemmed from the use of an HTTP POST login as the
possible DOS vector --- why? Why use that confusing example?

The 2FA question I saw had two valid examples of 2FA (push notifications and
chip-and-pin) but no "all-of-the-above". I'm curious which of those two the
author has demoted.

The phishing question I'm just irritated at the superficiality of; everyone
will get the right answer, but the right answer misses the point and the power
of phishing attacks, which isn't "websites that look like real websites" but
rather the lures (like targeted emails) that get people onto those sites. But,
whatever.

The password management advice question asked for a "best" from several
subjective answers.

The SMS 2FA question had two valid answers and no matching "these-two-answers"
answer. I'm pretty sure the answer that wss being looked for was the social-
engineering phone-porting one (which is weird, because the first answer is
"SMS 2FA is considered secure by current industry standards", which is
certainly true for most reasonable definitions of "industry standards"). But
also: that's not even the biggest problem with SMS 2FA.

There was a blockchain question. Who is this for? I'll buy that every working
programmer needs to know how a phishing attack works (though perhaps not what
distinguishes, in the test author's mind, a DDoS attack from a DoS attack).
How many working programmers know how blockchains work?

Similarly: there was a GAN question. Come on. What was the point of that?

Finally: it's a _six hour_ test. You weren't kidding when you called it "the
SAT for programmers". It's not an especially pleasant experience (I like the
interface, though). This test is a very big ask of candidates, and I think a
short ways through the test it becomes clear that there's not much intrinsic
merit to it; it's just a hurdle.

~~~
pryelluw
Six hour test on top of whatever time you spend interviewing. Because you know
companies will still have you interview.

~~~
cspa
Our initial goal is to replace the technical phone screen. We are asking
companies to guarantee an interview (or skip the phone screen), if they score
above a certain threshold.

Long term, we hope to replace as much of the onsite as possible. But we do
acknowledge we can never fully replace it.

~~~
pryelluw
May seem like I'm shitting on your product, but I'm not. I do a good amount of
hiring[0] and have had to use services that provide similar outcomes[1]. It's
mostly a waste of time and am currently working on removing that part of the
process. My main issue with your product is that I don't see how it would
provide me with a better outcome than what's out there.

I want to like it enough to try it out and pay you, but dont see value
proposition right now.

[0] Corporate Fortune 500 types in many verticals. [1]
[http://derricocomputers.com/](http://derricocomputers.com/)

------
georgecalm
There are two requirements that I personally don’t feel very comfortable with:
the first one is uploading a picture of my ID (How is it stored / backed up.
Who has access: third party validator or your team?) and the other is adding
logmein to let a proctor control my machine prior to the exam. I saw that one
may take the exam in person. I don’t know if it’s allowed by your rules, but I
suppose I can set up a VM for this, but either way that just makes it more
difficult for me to justify finding the time to do it.

~~~
cspa
Our third party proctoring service handles the KYC, as well as the LogMeIn. We
did some vetting, and they are used by some well-known MOOCs and universities'
online courses. It's a concern of ours as well, which is why we try to offer
in-person proctored tests or appointments.

------
WilliamEdward
I want to make another comment about how these questions are much more
trivial, when compared to the mathematics SAT which has a lot of problem
solving. I suggest have users solve more problems _using_ computers, and less
problems _about_ computers.

I'm only making these comments because I don't want this to fail.

~~~
cspa
Thanks for the feedback! We try to have a good mix of trivial, novice,
intermediate, and advanced questions.

Some of the multiple choice questions are fact-based and just require a Google
search -- this is intentional, since proper Googling skills are still needed
for the job :)

Problem solving questions are better formatted as open-response essay
questions, which we do have.

~~~
WilliamEdward
thanks for the replies

------
WilliamEdward
It's good but... I don't want this to turn out to be anything like the SAT in
that it's a requirement for schools and they won't have it any other way. I've
found when it comes to schooling, people are ready to market aggressively
because the students don't have a choice (we saw this with TI calculators)...
So what's your promise to prevent this from happening? Software Engineering is
good right now because anyone, absolutely anyone, can dive in. This is a step
in the wrong direction in that regard.

~~~
hardwaresofton
Yeah that's exactly how it's going to work. Standardized testing is very
rarely effective when employed in this fashion as a proxy for "talent" or
"drive" or "good employee". It usually ends up being a proxy for something
else.

This also pushed the onus on candidate selection even further on to the
candidate, when it's clearly a lack of trust from employers that _you are
capable of what you put on your CV_. Testing that should be the burden of the
employer, not the employee.

One key difference between software engineering and other engineering
disciplines is that there are many more ways to solve most problems, many more
problems to solve, and an endless number of customizations. At a concrete
level what does the canonical "web app" look like? We can say "3 tier
architecture" but that's abstract -- which technologies should you pick? When
should you pick ruby over python? What about when you should pick a document
store of a relational database (OK that ones kinda easy) -- _these_ are the
questions that good engineers can answer and refute -- there's usually not an
absolutely right answer, either.

I personally also feel the same way about the CKA (Certified Kubernetes
Administrator) -- hire someone with a CKA and sure that means they know the
Kubernetes basics/internals (as it stood whenever they took the test), but
does that mean they keep up? Experiment with new tech/RFCs/features (so your
company doesn't have to with the product on the line)? Can they choose _not_
to use Kubernetes when it's not necessary?

Rather than just building good, multi-layered interview questions with subtly
random parts and lots of room for interpretation (to see _how_ they interpret
and think about the problem), we just pick these solutions that are somewhat
broken from yesteryear (see: it/sysadmin/networking certs).

------
faitswulff
I have a strange fetish for multi hour standardized testing, so this looks
great to me. Thanks for posting and for the code!

One question based on your responses in this thread: is this test intended to
measure software engineering ability or computer science knowledge?

EDIT - one more question after looking through the grading: do you see people
studying for specific roles, or the entire test? While I think the latter is
possible, it seems much less feasible than studying for, say, the entire SAT.

~~~
cspa
Currently the Core Exam is 50/50\. It's designed for career-readiness. Our
Technical Steering Committee is composed of majority of hiring managers, so
it's intended to be useful for them -- that is, what are the qualifications
desired in an entry level software engineer?

We are debating having a Core general exam, and separate per-subject tests,
similar to SAT I vs SAT II.

------
mleonard
Hi James and Angel congrats on the launch! Great issue to try to solve, hope
you succeed.

Thanks for the LAUNCHHN voucher code. I'd seen cspa.io on hackernews
previously but the free voucher code was the motivation I needed to register
for the core exam in April! I'm a uk-based maths grad previously in a datasci
role, currently taking some time off to self-study computer sci and
frontend/backend. Looking forward to checking my progress against the test!

Quick bits of feedback in case they're helpful:

(1) I'd love there to be a second practise/past paper available (there's only
one currently right?). Ideally I'd like to go through one now to check where
my knowledge gaps are... then take one a couple of weeks before the exam in
April.

(2) I just gave this only a very quick read... and found it a bit unclear:
[https://cspa.io/scoring](https://cspa.io/scoring). I'm sure I could
understand it no problem with a more serious read-through... but thought I'd
feed that back anyway. [Specifically it's not clear in the examples section
how the score of 440 in FE is achieved, as the max mentioned is 400. Also the
6 subject scores all differ... but then presumably are scaled to 400 points.
Is that right? With my quick read through I'm not yet clear if in the exam I
should try to complete all 6 or just complete 1 of the 6 subject areas. I
guess as final score takes your best score out of the 6 you could just
complete one or two. What do most candidates do in practice? Perhaps you could
add some guidance on this to the site somewhere if it's not there already.]

Regards. Mike.

~~~
cspa
Thanks!

(1) When we retire old exam questions, we will make them available for
practice. We're working on creating more study materials! I can't promise
anything before April though.

(2) Ah the 440 FE is a typo. We updated the scoring algorithm a few months
back and didn't update those examples. Most people try completing all 6
subject sections. It's up to you, but we do discourage optimizing this kind of
test taking strategy, and "teaching to the test" ;P

------
tdeck
Sorry, but 5-6 hours is a non-starter for me. I've never taken an exam that
long in my life (the SAT is about 4 hours IIRC), and I'm so glad to be out of
college and done with this type of BS. At least interviews let me talk to a
person and learn something about them and their company - sometimes I get a
story out of it.

------
jatsign
Sample exam, for those interested: [https://cspa.io/sample-
exam](https://cspa.io/sample-exam)

~~~
redshirtrob
First off, I like the idea. Tech interviews are so broken that I'm happy to
see folks trying to make things better.

I just started clicking through this and stopped at question 7: "How many bits
are in 8 bytes?"

I don't know if this is intentional or not, but this is an ambiguous question.
I suspect the "right" answer is the one that assumes 8 bits in an byte, but
historically that assumption is not always true, e.g. lots of IETF RFCs refer
to "octets" rather than "bytes" to avoid the confusion.

I bring this up not to quibble about historical architectures, but to address
another item I recall from my SAT/ACT taking days--getting into the head of
the question writer. When I saw this question I thought, "Are they looking for
the current/common interpretation, or do they expect me to know mostly useless
trivia about the history of the definition of byte?"

Perhaps I'm overthinking it. I probably am. But for that specific question,
the answer is "it depends." And based on that, there is no 100% correct answer
offered, so I'm left choosing the next best answer, which is probably the one
most folks would select anyway.

Was this intentional? Again, it's been a long time since I've taken these
types of tests.

~~~
cspa
We've found that that is indeed true. The more senior and knowledgeable you
are, the most likely the answer is "it depends", because all the nuances and
edge cases you know about.

We try to solve this with the standard "choose the BEST answer". We also do
post-mortems and analyze the responses to disqualify or discount bad
questions.

These are issues the CSPA Technical Steering Committee, which is responsible
for the exam content, will deal with, in collaboration with our research
advisor: [https://cspa.io/about/team](https://cspa.io/about/team)

------
Kaveren
Why does it take 3 to 10 days to process results? What's stopping you from
delivering results within a second?

 _Edit: I didn 't see that there was open ended responses or a programming
project on some of the exams because I was looking at the sample exam, which
only has multiple choice._

------
fedotovcorp
Hi James and Angel! You are doing a great job with your project. Does CSPA
concentrate only on hiring processes or also helps to find partners,
contributors and the possibility to develop tech community?

------
comonad-colaboy
Was curious as to how tests like these guard against Goodhart's law (If
goodheart's law is applicable here at all)?

~~~
cspa
Thanks for the tip! Can you elaborate? Do you mean teaching to the test, or
something else? There's a whole research field for large scale assessment and
evaluation methodologies.

