
Apply HN: Skill Certification You Can Trust - danieltillett
Problem: Certificates and degrees don&#x27;t measure real understanding or ability to apply the knowledge learnt and are often little more than a sheet of paper. The cause is that most certifications, even when awarded with rigor, measure the students ability to cram and regurgitate knowledge a short time after the material was taught. No certificate or degree provides any measure of how much of knowledge or skill is retained 6 or 12 months later.<p>Since any potential employer can&#x27;t rely on a candidate&#x27;s certificate&#x2F;degree to indicate that they  actually knows the material (or can apply it), they are forced into retesting every candidate (i.e. the dreaded technical interview). This is an incredible wasteful process for both candidates and the employers. What is needed is a certification process that can&#x27;t be gamed that measures true retention of knowledge and skill.<p>Solution: Use an approach similar to WADA&#x27;s whereabouts system [1, 2] that is used to detect drug cheats in sport. Certificate&#x2F;degree holders would be tested on their knowledge and skills at some point after the certificate&#x2F;degree was awarded. Holders would be able to register their availability for testing at certain times and days on a system similar to the WADA ADAMS system [3]. They would be called at 12 hours notice anytime over the next 3 to 12 months and asked to sit a test in one of their nominated times slots. Failure to be available for the test would mean a fail.<p>Such as system would prevent cramming and it would only measure long term knowledge retention as there would be too short a warning of when they were to be tested. The knowledge tested could then be certified by the testing agency to be Real&amp;True™. Employers would not need to test a candidate again in the interview process.<p>1. http:&#x2F;&#x2F;www.usada.org&#x2F;testing&#x2F;whereabouts&#x2F;<p>2. https:&#x2F;&#x2F;www.wada-ama.org&#x2F;en&#x2F;questions-answers&#x2F;whereabouts<p>3. http:&#x2F;&#x2F;adams-docs.wada-ama.org&#x2F;display&#x2F;EN&#x2F;Welcome+to+the+ADAMS+Knowledge+Base+in+English
======
greenyoda
_" Certificate/degree holders would be tested on their knowledge and skills at
some point after the certificate/degree was awarded. Holders would be able to
register their availability for testing at certain times and days on a system
similar to the WADA ADAMS system. They would be called at 12 hours notice
anytime over the next 3 to 12 months and asked to sit a test in one of their
nominated times slots. Failure to be available for the test would mean a
fail."_

As an employee, I'd find this distasteful. Just as I wouldn't work for a
company that wanted me to piss in a cup at random intervals, I wouldn't work
for a company that wouldn't hire me unless I was re-tested for competence at
random intervals.

Also, I'd find it disturbing if a company that has a financial incentive to
make people take as many tests as possible would be given control over who
gets hired across a large number of companies. And the system seems to be
lacking the transparency that would allow people to determine whether the
testing company was itself competent to judge the competence of others.

Finally, in a field like software development, there's such a wide variety
between the skills needed in different jobs that a one-size-fits-all
certification would be meaningless. E.g., it's perfectly fine for someone who
works on operating system kernels to not know anything about HTML, CSS,
JavaScript or mobile app development.

~~~
danieltillett
I am not sure where I suggested that a company is forced to use the system or
not. I can't see any company testing their current employees as there is no
advantage for them doing so. The would only be interested in the
certifications of new candidates.

So your objection seems to be that a company offering these tests might make a
lot of money by having lots of people take these tests? It is not like
companies aren't doing these skills tests already - the nice thing about this
approach is that it makes the whole testing process more transparent. As a
candidate you know if you passed or not, which is more than can be said for
the average whiteboard test.

Of course the tests would be specific to the skills you want to be tested on.
If you want to be tested on front end material the test would be about that,
not operating system kernels. The candidate controls what skills they want to
be tested on.

------
ajsgarage
This is an interesting approach to a genuine issue; I've got a background in
Curriculum Design and can agree that an outside, "third party" type
certification system could have market potential for both graduates and
employers. One of the sticking points in my studies looking into a similar
concept was balancing the cost to the student with the expected utility of
receiving the certification. So, unless at the debut of the certification
there is a large, verified list of employers willing to pay a premium for
certified graduates, this concept doesn't make a market value connection in my
opinion.

On another fundamental note, the system as described would be effective in
long-term retention knowledge testing, true. Yet this model doesn't seem to
incorporate 'continuing education' into the remedial approach. Thus, the
certified prior knowledge can be tested and maintained, but there's no
inclusion of 'new' studies and knowledge that promote long-term growth on top
of the knowledge base.

As I've experienced in several industries, there are already licensing
('cerification') bodies which have deployed a Continuing Education (CE) credit
model (e.g. insurance, finance). This proposed model may benefit from
comparison and distinction from these existing, regulated approaches, in
addition to the above note regarding expected market value. Cheers for taking
on a genuine issue and working towards a better environment!

~~~
danieltillett
Thanks. It is always great to get thoughtful feedback.

 _One of the sticking points in my studies looking into a similar concept was
balancing the cost to the student with the expected utility of receiving the
certification. So, unless at the debut of the certification there is a large,
verified list of employers willing to pay a premium for certified graduates,
this concept doesn 't make a market value connection in my opinion._

I agree that traditional certification suffers from this problem, but my idea
gets around this. Traditional certification requires a large investment in
time from the candidate (study) where as mine requires no study. This lowers
the barrier to entry to sitting for the certification. With investor backing
the tests can initially be made low cost or free further lowering the barrier
to entry. I can see it appealing to people with skills but not a tradition
education who want to demonstrate they have these skills.

The employers need to value the certification (this is one of the difficulties
I listed in my follow-on post). The best way of achieving this is to start in
a very narrow industry/skill set, achieve critical mass, and then expand. This
solves the typical chicken and the egg problem that you need credibility to be
widely used and you need to be widely used be credible.

 _On another fundamental note, the system as described would be effective in
long-term retention knowledge testing, true. Yet this model doesn 't seem to
incorporate 'continuing education' into the remedial approach. Thus, the
certified prior knowledge can be tested and maintained, but there's no
inclusion of 'new' studies and knowledge that promote long-term growth on top
of the knowledge base._

There is no reason why the system is incompatible with continuing education -
you just test the effectiveness of the new education in the same way. In many
ways my idea complements continuing education since continued education is one
of the ways to keep the knowledge the candidate can draw on in the test fresh
in their mind.

 _As I 've experienced in several industries, there are already licensing
('cerification') bodies which have deployed a Continuing Education (CE) credit
model (e.g. insurance, finance). This proposed model may benefit from
comparison and distinction from these existing, regulated approaches, in
addition to the above note regarding expected market value._

If successful I would see a symbiotic relationship developing between the
providers of continuing education and the business running my idea. They could
use my approach to ensure that their education is making a lasting impact on
the participants and is not just a "checklist" requirement done and forgotten.

The think I feel most positive about this idea is it can catalyse a change in
education towards long term retention of skills and knowledge. Education
providers who use the approach will change the way they teach and move us all
away from the current cram and forget model.

------
commentzorro
From the employee's point of view this will always be a hated service because
people hate being tested. Especially the majority who are not naturally gifted
but only got by because of cramming stuff into their heads. These are
perfectly serviceable programmers who will learn what's needed and do a good
job in the industry. In addition, you're not forcing employees to put too much
emphasis on a single test for a potentially (hopefully?) large group of
employees. Does this become a stressor like an SATs for coders? If the
employee doesn't do well are they screwed? Or do you allow them to retest? If
so, you've become just like any number of certification services that have
come and gone over the years.

From the employers point of view you appear to have the opinion that most jobs
are in startups. In reality, the huge quiet slowly moving employer base is
below the visible portion of iceberg. These are the businesses that just need
coders to keep things moving from day to day. They are not startups, never
will be, and don't even exist in the world where founders are wanted or
needed. There needs in a coder is not a superstar, it's someone who can learn
to tow the line, get things done, and keep their heads down and code. A
average coder is fine here as long as they fill a niche.

Finally, how could you develop a standardized test for all these below the
waterline businesses? If a business needs to fill a slot for a person to crank
out Java reports for a few years do you test the same as one who needs to fill
a position interfacing C++ to POS/IoT devices?

Not that idea is a bad way to make money. It's just that you'll likely need to
move into a niche and then you'll need to drive test retakes, release cram
books, and become another paper certificate mill. Remember when paper CNAs and
such were all the rage.

~~~
danieltillett
Thanks for taking the time to write a detailed critic. Now to your specific
questions.

 _From the employee 's point of view this will always be a hated service
because people hate being tested. Especially the majority who are not
naturally gifted but only got by because of cramming stuff into their heads.
These are perfectly serviceable programmers who will learn what's needed and
do a good job in the industry._

The testing should be the least painful possible since you can’t study. You
either know the material or not. If you are actually able to do a good job in
industry then you will pass. The entire purpose of the testing is to separate
people into those that can do a serviceable job and those that can’t. With
normal tests you might be able to cram and pass but not be any good on the
job, while with the proposed tests this will not be possible.

 _In addition, you 're not forcing employees to put too much emphasis on a
single test for a potentially (hopefully?) large group of employees. Does this
become a stressor like an SATs for coders? If the employee doesn't do well are
they screwed? Or do you allow them to retest? If so, you've become just like
any number of certification services that have come and gone over the years._

I am not sure how you got the idea that this would be a single test. There
would ultimately be thousands of different tests.

I doubt any such system would become universal, but even if it did it would be
a failure of the testing process if people who are productive workers in a
field failed. Every test should be able to separate the competent from the
incompetent.

I would imagine that as people gain skills and experience in a field they
would do better on the tests. If you do improve you should want to retake the
test and demonstrate how you have improved. This is actually a great thing for
an employee as you will be able to demonstrate to your boss that you have
improved and are worthy of a higher salary.

 _From the employers point of view you appear to have the opinion that most
jobs are in startups. In reality, the huge quiet slowly moving employer base
is below the visible portion of iceberg. These are the businesses that just
need coders to keep things moving from day to day. They are not startups,
never will be, and don 't even exist in the world where founders are wanted or
needed. There needs in a coder is not a superstar, it's someone who can learn
to tow the line, get things done, and keep their heads down and code. A
average coder is fine here as long as they fill a niche._

Not at all. If you are an average coder you should get an average mark and get
an average job in a company that just wants average coders. Nothing about my
proposal changes anything here other than to make it easier to identify who
are the average programers and provide them with an un-gameable certificate of
what they really know. It also only measures a limited aspect of a person
(their skills) and companies will still hire or not on the basis of cultural
fit, etc.

 _Finally, how could you develop a standardized test for all these below the
waterline businesses? If a business needs to fill a slot for a person to crank
out Java reports for a few years do you test the same as one who needs to fill
a position interfacing C++ to POS /IoT devices?_

I think you are thinking that there is only one test. There would ultimately
be thousands of tests covering all areas. People would take the tests of the
skills they want to be certified for. If you are a java developer you would
take the java test and I would expect that even something like java would be
broken out into sub-categories over time.

 _Not that idea is a bad way to make money. It 's just that you'll likely need
to move into a niche and then you'll need to drive test retakes, release cram
books, and become another paper certificate mill. Remember when paper CNAs and
such were all the rage._

This is the key difference of my idea and all the other certificate mills -
you can’t cram for the test. There won’t be an ecosystem of cram books and
test prep providers since you can’t cram. You either know the area you are
being tested on or you don’t. Once people know this the value of such a
certificate rises.

------
danieltillett
Due to the 2000 character limit here is the rest :)

Monetization: Certificate holders could pay a fee to be certified Real&True™
for any skill they wished. Educational organisations could also charged to
have their students tested. This would "value add" to their education as well
enable them to be able to market to potential students/employers how well
their past students did in the Real&True™ tests.

An added benefit of the whole testing system is that it would encourage a
change in the way students are taught to ensure long term knowledge retention
and application of material, rather than encourage the current "cram and
forget" process.

Difficulties: The major one is the tests have to be perceived as being
valuable by both candidates and employers. The obvious solution is to start in
a niche industry and scale out over time as the testing process gains
credibility. Another potential pitfall is certificate holders may find the
whole process too onerous and refuse to sign up for the testing process (elite
athletes certainly complaint about WADA's whereabouts system). Once again this
could be trialed in a niche and scaled out if successful.

------
bestattack
OK, I think the number one thing to ensure about this business, if you want to
revolutionize hiring, is to show that it works. If I can apply this idea to my
potential hires, I'm going to give you all the dollars because the alternative
is spending my team's precious cycles on interviewing.

I think there are a lot of paths that lead to your business making money but
ultimately going down a path towards "yet another technical certification" \-
technical certifications are a negative hiring signal for a good reason.
Avoiding this pitfall, I think, is the hardest part but perhaps the key to
success.

~~~
danieltillett
My idea is not about revolutionizing hiring, but changing education. As you
rightly point out technical certification currently have a bad reputation
(well deserved). This is what my idea fixes.

------
mpbm
Have you confirmed with any employers, trainers, hiring managers, etc that
they actually need to be able to skip their internal preparatory steps?

Who would be the actual customer here? The potential employees? Are they
supposed to keep taking random tests while they're employed to maintain their
cert in your system?

~~~
danieltillett
I am sure that some employers would still go through the whiteboard process,
but once they were confident that the certification could be trusted they
would stop.

I explain below who the customers, etc are - I could not fit it all in because
of the HN posting limit.

How often to be tested would be up to the employee. My guess is few people
would retest, but would spend their time being tested on new skills.

------
kumarski
If you had to focus on a specific area, it could be building a low cost ABET
certified curriculum that has better unit economics than traditional
universities.

ABET is pretty legit.

~~~
danieltillett
It could certainly be used that way - in the end it would not matter where you
had studied or if you studied at a traditional institution at all, but what
you had really leant.

As I mention in my follow on post, it would be best to start in a niche and
build out to a more general testing process over time.

------
clementangerine
I'm not sure if this accurately represents how people remember things. It's
normal to be a bit fuzzy on a topic a year after learning it if you aren't
regularly applying it. What matters is that you can learn it again quickly,
and I think preparation for technical interviews works fine for testing that
already.

~~~
danieltillett
My idea is not about how people learn.

The current system of technical interviews works - my idea is about making
this process better. The market exists, my idea makes the process more
efficient and captures some of the value created.

------
skewart
This is a really interesting idea. I'm not convinced from your description
that you have a viable product just yet, but I think you're headed down a
really good path. Here are some more thoughts:

\- Testing and certification is the most economically valuable aspect of
education (at the high end the most valuable test a student can take is
getting accepted to an elite institution (I'm using "test" broadly here)).

\- Learning itself can and has scaled quite easily thanks to the internet. But
testing and certification hasn't really yet.

\- There is a huge opportunity to build a new authority for
testing/certification/credentialing. (In many ways YC is doing this almost by
accident, but it's not their main focus and they're not doing it in a scalable
way).

\- The kind of knowledge and skills that are valuable in white collar
"knowledge worker" jobs are typically not what you study for and get tested on
in school. It's more the various soft skills that let you succeed. In other
words, it's not the algorithms themselves, it's that you're disciplined and
smart enough to figure out how to learn them. The same is true for white
boarding interviews - and lots of management roles too - it's not just that
you know how to balance a binary tree, it's that you can explain how to do so
in a clear way, can handle curve balls without getting flustered, and seem
like a good person to work with.

\- There are lots of professional certifications that various professions
require. But it's not necessarily in anyone's interest for workers to take
these kinds of tests more often or in a more rigorous way.

\- There probably are some industries where concrete knowledge testing is
valuable. I don't know how big a market it is, but it's probably there. One
strategy would be to go after these industries. I'd look for fields with high
turnover, and/or where companies are losing money due to worker error.

\- Another strategy would be to go after the higher end, and try to become a
trusted authority for certifying employee skills. I think this is better
because this seems like the kind of thing where it's easier to go from high-
end to low-end than the other way around.

\- Certifying knowledge workers would probably look less like giving them pop
quizzes and more like conversations with them, of else just tracking their
work closely and assessing it.

\- To get it off the ground you need to decide who is your customer - the
worker or the employer? It sounds like you're thinking workers will be your
customer. In that case the most important thing will be to show a return on
investment for a few early adopters. I'm not sure how best to do that early
on, but I think you'd want to have some clear benefit that they get - i.e.
getting a job or a promotion indirectly as a result, but that probably way
beyond your control. Maybe there's something else?

~~~
danieltillett
Thanks for the feedback. The description is cut short because of the 2000
character limit here on HN. I did post a follow up comment with the rest of
the description, but it has now fallen to the bottom of the comments so nobody
seems to be seeing it :(

The basic idea is to bring technology to the certification process in
education (a huge industry). Right now all the new education institutions (and
a lot of the old) have no way of really proving to the world that their
graduates have learnt something and have valuable skills. Employers see a
piece of paper from some institution and wonder what does the candidate really
know. This is what the idea tries to solve. Now to your specific comments.

 _Testing and certification is the most economically valuable aspect of
education (at the high end the most valuable test a student can take is
getting accepted to an elite institution (I 'm using "test" broadly here))._

Exactly. The value of an elite institution is not in the actual education you
might get, but the hurdles needed to overcome to get in. It is an incredibly
crude proxy for measuring what an employee will actually be like.

 _Learning itself can and has scaled quite easily thanks to the internet. But
testing and certification hasn 't really yet._

Yes. The reason why is that there has been no way of preventing gaming of the
testing system. What you want to measure is if a person really know their
stuff, or has just crammed it all in before the exam. My idea is break this
link between cramming and test results by preventing cramming.

 _There is a huge opportunity to build a new authority for testing
/certification/credentialing._

Yes there is and once built you will have a very strong network effect that
will make the business very valuable.

 _...In other words, it 's not the algorithms themselves, it's that you're
disciplined and smart enough to figure out how to learn them._

This can actually be tested too. Once you make a test unable to be crammed
then you can start to do the sort of tests that measure real on the job
performance.

 _Another strategy would be to go after the higher end, and try to become a
trusted authority for certifying employee skills. I think this is better
because this seems like the kind of thing where it 's easier to go from high-
end to low-end than the other way around._

Yes this is my thinking too. Start in a narrow niche and become the authority
for certifying the skill set of people in that niche. If the testing authority
say you really know skill X then as an employer you will believe this is true.
You can add all sorts of layers to this where you can rank a persons skill as
beginner, intermediate, expert, etc. Once you have a strong foundation in a
niche then just expand into related fields.

 _Certifying knowledge workers would probably look less like giving them pop
quizzes and more like conversations with them, of else just tracking their
work closely and assessing it._

Because the tests can’t be crammed the tests that would be given would by
design have to be a much closer match to real work than a typical test. You
can make the tests as rigorous and challenging as the market wants. If people
want to be certified that are lateral thinkers that can solve real work
problems then that is exactly what can be tested. You don’t want to test
people on things that they can google in 30 seconds, but the sort of thinking
and problem solving that make a real difference in the workplace.

 _To get it off the ground you need to decide who is your customer - the
worker or the employer?_

There are two customers - the worker and the education provider (see my follow
on post for more details). It would be easier to go after the workers at
first, but the educational providers would be where the real money would be.

~~~
skewart
Awesome. Yeah, I think this could be really big.

> You don’t want to test people on things that they can google in 30 seconds,
> but the sort of thinking and problem solving that make a real difference in
> the workplace.

Do you have a plan - or at least a general idea - for how to do this?

Also, thinking about this more, here's another random thought:

Fundamentally the problem that testing and certification tries to address is
trust. If I say I can do something you don't have to just take my word for it,
you can see that a trusted third party vouched for it. A lot of people have
been experimenting with alternative mechanisms for providing trust lately -
from AirBnB and Uber relying on reviews/reputations instead of certifications
and brands, to even something like bitcoin. Maybe the solution isn't a better
testing system, but instead something that removes the need to test at all?

~~~
danieltillett
I used to be a university professor so I am highly familiar with the
difference between those students that really knew their stuff and those that
just crammed well. You really only want to hire the former.

In a sense this idea gets rid of traditional testing. You are not measuring
how many facts a test taker has managed to cram into their head, but how much
they retain and can apply. Grades as such would become irrelevant as we start
to focus on those retained skills that translate into day-to-day productivity.

I do have a plan about how this idea could be best developed, but HN is
probably not the place to discuss this. My hope is by putting this idea up I
will get feedback or suggestions about aspects I have not thought of. It is a
big idea that could change a massive industry like education and there are
many different ways it could be developed.

