Apply HN: Skill Certification You Can Trust

[danieltillett]

Problem: Certificates and degrees don't measure real understanding or ability to apply the knowledge learnt and are often little more than a sheet of paper. The cause is that most certifications, even when awarded with rigor, measure the students ability to cram and regurgitate knowledge a short time after the material was taught. No certificate or degree provides any measure of how much of knowledge or skill is retained 6 or 12 months later.

Since any potential employer can't rely on a candidate's certificate/degree to indicate that they actually knows the material (or can apply it), they are forced into retesting every candidate (i.e. the dreaded technical interview). This is an incredible wasteful process for both candidates and the employers. What is needed is a certification process that can't be gamed that measures true retention of knowledge and skill.

Solution: Use an approach similar to WADA's whereabouts system [1, 2] that is used to detect drug cheats in sport. Certificate/degree holders would be tested on their knowledge and skills at some point after the certificate/degree was awarded. Holders would be able to register their availability for testing at certain times and days on a system similar to the WADA ADAMS system [3]. They would be called at 12 hours notice anytime over the next 3 to 12 months and asked to sit a test in one of their nominated times slots. Failure to be available for the test would mean a fail.

Such as system would prevent cramming and it would only measure long term knowledge retention as there would be too short a warning of when they were to be tested. The knowledge tested could then be certified by the testing agency to be Real&True™. Employers would not need to test a candidate again in the interview process.

1. http://www.usada.org/testing/whereabouts/

2. https://www.wada-ama.org/en/questions-answers/whereabouts

3. http://adams-docs.wada-ama.org/display/EN/Welcome+to+the+ADAMS+Knowledge+Base+in+English

[greenyoda]

"Certificate/degree holders would be tested on their knowledge and skills at some point after the certificate/degree was awarded. Holders would be able to register their availability for testing at certain times and days on a system similar to the WADA ADAMS system. They would be called at 12 hours notice anytime over the next 3 to 12 months and asked to sit a test in one of their nominated times slots. Failure to be available for the test would mean a fail."

As an employee, I'd find this distasteful. Just as I wouldn't work for a company that wanted me to piss in a cup at random intervals, I wouldn't work for a company that wouldn't hire me unless I was re-tested for competence at random intervals.

Also, I'd find it disturbing if a company that has a financial incentive to make people take as many tests as possible would be given control over who gets hired across a large number of companies. And the system seems to be lacking the transparency that would allow people to determine whether the testing company was itself competent to judge the competence of others.

Finally, in a field like software development, there's such a wide variety between the skills needed in different jobs that a one-size-fits-all certification would be meaningless. E.g., it's perfectly fine for someone who works on operating system kernels to not know anything about HTML, CSS, JavaScript or mobile app development.

[danieltillett → greenyoda]

I am not sure where I suggested that a company is forced to use the system or not. I can't see any company testing their current employees as there is no advantage for them doing so. The would only be interested in the certifications of new candidates.

So your objection seems to be that a company offering these tests might make a lot of money by having lots of people take these tests? It is not like companies aren't doing these skills tests already - the nice thing about this approach is that it makes the whole testing process more transparent. As a candidate you know if you passed or not, which is more than can be said for the average whiteboard test.

Of course the tests would be specific to the skills you want to be tested on. If you want to be tested on front end material the test would be about that, not operating system kernels. The candidate controls what skills they want to be tested on.

[ajsgarage]

This is an interesting approach to a genuine issue; I've got a background in Curriculum Design and can agree that an outside, "third party" type certification system could have market potential for both graduates and employers. One of the sticking points in my studies looking into a similar concept was balancing the cost to the student with the expected utility of receiving the certification. So, unless at the debut of the certification there is a large, verified list of employers willing to pay a premium for certified graduates, this concept doesn't make a market value connection in my opinion.

On another fundamental note, the system as described would be effective in long-term retention knowledge testing, true. Yet this model doesn't seem to incorporate 'continuing education' into the remedial approach. Thus, the certified prior knowledge can be tested and maintained, but there's no inclusion of 'new' studies and knowledge that promote long-term growth on top of the knowledge base.

As I've experienced in several industries, there are already licensing ('cerification') bodies which have deployed a Continuing Education (CE) credit model (e.g. insurance, finance). This proposed model may benefit from comparison and distinction from these existing, regulated approaches, in addition to the above note regarding expected market value. Cheers for taking on a genuine issue and working towards a better environment!

[danieltillett → ajsgarage]

Thanks. It is always great to get thoughtful feedback.

One of the sticking points in my studies looking into a similar concept was balancing the cost to the student with the expected utility of receiving the certification. So, unless at the debut of the certification there is a large, verified list of employers willing to pay a premium for certified graduates, this concept doesn't make a market value connection in my opinion.

I agree that traditional certification suffers from this problem, but my idea gets around this. Traditional certification requires a large investment in time from the candidate (study) where as mine requires no study. This lowers the barrier to entry to sitting for the certification. With investor backing the tests can initially be made low cost or free further lowering the barrier to entry. I can see it appealing to people with skills but not a tradition education who want to demonstrate they have these skills.

The employers need to value the certification (this is one of the difficulties I listed in my follow-on post). The best way of achieving this is to start in a very narrow industry/skill set, achieve critical mass, and then expand. This solves the typical chicken and the egg problem that you need credibility to be widely used and you need to be widely used be credible.

On another fundamental note, the system as described would be effective in long-term retention knowledge testing, true. Yet this model doesn't seem to incorporate 'continuing education' into the remedial approach. Thus, the certified prior knowledge can be tested and maintained, but there's no inclusion of 'new' studies and knowledge that promote long-term growth on top of the knowledge base.

There is no reason why the system is incompatible with continuing education - you just test the effectiveness of the new education in the same way. In many ways my idea complements continuing education since continued education is one of the ways to keep the knowledge the candidate can draw on in the test fresh in their mind.

As I've experienced in several industries, there are already licensing ('cerification') bodies which have deployed a Continuing Education (CE) credit model (e.g. insurance, finance). This proposed model may benefit from comparison and distinction from these existing, regulated approaches, in addition to the above note regarding expected market value.

If successful I would see a symbiotic relationship developing between the providers of continuing education and the business running my idea. They could use my approach to ensure that their education is making a lasting impact on the participants and is not just a "checklist" requirement done and forgotten.

The think I feel most positive about this idea is it can catalyse a change in education towards long term retention of skills and knowledge. Education providers who use the approach will change the way they teach and move us all away from the current cram and forget model.

[commentzorro]

From the employee's point of view this will always be a hated service because people hate being tested. Especially the majority who are not naturally gifted but only got by because of cramming stuff into their heads. These are perfectly serviceable programmers who will learn what's needed and do a good job in the industry. In addition, you're not forcing employees to put too much emphasis on a single test for a potentially (hopefully?) large group of employees. Does this become a stressor like an SATs for coders? If the employee doesn't do well are they screwed? Or do you allow them to retest? If so, you've become just like any number of certification services that have come and gone over the years.

From the employers point of view you appear to have the opinion that most jobs are in startups. In reality, the huge quiet slowly moving employer base is below the visible portion of iceberg. These are the businesses that just need coders to keep things moving from day to day. They are not startups, never will be, and don't even exist in the world where founders are wanted or needed. There needs in a coder is not a superstar, it's someone who can learn to tow the line, get things done, and keep their heads down and code. A average coder is fine here as long as they fill a niche.

Finally, how could you develop a standardized test for all these below the waterline businesses? If a business needs to fill a slot for a person to crank out Java reports for a few years do you test the same as one who needs to fill a position interfacing C++ to POS/IoT devices?

Not that idea is a bad way to make money. It's just that you'll likely need to move into a niche and then you'll need to drive test retakes, release cram books, and become another paper certificate mill. Remember when paper CNAs and such were all the rage.

[danieltillett → commentzorro]

Thanks for taking the time to write a detailed critic. Now to your specific questions.

From the employee's point of view this will always be a hated service because people hate being tested. Especially the majority who are not naturally gifted but only got by because of cramming stuff into their heads. These are perfectly serviceable programmers who will learn what's needed and do a good job in the industry.

The testing should be the least painful possible since you can’t study. You either know the material or not. If you are actually able to do a good job in industry then you will pass. The entire purpose of the testing is to separate people into those that can do a serviceable job and those that can’t. With normal tests you might be able to cram and pass but not be any good on the job, while with the proposed tests this will not be possible.

In addition, you're not forcing employees to put too much emphasis on a single test for a potentially (hopefully?) large group of employees. Does this become a stressor like an SATs for coders? If the employee doesn't do well are they screwed? Or do you allow them to retest? If so, you've become just like any number of certification services that have come and gone over the years.

I am not sure how you got the idea that this would be a single test. There would ultimately be thousands of different tests.

I doubt any such system would become universal, but even if it did it would be a failure of the testing process if people who are productive workers in a field failed. Every test should be able to separate the competent from the incompetent.

I would imagine that as people gain skills and experience in a field they would do better on the tests. If you do improve you should want to retake the test and demonstrate how you have improved. This is actually a great thing for an employee as you will be able to demonstrate to your boss that you have improved and are worthy of a higher salary.

From the employers point of view you appear to have the opinion that most jobs are in startups. In reality, the huge quiet slowly moving employer base is below the visible portion of iceberg. These are the businesses that just need coders to keep things moving from day to day. They are not startups, never will be, and don't even exist in the world where founders are wanted or needed. There needs in a coder is not a superstar, it's someone who can learn to tow the line, get things done, and keep their heads down and code. A average coder is fine here as long as they fill a niche.

Not at all. If you are an average coder you should get an average mark and get an average job in a company that just wants average coders. Nothing about my proposal changes anything here other than to make it easier to identify who are the average programers and provide them with an un-gameable certificate of what they really know. It also only measures a limited aspect of a person (their skills) and companies will still hire or not on the basis of cultural fit, etc.

Finally, how could you develop a standardized test for all these below the waterline businesses? If a business needs to fill a slot for a person to crank out Java reports for a few years do you test the same as one who needs to fill a position interfacing C++ to POS/IoT devices?

I think you are thinking that there is only one test. There would ultimately be thousands of tests covering all areas. People would take the tests of the skills they want to be certified for. If you are a java developer you would take the java test and I would expect that even something like java would be broken out into sub-categories over time.

Not that idea is a bad way to make money. It's just that you'll likely need to move into a niche and then you'll need to drive test retakes, release cram books, and become another paper certificate mill. Remember when paper CNAs and such were all the rage.

This is the key difference of my idea and all the other certificate mills - you can’t cram for the test. There won’t be an ecosystem of cram books and test prep providers since you can’t cram. You either know the area you are being tested on or you don’t. Once people know this the value of such a certificate rises.

[danieltillett]

Due to the 2000 character limit here is the rest :)

Monetization: Certificate holders could pay a fee to be certified Real&True™ for any skill they wished. Educational organisations could also charged to have their students tested. This would "value add" to their education as well enable them to be able to market to potential students/employers how well their past students did in the Real&True™ tests.

An added benefit of the whole testing system is that it would encourage a change in the way students are taught to ensure long term knowledge retention and application of material, rather than encourage the current "cram and forget" process.

Difficulties: The major one is the tests have to be perceived as being valuable by both candidates and employers. The obvious solution is to start in a niche industry and scale out over time as the testing process gains credibility. Another potential pitfall is certificate holders may find the whole process too onerous and refuse to sign up for the testing process (elite athletes certainly complaint about WADA's whereabouts system). Once again this could be trialed in a niche and scaled out if successful.

[bestattack]

OK, I think the number one thing to ensure about this business, if you want to revolutionize hiring, is to show that it works. If I can apply this idea to my potential hires, I'm going to give you all the dollars because the alternative is spending my team's precious cycles on interviewing.

I think there are a lot of paths that lead to your business making money but ultimately going down a path towards "yet another technical certification" - technical certifications are a negative hiring signal for a good reason. Avoiding this pitfall, I think, is the hardest part but perhaps the key to success.

[danieltillett → bestattack]

My idea is not about revolutionizing hiring, but changing education. As you rightly point out technical certification currently have a bad reputation (well deserved). This is what my idea fixes.

[mpbm]

Have you confirmed with any employers, trainers, hiring managers, etc that they actually need to be able to skip their internal preparatory steps?

Who would be the actual customer here? The potential employees? Are they supposed to keep taking random tests while they're employed to maintain their cert in your system?

[danieltillett → mpbm]

I am sure that some employers would still go through the whiteboard process, but once they were confident that the certification could be trusted they would stop.

I explain below who the customers, etc are - I could not fit it all in because of the HN posting limit.

How often to be tested would be up to the employee. My guess is few people would retest, but would spend their time being tested on new skills.

[kumarski]

If you had to focus on a specific area, it could be building a low cost ABET certified curriculum that has better unit economics than traditional universities.

ABET is pretty legit.

[danieltillett → kumarski]

It could certainly be used that way - in the end it would not matter where you had studied or if you studied at a traditional institution at all, but what you had really leant.

As I mention in my follow on post, it would be best to start in a niche and build out to a more general testing process over time.

[clementangerine]

I'm not sure if this accurately represents how people remember things. It's normal to be a bit fuzzy on a topic a year after learning it if you aren't regularly applying it. What matters is that you can learn it again quickly, and I think preparation for technical interviews works fine for testing that already.

[danieltillett → clementangerine]

My idea is not about how people learn.

The current system of technical interviews works - my idea is about making this process better. The market exists, my idea makes the process more efficient and captures some of the value created.

[skewart]

This is a really interesting idea. I'm not convinced from your description that you have a viable product just yet, but I think you're headed down a really good path. Here are some more thoughts:

- Testing and certification is the most economically valuable aspect of education (at the high end the most valuable test a student can take is getting accepted to an elite institution (I'm using "test" broadly here)).

- Learning itself can and has scaled quite easily thanks to the internet. But testing and certification hasn't really yet.

- There is a huge opportunity to build a new authority for testing/certification/credentialing. (In many ways YC is doing this almost by accident, but it's not their main focus and they're not doing it in a scalable way).

- The kind of knowledge and skills that are valuable in white collar "knowledge worker" jobs are typically not what you study for and get tested on in school. It's more the various soft skills that let you succeed. In other words, it's not the algorithms themselves, it's that you're disciplined and smart enough to figure out how to learn them. The same is true for white boarding interviews - and lots of management roles too - it's not just that you know how to balance a binary tree, it's that you can explain how to do so in a clear way, can handle curve balls without getting flustered, and seem like a good person to work with.

- There are lots of professional certifications that various professions require. But it's not necessarily in anyone's interest for workers to take these kinds of tests more often or in a more rigorous way.

- There probably are some industries where concrete knowledge testing is valuable. I don't know how big a market it is, but it's probably there. One strategy would be to go after these industries. I'd look for fields with high turnover, and/or where companies are losing money due to worker error.

- Another strategy would be to go after the higher end, and try to become a trusted authority for certifying employee skills. I think this is better because this seems like the kind of thing where it's easier to go from high-end to low-end than the other way around.

- Certifying knowledge workers would probably look less like giving them pop quizzes and more like conversations with them, of else just tracking their work closely and assessing it.

- To get it off the ground you need to decide who is your customer - the worker or the employer? It sounds like you're thinking workers will be your customer. In that case the most important thing will be to show a return on investment for a few early adopters. I'm not sure how best to do that early on, but I think you'd want to have some clear benefit that they get - i.e. getting a job or a promotion indirectly as a result, but that probably way beyond your control. Maybe there's something else?

[danieltillett → skewart]

Thanks for the feedback. The description is cut short because of the 2000 character limit here on HN. I did post a follow up comment with the rest of the description, but it has now fallen to the bottom of the comments so nobody seems to be seeing it :(

The basic idea is to bring technology to the certification process in education (a huge industry). Right now all the new education institutions (and a lot of the old) have no way of really proving to the world that their graduates have learnt something and have valuable skills. Employers see a piece of paper from some institution and wonder what does the candidate really know. This is what the idea tries to solve. Now to your specific comments.

Testing and certification is the most economically valuable aspect of education (at the high end the most valuable test a student can take is getting accepted to an elite institution (I'm using "test" broadly here)).

Exactly. The value of an elite institution is not in the actual education you might get, but the hurdles needed to overcome to get in. It is an incredibly crude proxy for measuring what an employee will actually be like.

Learning itself can and has scaled quite easily thanks to the internet. But testing and certification hasn't really yet.

Yes. The reason why is that there has been no way of preventing gaming of the testing system. What you want to measure is if a person really know their stuff, or has just crammed it all in before the exam. My idea is break this link between cramming and test results by preventing cramming.

There is a huge opportunity to build a new authority for testing/certification/credentialing.

Yes there is and once built you will have a very strong network effect that will make the business very valuable.

...In other words, it's not the algorithms themselves, it's that you're disciplined and smart enough to figure out how to learn them.

This can actually be tested too. Once you make a test unable to be crammed then you can start to do the sort of tests that measure real on the job performance.

Another strategy would be to go after the higher end, and try to become a trusted authority for certifying employee skills. I think this is better because this seems like the kind of thing where it's easier to go from high-end to low-end than the other way around.

Yes this is my thinking too. Start in a narrow niche and become the authority for certifying the skill set of people in that niche. If the testing authority say you really know skill X then as an employer you will believe this is true. You can add all sorts of layers to this where you can rank a persons skill as beginner, intermediate, expert, etc. Once you have a strong foundation in a niche then just expand into related fields.

Certifying knowledge workers would probably look less like giving them pop quizzes and more like conversations with them, of else just tracking their work closely and assessing it.

Because the tests can’t be crammed the tests that would be given would by design have to be a much closer match to real work than a typical test. You can make the tests as rigorous and challenging as the market wants. If people want to be certified that are lateral thinkers that can solve real work problems then that is exactly what can be tested. You don’t want to test people on things that they can google in 30 seconds, but the sort of thinking and problem solving that make a real difference in the workplace.

To get it off the ground you need to decide who is your customer - the worker or the employer?

There are two customers - the worker and the education provider (see my follow on post for more details). It would be easier to go after the workers at first, but the educational providers would be where the real money would be.

[skewart → danieltillett]

Awesome. Yeah, I think this could be really big.

> You don’t want to test people on things that they can google in 30 seconds, but the sort of thinking and problem solving that make a real difference in the workplace.

Do you have a plan - or at least a general idea - for how to do this?

Also, thinking about this more, here's another random thought:

Fundamentally the problem that testing and certification tries to address is trust. If I say I can do something you don't have to just take my word for it, you can see that a trusted third party vouched for it. A lot of people have been experimenting with alternative mechanisms for providing trust lately - from AirBnB and Uber relying on reviews/reputations instead of certifications and brands, to even something like bitcoin. Maybe the solution isn't a better testing system, but instead something that removes the need to test at all?

[danieltillett → skewart]

I used to be a university professor so I am highly familiar with the difference between those students that really knew their stuff and those that just crammed well. You really only want to hire the former.

In a sense this idea gets rid of traditional testing. You are not measuring how many facts a test taker has managed to cram into their head, but how much they retain and can apply. Grades as such would become irrelevant as we start to focus on those retained skills that translate into day-to-day productivity.

I do have a plan about how this idea could be best developed, but HN is probably not the place to discuss this. My hope is by putting this idea up I will get feedback or suggestions about aspects I have not thought of. It is a big idea that could change a massive industry like education and there are many different ways it could be developed.