

Ask YC: A tough problem - how do I pick African children for a tech scholarship? - markessien

My opinion has been asked for a unique task - how to select which children should get a scholarship based on ability.<p>The situation is this - there is a large school in Africa with about 1500 students from ages 6 to 12. The students come from wildly differing income groups, and some have no problems paying the fees, and some can just barely afford the fees. There are various programs in place to help the really poor off, but the school has to maintain fees to run.<p>The school would like to encourage students who are talented, but don't have a lot of money by giving them scholarships - i.e, academic scholarships.<p>But there is a HUGE problem - the scores of all the students is exactly the same as their family income. The richest students come out at the top of the class, and the poor end up at the bottom. So simply setting an exam and giving the scholarship to the best always ends up with the scholarship going to those who don't need it. All tests always follow the same curve, and are always directly proportional to the income of the family.<p>The teachers cannot be trusted to give accurate assessments of the children, because a lot of the teachers have family/relatives in the classroom.<p>How can this problem be solved? How can we identify students who have ability, but are not particularly rich? And using a system that is somehow fair and not completely arbitrary?
======
lee
Speaking as someone who was a beneficiary of several talent/need based
scholarships, I'd like to suggest a slight refactoring of your problem. Allow
me to invoke Gladwell. Think of this as trying to help someone get their
10,000 hours in to become an expert.

I was lucky in that I received scholarships to 'be' at a prestigious schools
or programs, but always had to work at jobs much more than my peers not only
because of my family's need, but also because of my family's attitudes about
what children were for. The wealthy kids' families had a much different idea
of how to raise a child than my family did and they had a different
understanding of the types of resources beyond the financial that kids need to
be successful.

Just because I had received a scholarship to excellent schools did not release
me from the expectation that I needed to contribute financially to my parents'
well-being while I was in high school and college. In fact, it increased their
expectations.

The pressure was intense. I knew that the only way for me to get out and to
stay out was to do well academically, but relinquishing duty was anathema. The
schools where I went were hard, with heavy academic workloads, and having to
work around 35-40 hours a week through high school and college and keep strong
academic standing extremely stressful and time consuming. Very difficult to
get the 10,000 hours in when you're asking people if they want fries with
that.

Kids need time.

If it's as all possible, interview the families of the most likely candidates.
Look for red flags. Will the parents be able to afford or allow the child to
actually reap the benefits you're offering? If they can't afford it, are they
willing to give the child extra time to pursue academics more seriously or are
they likely to consume their young?

When you do give out the scholarship, please spend time with the family to
help them structure the student's life so they can properly take advantage of
what you're giving them.

------
ChaitanyaSai
This is what those OLPC laptops are for.

1)Pass one around. Use some flash based game, which is actually an aptitude
test in disguise and measure performance on that.

2) Have a Webcam conversation with each (if 1500 is too many for your team,
farm this task out. * ). See how they answer non generic questions and look
for evidence for analytical abilities or quick thinking.

I think some internet-based live-streaming solution is the way to go as there
is no corrupting intermediary that adds noise to the evaluation process.

*Would be cool to have a website where you can sign up to interview a student, something like MTurk, but limited to invited candidates?

~~~
markessien
This is actually a good idea, but one could just use normal computers. The
school has 30 PCs or so with an internet connection.

All the same, you can't interview 1500 students over a webcam, and what can
one learn in a 10 minute interview? It's like a job interview then, and they
are a bit young for that...

The best would be if there were some type of test that one could administer
that does not show the same bias towards the wealthier students.

~~~
ChaitanyaSai
I, of course, have never done this, but my hunch is that it should be easy to
ask students questions that reveal some abilities. I am reminded of one of the
studies cited in Gladwell's Blink about how you can thin-slice (speedily
evaluate relative merit) a person's abilities in pretty diverse fields (speed-
dating, classroom teaching) with only a 10 (or so) second video of the person.

No test administered by classroom teachers can escape corruption, unless the
teachers are themselves unaware of what constitutes a right answer.

------
curiousgeorge
Regression analysis. Then factor out the expected effects of wealth.

~~~
espadagroup
This would probably be the best and most objective method. Even though income
plays a large roll you should still be able to to tell out of 1500 a few
students who perform better than their income levels would indicate.

------
byrneseyeview
It looks like you've completely skipped over the possibility that intelligence
and ambition are heritable -- whether genetically, or through parenting
strategies -- and thus that academic ability and parental income correlate
very well. I think this pattern of wishful thinking over actual data is part
of why aid to Africa has been so counterproductive.

~~~
nanijoe
Not to nitpick, but you may yourself have missed the fact that ambition does
not always translate to a high income. This is especially true in a third
world country where a good percentage of rich people get to be that way by
simply stealing money from the government

~~~
byrneseyeview
I disagree. In a first-world country, an ambitious person might claw his way
up to being regional director of cardboard issues for the EPA, and could earn
low six figures. In a third-world country, that same ambitious person would
get to that same level of authority, with the same proportionate salary -- and
would be earning many times that amount in bribes. Outside of Western
countries, there are very few things an ambitious person could achieve that do
not in some way translate into money.

------
tsally
Well if the scholarship is being given out based on ability like you say, you
have no choice but to give it to the students who perform the best.

If there is a need based component, it would be different. But you do not seem
to indicate that there is. I'm all for leveling the playing field, but if the
person who created the scholarship wanted it to be talent based, that's what
you have to go on.

EDIT: We also have a relationship between income and test scores in the US.
It's a fact of life.

~~~
markessien
Well, the point of the scholarship is to give academic scholarships, but to
those students who need it. I.e, encourage a culture of learning, but avoid
giving to those who already have. It's not to be based solely on ability, but
also on need.

~~~
tsally
Don't all students who want to study technology in Africa need it? I would
imagine there are not too many opportunities.

If that isn't the case, you could give the scholarship to the middle of the
pack. This would maximize the need and intelligence of the people that the
scholarship would help.

------
bradcbennett
I co-founded a group that provides secondary school scholarships to children
in Southern Tanzania and we ran into a very similar problem (jiamini.org).
Like your scholarship, ours was based on a combination of need and merit where
the ideal candidates are those that have the capacity to excel in school, but
don't have the resources to fulfill their potential.

The first step in our solution to the selection problem was to have children
nominated for the scholarship by teachers, headmasters and other community
leaders. Nominees could not be direct relatives. They had to be in the top 10%
of their class or had to be vouched for by community leaders that they were
capable of this type of ranking given changes in their home life. They also
had to have a difficult home life - i.e. living with relatives living with
AIDS, orphans, living in a mud hut, etc. We also required any set of nominees
offered up by a teacher or headmaster to be half female.

Given a large enough sample size - typically we select about 15 kids per year
from 2 primary schools - we are able to get about 30 nominees. School ranking,
sex and kinship are relatively easy to verify. We can't directly verify home
life or income so we provide a simple questionnaire for all nominees to fill
out. Answers provided on the questionnaires are then confirmed by other
teachers and community leaders we trust.

After all of this, it is relatively easy to identify a group of 15 kids who
demonstrate ability and have few available resources. In other words, we've
also found there is a strong correlation between income and scores, but there
always seem to be kids who score high from the low income groups. People at
school and in the community often know who these kids are. You're right, you
can't trust a single source of anecdotal information, so the key seems to be
to build in redundant sources of qualitative data. This approach won't pass an
audit with the same ease as a regression analysis and it may miss some ideal
candidates, but it does help find some kids that could use the help.

What group are you working for?

------
bokonist
Are the test scores you refer to based on learned skills, or on intelligence?
If the former, you could try giving the students a test like the Raven's
Progressive Matrices.
<http://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices> It's a test of
intelligence that does not require the student to be able to read, know math,
or possess any cultural knowledge. It might be useful for finding bright
students who were just not brought up in an environment that encouraged
learning.

If that doesn't work, I suppose the thing to do is give the scholarship to the
highest performing student who could not otherwise afford the tuition.

~~~
markessien
I certainly like the idea. All that one has to do is now somehow combine it
with a social factor where we can filter children without hurting their
feelings.

I.e, a student from a wealthy family who scores high deserves special
attention, but giving a scholarship does not help the student, because the
family can afford the tuition anyways.

~~~
bd
Maybe you could split "reward" into two parts:

a) some award that will be prestigious by itself (by being labeled as one of
the top kids from the school); and

b) financial help available for winners of a) with family income under X.

Isn't it how US system works? Getting high SAT will get you to Harvard
(prestige part of the reward), and if your family has income under $60K, your
tuition is dropped (financial part of the reward selectively applied to poorer
students).

------
jamess
Have you tried the simplest possible means, merely asking them? Ask students
who want the scholarship to apply, and discourage those who already have the
means to pay from applying. Then simply award the scholarship to the most able
from the group of applicants. Am I being naive in believing that this trivial
solution works reasonably well in most other such situations?

~~~
markessien
They will all say yes. It's free money in a poor country, who would say no?

~~~
jamess
Are you sure that's not a bias you're bringing to the situation? I fail to see
how it being a "poor country" comes in to the matter at all. You could say the
same about any country, that people aren't going to turn down "free money" but
in reality they do, on a daily basis. Do you have any evidence to suggest that
this is the case, or is that just a knee jerk reaction?

~~~
markessien
Yes, I have a lot of experience, and it most certainly won't work. Not only
that, it's children between 6 and 12. They are not going to make rational
decisions.

~~~
cabalamat
As I understand it, your position is:

1\. they will not turn down free money

2\. not turning down free money would be rational

3\. they are not going to make rational decisions

Surely all three of these cannot be true? Or do I misunderstand you?

~~~
markessien
Okay, don't try to confuse me with logic :) Fact is - asking the students does
not work. It has been tried, it does not work, they just take everything they
can get.

~~~
brentr
This is an amazing statement. It is sad that those who want something can't
get it, while at the same time those who are offered many chances take those
chances for granted.

My comment is meant to be a blow to the children of the United States. I went
through the U.S. public education system. I could never figure out why so many
people chose not to take advantage of the education the state was required to
offer.

------
DaniFong
Perhaps a tech scholarship is the wrong tool in this situation? What would it,
invested in a corresponding number of xo's, or in a technology teachers
program, give you instead?

------
wumi
Africa is a big place. What country/city? (it might be the difference between
saying a large school in the Americas vs. a large school of 1500 in Chicago)

------
adamb
i'd suggest a simple split:

\- select the highest scores on the exam without question (these students will
make your program prestigious)

\- normalize actual scores based on "expected scores", select highest
normalized scores (these students will highlight your program's accessibility,
rewarding those who do much with little)

~~~
markessien
Who decides what the 'expected score' is then? This would have to be the
teacher, and as I mentioned in the post, the teachers cannot be trusted to be
fair.

Also, it's just 2 scholarships per class of 30 people per term. So one just
has to pick 2 people from the class.

~~~
adamb
you mentioned that when you plotted score vs income, there was a clear
correlation- i was suggesting using this data to normalize the scores.

if trusting teachers is an issue, there are two options.

\- polling multiple teachers (hopefully eliminating individual bias)

\- polling multiple students.

for polling teachers, you might suggest that each teacher recommend 4
students.

for polling students, you might suggest that each applicant write 2 essays:
why they should receive the award, and which of their fellow classmates should
as well.

correlating these with exam scores might provide a clearer picture still.

~~~
markessien
The students get one teacher per year. I.e, they don't have multiple teachers
for each subject, they just have 1 teacher who takes them for an entire year,
then they get a new teacher.

The essay suggestion would work maybe for the 10-12 year olds, but the 6-10
year olds would not be able to write that well. Also, would they not just pick
their 'clique'?

~~~
adamb
\- ask for input from the previous year's teacher

\- students only pick one other person, so picking their entire clique
wouldn't fit as an answer. further, as good students tend to find one another,
it's a good filter for finding people that should be recommended by the
teacher but aren't (or vice-versa).

\- if a full essay is too much, just ask them to write what they'd do with
their scholarship in a few sentences (prose, poem, any form will do). since
you're only looking for 2 kids, you only need a question that will highlight
the top candidates. no need to properly rank the entire class.

------
sachinag
I've spent some time thinking about this - and you have a devil of a test
issue. You need a test that doesn't correlate with income, but that can be
seen as fair and transparent.

You're better off doing an evaluation that has absolutely nothing to do with
actual skills. If you measured, say, reaction time, you'd end up with
something that's just not likely to be correlated at all to income - just to
innate ability and practice. Assuming all the kids from all the income levels
would practice the same amount and that families of wealth do not have higher
reaction times, you're going to end up with "winners" that aren't limited to
the high income.

Doing a test based on reaction time seems incredibly unfair - you're not
looking for kids who are the most promising from the get go - but you're
guaranteed to not just get the wealthy. In the end, you'll be helping more
kids who need the scholarship than you would be with an aptitude test. Since
that's the goal, this plan has a net benefit. It's not perfect, and it's not
pretty, and it's probably duplicitous, but you'll at least have a better
outcome than you would with your existing tools.

Ideally, yes, you'd want to find a "fair" aptitude test that wasn't income-
correlated, but that is going to be incredibly difficult to do.

~~~
markessien
You know, you may have a point there. If one makes a test that involves
something like who is the quickest to figure out how to use some program like
excel, this _may_ not correlate to income, assuming that none of the children
has previous experience with excel. But on the other hand, it may still
correlate with income.

It's just a waste to give a scholarship to children of extremely wealthy
people when there are people struggling to put their children through school.

Ideally, what one would like to do is make a test, then just eliminate all the
children from wealthy homes, and then give the scholarship to the top of the
list. But that would cause resentment if everyone took the same test.

Also, remember, it's just 2 students from a class of 30.

------
asm
Perhaps an assessment of learning ability rather than achievement would
distinguish students who could get the most out of a scholarship.

Perhaps by assessing all students to find the easiest concept the child has
not mastered, teaching it to them, and then measuring their ability after
being taught, we would be able to measure their ability to use new information
effectively.

------
ashleyw
So overall:-

    
    
        - You can't base your choices on the results
        - You can't base your choices on the teachers opinions.
    

…the only decent way I see is going and spending a term with the kids
yourself, for an unbiased view. Not sure if that an option or not, but I see
no other way!

Good luck and keep up the amazing work your doing! :)

~~~
markessien
I can't spend time with all 1500 students myself, which is why we need some
method of determining the students. And if I spend time in a class and simply
pick someone, that would appear too random, which will be demotivating to
students who try hard.

~~~
ashleyw
Ah yes, sorry — the "1500 students" part slipped my mind when writing my
message!

Really unsure to be honest…its an extremely tricky situation!

------
jpark
1.Develop an exam that determines a minimum or base ability to take some
advantage of the opportunities provided by the scholarship. Randomly select
from the pool of students who pass this test.

Or

2\. Split the scholarships in half. Half are awarded purely based on ordered
rank for scores on an exam. Half are awarded randomly.

I think you need some type of random element to ensure that those who are
merely disadvantaged by income or who are later bloomers for their age can be
awarded scholarships. The random nature would also eliminate the subjective
nature of teacher assessments.

That said, maybe I'm being overly influenced by having just read Outlier by
Gladwell. Flawed as I think the book is, it did have some good nuggets.

------
vitaminj
How about getting the kids to submit a little creative art project, like a
drawing, short story, plasticine figure, whatever. It could probably be left
up to the child to decide what to make, and you just need to provide them with
a theme eg. animals in the jungle or something. Give them 2-3 hours in a
classroom to complete the task and then collect up the pieces for assessment.

The art projects could be used in conjunction with the test scores as another
data point on which to base your decision. Presumably, it would provide some
insight into the creativity / imagination of the kids, which I think is of
equal importance to raw academic numbercrunching ability.

------
ericb
Use a long, challenging, tedious, or better yet, puzzle-based, task to measure
persistence. Award the most persistent students the scholarship.

------
bayareaguy
I'm not sure you can solve the problem if you can't trust the teachers. In Rob
Gifford's "China Road", there is a story of a chinese girl who had her
identity stolen by a corrupt teacher. She was told she failed a school
entrance exam when in truth she passed - the teacher helped another student
from a wealthier family use her identity to get admitted to a school.

------
d0mine
Take a look at PISA results (just an example of formal evaluation of teens
abilities): <http://news.ycombinator.com/item?id=245192>
<http://news.ycombinator.com/item?id=257836>

------
joubert
You cannot do this from your armchair.

Go to the school; you'll figure out what to do once there.

~~~
markessien
I've been there, still could not come up with a good solution.

~~~
run4yourlives
Which kids did you like the best? Best being defined as which do you think
would make the best use of the scholarship both for themselves and their
family, and for their peers and country?

This method is as good as any. I know you want to make the best decision
possible, but this is one of those situations where you just have to go with
your gut and to a large degree throw the numbers out the window.

------
fragsworth
How about you give the aptitude test and scholarship(s) only to the poorest
students? Assuming the scholarship is being donated because Africa is poor,
this seems the most obvious thing to do. Also, the poorest students and
families would be most grateful for it.

------
hadley
Why is wealth correlated with test scores? Is it because wealth is correlated
with intelligence (in this case presumably mediated through diet)? Or is it
some other factor?

What is this scholarship and how is it expected to help the students?

~~~
markessien
I don't know why wealth is correlated with test scores scientifically, but my
theory is that:

1\. The children of parents who are rich are likely to have parents who are
university professors or successful business men

2\. The children of wealthier parents have more free time to concentrate on
their school, instead of helping around the house

3\. The parents who are wealthier are generally more ambitious, and make their
children study

4\. The children from richer families have access to after-school lessons,
better books

5\. The children of wealthier parents are more likely to have someone who
takes care of them all the time, while those from poorer families probably
have parents who are working all the time

6\. Diet of course

The scholarship comes from the school. The school is pretty big and tries to
give a lot back to the community. So there is always an attempt to help
whatever students require help. The poorest children get educated for free,
but this particular program is there to help those who are perhaps in the
lower 3rd income class, but show strong potential.

------
raquo
You could incentivize an increase in performance rather than performance per
se. Broadly said, I know, but you know more about this than I do, you'll
figure out how to do it better.

------
anamax
Don't look at test scores, look at how test scores change.

~~~
markessien
This can easily be gamed, as students will simply drop their scores one term,
so they can get the scholarship the next term when they improve.

~~~
anamax
That's why Hamming invented filters.

------
tlrobinson
How about having interested students write an essay?

------
drhowarddrfine
You don't. There are plenty of disadvantaged children in your country who
really need your help, will use it more wisely, and will gain from it than you
will ever find in Africa.

Dumping money into Africa is a huge waste since it will not save anything or
anybody. Never has, never will.

