
The "overlearning the game" problem - sendos
http://andrewoneverything.com/the-overlearning-the-game-problem
======
wisty
It's related to Goodhart's Law:

 _Any observed statistical regularity will tend to collapse once pressure is
placed upon it for control purposes._

This is often the result of attempting to overoptimize a system. You can
optimize a race car to a huge degree, because you know _exactly_ what you want
it to do.

You can't optimize a schooling system, because you _don't_ know exactly what
you want it to do. A little noise is a good thing, because the you want a
little wiggle room for teachers to sidestep the dictums of education czars,
and students to sidestep the dictums of teachers.

The Greeks solved this quite a few years ago, with sortition. Under sortition
(injecting noise into elections - <http://en.wikipedia.org/wiki/Sortition>),
Bush and Gore would have been forced to pay "paper, scissors, rock" for the
presidency. Under the US's more pure democracy, they would have been tempted
to make all kind of Faustian bargains with sordid players to nail down the
last 0.01% of votes.

Randomization means that the last percent is just not worth chasing, so
players in a competition won't be tempted to bend the rules for a tiny
advantage.

The same process could be used for tests. If you allocate places in desirable
courses (say medicine) randomly to anyone above a certain score, the top
students won't bother drilling as hard just to get the top score.

Stocks are the same - quants wouldn't sweat timing as much if their placement
in order books was randomized. It would be more efficient to pay attention to
fundamental value than momentary fluctuations if they weren't guaranteed to
make large profits on the momentary fluctuations. Some would still work on
timing, but not as many.

Patents are just bad policy badly implemented at the moment, not over-
optimized.

~~~
tomjen3
I don't think it is a good idea at all. If the political power boiled down to
picking paper over scissors, the loosing side may very well ascribed no
legitimacy to the outcome. That is almost certainly going to end far worse
than actually counting all the votes (i am looking at you, supreme court).

The same problem happens with your exam example. Why is it fair that someone
less competent than you got in? A better solution would be to let everyone
over the cut off point in.

I can't disagree with your example of the stock market, so long as the trades
were still executed with a few seconds.

~~~
SoftwareMaven
It would just be recognizing the jitter that exists _inherently_ in the
system. You will always get a different number when counting the votes, so you
can end up with a "50% voted for both" situation.

Why is flipping a coin worse than pretending the next count will be more
correct? You kind of contradict your own point by implying the supreme court
didn't supply legitimacy to the election. Nobody can blame a quarter for being
stiffed with neocons on the heads side.

~~~
mikeash
The trouble is that getting people to recognize that is really hard, and just
changing how the system works probably won't do it. If you want to politically
antagonize nearly any American with a strong political affiliation, regardless
of what that affiliation is, just tell them that the margin of error in the
2000 presidential elections was far larger than the difference in votes, and
therefore both candidates were about equally legitimate winners.

------
drewcrawford
If you pay a man by the hour, he'll work a lot of hours. If you pay him by the
brick, he'll lay a lot of bricks.

These "games" are basically the equivalent of counting lines of code or
checkins. We're measuring poor proxies instead of the things we're actually
interested in. The solution isn't an arms race to build bigger and better
proxies, the solution is to measure real things instead of artificial ones.

Here's just one example of what I mean by "measure real things". Electing
representatives every X years to decide the laws of the land was once upon a
time the fairest and best way to have the voices of the masses heard. Today it
is feasible to directly poll everybody about every issue, so we no longer need
the proxy. If you say everyone cannot be educated about every issue, fine, I
can "follow" PG's votes on wall street reform and grellas's votes on IP tort
reform and Schneier's votes on TSA etc just by copying their votes on those
issues into my ballot, a permission which I can revoke at any time or on a
vote-by-vote basis, as easy as unfollowing them on VoteTwitter. This is better
than the proxy of professional politicians deciding every issue with fixed
terms.

~~~
redthrowaway
As you allude to, the problem with direct democracy is that people are stupid.
Not all people about all things, but most people about most things. The realm
of human knowledge is simply too vast for anyone to have a broad command of,
so we specialize. This means that, for any given issue, the vast majority of
people are ignorant about its specifics. Thus we cannot expect the masses to
make wise decisions about most issues. The obvious counter-argument to this is
education, that educating the public on the issues at hand would lead to
better decisions. However, that would fall victim to the same demagoguery and
corruption that we see infecting politics today.

By way of illustration, think of an issue that you care deeply about and are
well-informed on. Now ask yourself if you would trust the public at large to
make policy on it. For every pg, grellas, or Schneier, you'll have far more
popular Becks, Palins, and Bachmanns. Direct democracy would be inherently
susceptible to such demagogues, and as such cannot be seen to be any more
reliable, or likely to produce wise policy, than the current system.

~~~
seats
>> The realm of human knowledge is simply too vast for anyone to have a broad
command of, so we specialize.

This statement could actually be used just as easily against voting for
representatives as we do today. The real question is do you trust public at
large to make a better or worse opinion than <name your representative> on
some issue you are deeply informed on? In other words the comparison should be
versus the existing system as a whole, not versus an expert on a single topic.

My perception of the demagogue is that their power comes from railing against
the current authority. So while I can agree that it seems likely that a 'true
democracy' would instantly prop up the Becks and Palins of the country, who do
they blame when a majority of people vote that they are wrong? Or maybe even
more extreme who do they blame when their views and proposals are voted for
but still fail?

There is no longer a 'bad guy' to blame and arguments put forth by a demagogue
will have to deal with two new realities-

1- They can no longer claim they are part of the majority opinion in an effort
to gain further support. It'd be hard for Sarah Palin to keep up her 'regular
joe' persona and reconcile it with being in major disagreement with a majority
of the country

2- They can no longer claim obstruction by some authority steam rolling them
in legislature. It would become pointless to argue about which pundit
supported which policy or law in the past, because every individual would have
their personal voting record.

~~~
redthrowaway
>> This statement could actually be used just as easily against voting for
representatives as we do today.

True. I'm no big fan of representative democracy, it just happens to be the
least bad system we've tried.

>> The real question is do you trust public at large to make a better or worse
opinion than <name your representative> on some issue you are deeply informed
on?

Well, yes, because they are at least (usually) experts in business or law,
which are the domains that are most important to government. Some will be
military or foreign policy experts, and they will (hopefully) be appointed to
the relevant subcommittees. It's a fairly broken system, open to all kinds of
abuses and with little assurance of wise leadership, but I still think the
competence of the average politician in crafting policy is higher than that of
the average citizen.

>> My perception of the demagogue is that their power comes from railing
against the current authority.

I think there's quite a bit of truth in that. I don't, however, think it's a
necessary truth. Hitler was a classic demagogue, even and especially after he
gained power. He appealed to antisemitism and a fierce nationalism, rather
than to the incompetence of politicians. While the current crop in the US gain
power by railing against the established political class in DC, that's not the
only possible source of power for them.

>> It'd be hard for Sarah Palin to keep up her 'regular joe' persona and
reconcile it with being in major disagreement with a majority of the country.

Reconciling her views with reality has never been particularly important to
her, and I don't see how that would change in a direct democracy. She could
just as easily rail against the liberals in the MSM who poison the electorate
with their propaganda.

Ultimately, I picture direct democracy working quite similarly to the reddit
hivemind, and I'm not convinced that's a good thing. It would likely be
passionate and dynamic, but liable to go off the rails from time to time.

------
jacques_chester
Systems theorists say that "structure predicts behaviour". It's a bit trite,
but also deep. Here's an example.

The US political system pretty much guarantees bad budgeting. Members of
Congress are elected fairly independently. There is no party discipline, so
each member will operate independently to maximise pork. This encourages
horse-trading within and across party lines; nobody can be forced to give up
something for a general good.

There's more: there's no incentive to balance the budget. The Executive's
separation means that Congress does not need to concern itself with proper
administration; it only doles out the cash. It has every incentive to ...
maximise pork.

The dynamic behaviour of American government arises from the static structure.
The drafters of the constitution drew on their knowledge of history and
current affairs to try and avoid certain pitfalls. The US Constitution was
state-of-the-art when it was written. It's less so now.

Countries where the Executive is formed out of the Legislative -- the
Westminster system -- tend to have much stronger party discipline, because
that discipline is required to pass budgets, enact legislation and to form the
Executive. This tends to almost eliminate horse-trading, except between
parties and independents. It's not perfect -- whole parties can engage in pork
too -- but when policy emerges that benefits the many at the cost of a few,
countries with party discipline will find it easier to adopt than those
without.

Australia, which has the amongst the toughest party discipline in the
democratic world, is also a reform leader. And I think a lot of that is
explained by our constitutional arrangements.

------
loup-vaillant
I think "overlearning" is not a good term (edit: it is quite accurate,
though), because it tend to suggest the worst solution of all: that we
_refrain_ from learning. I prefer "lost purposes"[1].

The primary purpose of a game is generally to Have Fun. This purpose is lost
if you have "solved" the game. The stated purpose of patents is to foster
innovation. However it doesn't work[2]. The purpose of schools is learning.
However its methods are flawed [3,4]. And so on.

The trick is to know your goals, and then find out means to best achieve them.
The author said:

> _But, in real life, we need to keep "playing the game": we need to have
> elections, and protections for inventors, and laws that govern society, and
> a market where companies can raise money._

But the _actual_ goals are different: We don't need election, we need a
working democratic system (which may, or may not, mandate elections). We don't
need protections for inventors, we need innovations. We don't need laws, we
need a fair and working society. We don't need a market where companies can
raise money, we need a working economy.

Well, I could attempt recurse further up until pure morality, but that would
be intractable. But at least you get the idea. If something looks broken,
think about its ultimate purpose before you try to fix it.

[1]: <http://lesswrong.com/lw/le/lost_purposes/>

[2]: <http://www.dklevine.com/general/intellectual/against.htm>

[3]: <http://vimeo.com/5513063> (Dr Tae)

[4]: <http://www.khanacademy.org/>

~~~
jorleif
Overlearning/Overfitting is a term from machine learning/statistics that means
that a learning system fits its learning data so well that it does not
generalize to new data. The examples given in the article seem like perfect
examples of this, e.g. elected officials fit the election process very well,
but do not "generalize" to leading a country.

~~~
loup-vaillant
Ah. I didn't see it as jargon. I recognize the term is accurate, but my main
point still stands: the _connotation_ is still there, and still does suggest
we should learn _less_ about those systems if we want to preserve their
original purpose.

Worse, the end of the post suggest we _have_ to preserve things like schools,
markets and elections, even though we learned enough to spoil them. Therefore,
the only thing left to sacrifice is learning itself.

That would really really sucks: I'm curious, and I always want to learn more.
There are many things I would sacrifice before I accept to put hard limits on
what I am allowed to learn.

------
Joakal
Hey USA, your first past the post system actually does tend towards dual party
governments [0]. Compare it to preferential system where voters can pick
several parties or vote for one party for the preferences listed.

For example, look at this:
<http://www.abc.net.au/elections/federal/2010/guide/deni.htm> With USA's
system, Australia Labor Party would have got re-elected.

There's still improvements to be made for democracy that doesn't need to
involve such fancy technology yet [1].

[0] <http://en.wikipedia.org/wiki/Duvergers_law>

[1] <http://techpp.com/2011/06/08/homomorphic-encryption/>

------
shubber
I'm reminded of Mechanism Design: a Nobel prize winning theory of economics
that starts with the supposition that agents in any system will exploit its
rules to maximize their personal gain. The corollary, which I've been quite
taken by, is that where we can influence the rules of the game, we should
design them such that exploitation serves a social good.

------
erikb
Many people see this from a macroeconomic or political point of view. So I
think adding a more philosophical point of view might also be of value. Of
course, because it is philosophy, I can only present my own point of view.
There is no right or wrong.

For me, as a Zen student, the solution is to just accept that that it is, what
it is. No system (shape, for Zen students) can be perfect. It is created to
solve a problem and with time starts to fail badly at doing anything about the
problem. Then another system is created by someone else and the cycle
restarts. You might think for example that the stock market or democracy is a
thing that doesn't change, because it exists longer then you live. But in the
end it will change. We had different political systems before and we will
develop different systems in the future.

So in the end the system is one of the most important things we have, because
it gives us something to base our decisions and actions on and goals to strive
for. But also the system is nothing, just an illusion we create for us, maybe
based on how we understand the illusions other people created for themself.

The thing that is interesting to me personally is that every system itself is
instable and will change or die in the end. But the life cycle of a system,
what it does for us and doesn't, that all will always stay the same and even
though we try to change that, we will never succeed. So while it changes a lot
in one way it is totally unchangable in another. But that just as a side note.

------
groaner
The cynic inside me would extend this problem to the entire economy or even
human society itself. Any economic transaction is supposed to benefit both
parties: the buyer offers an amount he is willing to pay for what he wants,
the seller provides his goods or services at a price he is willing to accept.
As PG might say, wealth is created in tandem with creating stuff that people
want.

The problem is that some elements in society have become extremely effective
at creating a perversion of "stuff people want" for their own benefit. They
exploit loopholes in the system, whether by preying upon the poorly-informed,
shifting costs onto externalities that we can't price properly, engaging in
corruption, or producing items of questionable actual value but very
attractive perceived value. They take advantage of our desire for easy answers
to our problems with minimal effort expended.

Case in point: Ponzi schemes, coal power plants, Halliburton, the tobacco
industry, Zynga, and even to an extent, religion

We've tried to control this with laws, education, and social norms, but
ultimately it seems the invisible hand reigns supreme.

------
drhouse_md
This article seems to be addressing 'the spirit' of a document or system.

Perhaps another way to look at the problem is to imagine creating an A.I. that
you want to succeed at whatever system you present. In most cases, an A.I.
will take the literal interpretation of the system and become a test-taker, an
electable 'gotcha-game' politician or even an entity that finds it can
maximize game theory to its own ends by complicating the rules of an existing
system to the point of absurdity once it becomes powerful enough to modify and
create rules.

So then how do we create systems resistant to beings that take everything
literally? I suppose the only way is to reward certain outcomes as opposed to
rewarding the direct product of the system itself.

Examples: After an election, have we elected someone who has met with a high
degree of favorability in the electorate by the end of his term?

After having students become proficient test-takers, do they then become
excellent doctors, lawyers, engineers, etc.. ?

In a game invented to be fun or fair, once overlearned, do they produce fun or
fairness?

If not, then a new game needs to be created or the existing one might need to
be extensively modified to produced the desired product. This is where the
internet shines, where everyone is welcome to take an existing system and
modify it to something better. The problem with politics, law, stock market,
etc... is that they have become the only method adopted in real society (there
is only one game in town).

If reality were allowed to adopt, incorporate or evolve from systems/games
from virtual reality (i.e. internet) there might be some productive change.
But first we would need to see the first step taking place, that being even
flawed virtual systems are allowed to manifest in significant proportion
within real life society.

Don't hate the game, hate the player. Every game that profits a winner will
have its cheaters.

------
DanielBMarkham
We run into this situation in programming quite a bit. The answer is
refactoring: taking the principles and patterns that work, streamlining them,
ditching the cruft that works for a few edge cases but mostly gets in the way,
simplifying the underlying metaphor of the system, and re-asssembling.

The only way to do this with much larger systems, such as systems of
governance, is revolution or exploring new lands. Personally I'm a bit
concerned about revolution -- the assumption with refactoring is that the
people refactoring understand what the "good" parts are and what the "bad"
parts are. They also need to be able to generalize and simplify in order to
keep the system cognitively approachable. In my experience, it's very easy to
be angry-tear-down-the-system guy, very difficult to actually refactor. As the
author points out, it's not that these systems are entirely useless. The hell
of the thing is that the reasons for creating these systems are still very
valid.

I remain convinced that programmers (hackers) have a lot to add to the
discussion when talking about complex, brittle systems. After all, we spend a
lot of time both working with them and fixing them. To me, programming and
systems architecture is applied philosophy. Very cool stuff.

~~~
jacques_chester
Refactoring is what political types call 'reform'. Reformations are safer than
Revolutions, because revolutions dissolve all civil structure. They allow the
ruthless sociopath to prosper.

What programming has taught _me_ is humility in the face of complexity.
Refactoring only works because we can, at a stroke, reverse our changes;
because we can place those changes in isolation and test the heck out of them.

But society, constitutions, laws: they are not software systems. A revolution
can't be rolled back. A constitutional change is a hotfix applied to a running
system. It's important to be careful.

You might find reading Burke on the French Revolution interesting; also
Oakeshott on conservatism (the intellectual kind, not the god-n-guns kind).

~~~
DanielBMarkham
I think we completely agree. I did not mean to imply anything but humility was
indicated here.

"Reform" is a big, fuzzy word. A lot more brittleness and complexity has been
added under that rubric. Systems of governance eventually get "over learned",
that is, people adapt to the system and use it in more ways than it was
intended. The ability of hundreds of millions of people to adapt is much
greater than any amount of complexity that we can create. Yet we persist in
the idea that by continuing to "tweak" the system, somehow magic will come
from it. To me this is where humility comes in. We can never create a perfect
government. The best we can do is have one that balances stability and
refactoring.

Election cycles and constitutional amendment processes were supposed to do
this. What we've found, though, is that legal precedent, statutory code, and
an expansion of pure democracy and the role of career politicians, given
enough time, puts us exactly where we didn't want to go.

Interesting side note: overly complex and brittle systems that people execute
actually give dictatorial powers to those responsible for executing them. That
is, in a computer system, the complexity creates program crashes and
dysfunction, because the computer has to treat each instruction cycle and
piece of data the same. In a system executed by people, a subtle social goal
takes over: things still appear to happen. The system still superficially
looks like it is working somewhat -- those in authority just selectively apply
and bend the rules depending on whatever their whims are. If you've ever
worked in a large organization with too much process, you've easily observed
this: most struggle under burdensome rules, never being able to get anything
done. A few, however, work completely outside the system, given special
permission. These folks are usually kept off the radar so as not to upset the
troops. Complex people systems don't ever _stop_ working, they just make
things miserable on most everybody and give a few permission to do whatever
they want.

You'd think that modern tyrannies would look like old Libya: one guy in a
funny hat ruling totally. But that's not the way it works any more. It's not
one guy, it's a distributed oligarchy, and it's not out in the open, it's
obfuscated.

------
kragen
Less Wrong talks about this a lot; their term for one variant of it is
"superstimuli".

What's the difference between your "overlearning" and "hacking"? They sound
like the same thing.

This phenomenon is the reason for the Wikipedia rule, "Ignore All Rules".

~~~
sendos
> What's the difference between your "overlearning" and "hacking"? They sound
> like the same thing.

They are similar, but I think one difference between "hacking" and
"overlearning" is that a hacker can learn to subvert/circumvent a system quite
quickly, and usually does so in a way that breaks the rules. Overlearning
usually takes longer to achieve, and is a more insidious problem because it
achieves its goals without actually breaking any rules.

~~~
kragen
It depends on what you think "the rules" are. Generally speaking, hacking
happens in a framework that has unbreakable rules, whether those are rules of
a game, of a CPU, or of an OS. A solution becomes a hack when it exploits
those rules in a way that was previously thought impossible.

Of course, there may be _other_ , breakable, rules that are broken — "You
can't render 3-D in real time on a PC!" — but those aren't the relevant rules.

Many hacks are not quick at all to achieve.

~~~
sendos
I guess an example of hacking would be someone hacking the phone system to
make free long distance calls. An example of overlearning would be someone
learning the rules of society to such a degree that he gets free long distance
calls (e.g. using lawsuits against phone companies).

The former would get fined or imprisoned if caught, but the latter is fine
under our legal system.

~~~
kragen
It's pretty bizarre that you're using the derogatory definition of "hack"
given that you're posting on _Hacker News_. And it doesn't incline me to want
to talk to you further.

~~~
sendos
Sorry, I was just using the colloquial meaning of hack. Did not at all mean to
appear derogatory for the type of hacking exemplified by HN, of which I take
part in.

In any case, I do see your point about similarities between hacking and
overlearning, but I think there are some differences. If I can articulate
those difference better, I'll reply a little later.

~~~
DasIch
I think overlearning can be considered to be a subset of hacking. In both
cases laws (as in physics) are exploited however while hacking may or may not
break rules as imposed by society overlearning does not.

------
Jun8
Very interesting post!

However, I got hooked on thinking about the childhood game that he mentions
briefly in the introduction. Asking more and more questions about an object
and wanting all of them to be consistent is a good description of mathematics.
According to Godel, both teams are _bound_ to lose, because you cannot create
a system of descriptions/properties about a system that are self-consistent
(as their number increases), there will always be questions to a team whose
answer will be inconsistent with the previous set of questions. The game is
then to see which team can push the inevitable further.

IDEABOLT: It would be interesting to develop a program that plays this game.
Each answer could be stored as an RDF statement in database.

------
aconbere
There was a book written in the 80's that covered the overarching
philosophical discussion on this kind of thinking about games. The author
broke games into two categories: finite and infinite.

Finite games like the one invented by the children described by the OP have
end states, they have winners, and the goal within them are always framed from
that perspective. While infinite games have no winners, have no end, and the
goals are often framed around ensuring that the game never ends.

It's easy to think of finite games in our lives, they are everywhere in our
society, and the OP does a good job of pointing out some of the less obvious
cases. Infinite games are less obvious, the one that I found most illuminating
was "language", a game where people actively collude to extend the game to the
end of time, and has no winner.

Anyay, it's an interesting read, but a little bit mumbo-jumbo by the middle of
the book. I would recommend it.

[http://www.amazon.com/Finite-Infinite-Games-Vision-
Possibili...](http://www.amazon.com/Finite-Infinite-Games-Vision-
Possibility/dp/0345341848/ref=sr_1_1?ie=UTF8&qid=1314621221&sr=8-1)

------
resatori
I think it is time we recognize that it is not the systems that shape our
world but the individuals.

There is no optimal system, be it political, economic or whatever.

You can always find loopholes.

I think it is time we take responsibility for what we do - then there is no
need for a better system.

The patent system is not responsible for people attacking each other - its the
people.

------
One_adm12
Great post. I face this everyday in my job at an "old media" company. There
are policies, rules and constraints in place which served some purpose at some
point in time, and haven't been challenged for years. Unlearning the rules of
established games is almost as important as evolving them. Everything should
be questioned and challenged to find the root of the point of the
rule/law/restriction and if it doesn't make sense any longer, throw it out.

------
thewisedude
Doesn't "Exploiting the System" capture the same idea? I think basically what
people are doing here is understanding the core weakness that is intrinsic to
the system(which probably is not perfect) and using(abusing) it to their
advantage.

------
donaq
Regarding politics, it is interesting to note that the people have not, for
their part, also overlearned the game of electing the best man for the job. I
wonder why this asymmetry exists?

------
useflyer
Have you noticed that yesterday every comment was concise, and now that
applications are open, every comment is elaborate, long-winded, and footnoted?

------
stretchwithme
Seems most of the things mentioned are an inevitable result of centralizing
and expanding government power. Someone is being gamed alright.

------
Greedy_Fools
People are predators and that's what predators do, never stop searching for
what they need. Predatory behavior is the problem, and there is little will in
successful predators to outlaw there own behavior.

