
Markets are efficient if and only if P = NP - confluence
http://arxiv.org/abs/1002.2284v2
======
meric
A while ago I studied a paper where the author proposes using high frequency
trading to exploit price patterns. If that can be done profitably it would
mean markets aren't even weak form efficient! Lately however there have been
emerging views that "efficiency" is not black or white but a continuum. i.e
The market will only be efficient enough such that the marginal cost of making
the market more efficient is equal to the marginal benefit of doing so.

Anyway, I wonder if solutions to NP-complete problems can be approximated
using a market.

EDIT: According to the paper, they can.

~~~
justincormack
"The market will only be efficient enough such that the marginal cost of
making the market more efficient is equal to the marginal benefit of doing
so."

Do we have a model of what markets are like under this type of model? It is
not clear to me that prices are even close to what efficient ones might be
under this situation (though they could be).

Also of course most of the missing markets are the contingent and future ones,
which suggests the ways in which things might be biased (forward planning, the
firm, the business cycle etc) which are quite important inefficiencies.

~~~
meric

      "Do we have a model of what markets are like under this 
      type of model? It is not clear to me that prices are even 
      close to what efficient ones might be under this situation 
      (though they could be)."
    

I don't know what markets are like under this type of model but here are a two
papers recommended by my lecturer on how efficiency is not an either-or
proposition.

 _Adaptive Markets and the New World Order_

"Under the AMH, markets are not always efficient, but they are highly
competitive and adaptive, and can vary in their degree of efficiency as the
economic environment and investor population change over time."

[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1977721&#...](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1977721&);

 _Efficient Markets II_ (by Fama who was the one who originally came up with
the efficient market hypothesis describing the three forms of market
efficiency)

"A weaker and economically more sensible version of the efficiency hypothesis
says that prices reflect information to the point where the marginal benefits
of acting on information (the profits to be made) do not exceed the marginal
costs (Jensen (1978))."

<http://efinance.org.cn/cn/fm/Efficient%20markets%20II.pdf>

    
    
      "Also of course most of the missing markets are the 
      contingent and future ones, which suggests the ways in 
      which things might be biased (forward planning, the firm, 
      the business cycle etc) which are quite important 
      inefficiencies."
    

You can often find sets of stock options' prices that you can arbitrage and
make a profit, but only if there were no transaction costs. Information will
only be taken into account of if the cost of the information (after taking
into account all costs including opportunity costs and risk) is less than the
benefit from exploiting that information. So, there are probably lots of
information from the future that are missing in the market, which if properly
exploited will provide a lot of benefit; but the costs of gathering that
information is even greater, possibly requiring the use of a time machine.

 _My thinking: It could very well be that markets are only efficient only as
far as everything else allows it to be; If investors are not able to evaluate
information over more than one business cycle, (due to the cost of doing so,
economical or psychological or otherwise), then so be it. The market will move
its efficiency to match the environment and its participants._

------
stevenrace
Previous discussions [Aug. 2011]:

\- <http://news.ycombinator.com/item?id=2895474>

\- <http://news.ycombinator.com/item?id=2868498>

------
josephlord
This is I think a bogus debunking of a bogus theory.

Try 'Debunking Economics' by Steve Keen for a good breakdown of real problems
with many economic theories including EMH. Mostly by showing the logical
fallacies in assumptions but also the naive maths and divergence from
empirical reality.

[http://www.amazon.com/Debunking-Economics-Expanded-
Dethroned...](http://www.amazon.com/Debunking-Economics-Expanded-Dethroned-
ebook/dp/B006BG8UFY/ref=sr_1_1?s=digital-
text&ie=UTF8&qid=1348943992&sr=1-1&keywords=steve+keen)

His blog is <http://www.debtdeflation.com/blogs/>

------
kenster07
Economics is clear when explaining its foundation: Markets are only as
efficient as its participants are rational and fully informed. The entirety of
modern economics is built upon the presumption that the last two conditions
are 100% true.

~~~
aristidb
No, it's based on them being true enough that other approximations are less
useful models.

~~~
wisty
Unfortunately, most economists don't develop other models, so the other models
are very immature.

------
csense
From the paper:

> Now, one may argue that a weaker form of market efficiency as per Fama
> (1991) and Jensen (1978) is that patterns are exploited until the marginal
> revenue of further discovery equals the marginal cost of further search. A
> counterargument to that would be graduate students and other hobbyists and
> day traders searching for an edge: to them, the marginal cost is zero and at
> times negative because the value of the search itself, regardless of the
> outcome, is a positive learning experience for them...

A hobbyist has to spend more computation time for every additional strategy
they search, which costs real money. The hobbyist's cost is near-zero for
small searches -- those that can be done by one person using the computers
they already own. Once the size of the search increases beyond what they can
manage with their immediately available resources, they have to hire
assistants and/or buy computation time like anybody else.

------
anonymouz
Is this a serious paper? I can't judge at all if it is, because I cannot find
any useful definitions, exact statement of the claimed theorem or proofs.

Then again, I'm not an economist (coming from a mathematics backround), so
maybe this is the expected form of a paper here?

~~~
yummyfajitas
The original papers proving similar results do prove theorems. This is one
example which springs to mind:

[http://dpennock.com/papers/chen-ec-2007-betting-on-
permutati...](http://dpennock.com/papers/chen-ec-2007-betting-on-
permutations.pdf)

Here is a review of the literature from a while back, it cites plenty of
actual math papers:

[http://dpennock.com/papers/pennock-ijcai-workshop-2001-np-
ma...](http://dpennock.com/papers/pennock-ijcai-workshop-2001-np-markets.pdf)

------
Bakkot
Of course this doesn't mean markets aren't within some small constant factor
of being perfectly efficient.

~~~
confluence
Generally with NP-complete problems - that small constant factor is still
often an order of magnitude away from optimal (path
finding/search/salesman/shortest subsequence etc).

~~~
sold
In general, approximation of NP-complete problems is not that simple. Assuming
P /= NP,

Subset-sum can be approximated within 1+epsilon for any requested epsilon>0 [a
PTAS]

MAX-3SAT can be approximated within 8/7 and not better. Vertex cover can be
aproximated within 2 and its not known if that's optimal.

Set cover can be approximated within O(log n) and not better.

Clique and TSP have no sensible approximation.

I would argue that potentially a loss of factor 2 is not an order of magnitude
away from optimal. I don't know what is the situation for markets.

------
Symmetry
Does this actually contradict the _weak_ form of the EMH, that "future prices
cannot be predicted by analyzing prices from the past"? If it's
computationally unfeasible for the market to incorporate new information,
surely its then equally computationally unfeasible for some outside observer
to predict future prices? I'm not sure this even contradicts the semi-strong
form of the EMH, though it clearly does rule out strong EMH.

~~~
confluence
Weak form EMH is so pointless so as to be essentially useless in predictive
value.

It's like those horoscopes that state - "you will meet someone of great
personal importance soon!"

------
brador
Direct link: <http://arxiv.org/pdf/1002.2284v2> [PDF]

------
confluence
A slightly long and/or dodgy video by the professor who wrote this (I'm
setting your expectations): <http://www.youtube.com/watch?v=7iOJZZFDKpc>

I found it rather enlightening - but the paper is still highly readable and I
encourage HNers to peruse it and perhaps check out the video as well.

Previous discussions:

<http://news.ycombinator.com/item?id=2895474>

<http://news.ycombinator.com/item?id=1144548>

More from the professor: <http://philipmaymin.com/cv.pdf?q=phil/cv.pdf>

------
drfloob
... and if P = NP, the market doesn't matter because everyone will have
cracked all your bank passwords.

------
pron
I have only had time to skim through this very quickly, but it seems entirely
based on false definitions.

He begins so:

 _The efficient market hypothesis claims that all information relevant to
future prices is immediately reflected in the current prices of assets. In
other words, you cannot consistently make money using publicly available
information._

Then, he encodes a 3-SAT problem into market orders and claims:

 _So what should the market do? If it is truly efficient, and there exists
some way to execute all of those separate OCO orders in such a way that an
overall profit is guaranteed ... then the market,_ by its assumed efficiency,
_ought to be able to find a way to do so._

So he defines market efficiency to be all-knowing, and then gives at an NP-
hard problem to solve. It seems like he could have given it the halting
problem as well, only it might be harder to encode as market orders. But the
point is that an efficient market is "all knowing" only that which is
"knowable". The market isn't an oracle that can answer any question about the
past, but a true machine that can answer all questions about the past that are
answerable using the machine.

What he's doing seems similar to saying "if God exists then P=NP". After all,
if God is omniscient and omnipotent, He could solve all NP problems in
polynomial time. :)

~~~
uvdiv
I skimmed it quickly too, but I think you've misread. The "interesting" claim
isn't the 3-SAT encoding; it's earlier where he claims that markets
"naturally" have to solve NP-hard problems, in particular that to optimize an
investment strategy you have to solve the Knapsack problem. (?!) It's hand-
waving nonsense.

~~~
pron
It doesn't matter. When people say efficient market, they mean that the market
knows whatever is knowable by the present time using current technologies.
They don't mean that the market knows everything that's theoretically knowable
using unknown/impossible algorithms.

Before there were computers in an efficient market (supposing one exists), the
current prices would reflect everything that is known and can be
computed/deduced without computers by the present time.

Anyway, the paper misunderstands "known" to mean "anything which can
theoretically be known", which obviously includes solutions NP-hard problems.
It's just that nobody claims that's what efficient markets are.

~~~
001sky
RE: _When people say efficient market_

The hypothesis is predicated on perfect rationality.
<http://en.wikipedia.org/wiki/Perfect_rationality>

vs

<http://en.wikipedia.org/wiki/Bounded_rationality>

------
confluence
Honestly - stuff like this reinforces a feeling I have had recently that the
social sciences are nothing more than pseudoscience dressed up most of the
time.

Economists have essentially no idea what is going on and neither do their
financial compatriots and the EMH is by far the clearest example of this.

I'm trying to make it a personal rule that anything outside of the hard
sciences (math/physics/chemistry/statistics etc) is bullshit to me until
further notice.

I mean really - I do advanced math and statistics in moderately complex
systems - AI/robotics - and I can barely wrap my head around them. I shake my
head in disgust at those who apply platonic and unrealistic theories to the
extremely complex system that is the world. Indeed, the surety they display in
their theories amuses me, because to even think that one can reason about the
entire world all at once is, in fact, hilarious.

~~~
throwaway64
This kind of armchair criticism is not particularly enlightening, or
productive. You can still make useful and largely correct predictions without
understanding all of the dynamics in a system.

After all, all science amounts to an approximate model, even your vaunted
"hard sciences".

~~~
confluence
I'm sorry but were you alive during the last couple hundred financial crises?

I mean if your entire field is dedicated to predicting the future of finance
and economics and you don't predict them - it kind of shows that you really
don't know what you are doing.

All I'm doing is pointing out that the social sciences have no clothes on.

~~~
jbrechtel
right....because the field of math has never changed course (see: Godel), nor
physics (see: Newton, Einstein, )...

You're reading too much into the disproportionate effect mistakes in economics
have had on society. Science get things wrong all the time...failure is part
of the learning process.

~~~
confluence
True but at least the hard sciences modify after being falsified.

The social sciences don't have to because they aren't based on the scientific
method - mainly the design and independence of repeatable experiments with
controlled variables.

Hard science equations have no room for bullshit whereas the social ones do -
hence EMH is still taught.

~~~
yummyfajitas
_The social sciences don't have to because they aren't based on the scientific
method - mainly the design and independence of repeatable experiments with
controlled variables._

The same is true of many physical sciences - geophysics, oceanography, climate
science, astronomy, etc. Your criticism applies to basically any scientific
theory which has poorly understood microfoundations (i.e., a lot of them).

The EMH is taught because it's a useful approximation to reality, even if it's
imperfect. Or, as the article puts it: _Whether markets are efficient or not,
and whether P = NP or not, there is no doubt that there will be markets that
can allocate resources very close to efficiently and there will be algorithms
that can solve problems very close to efficiently._

Incidentally, the EMH claims that financial crises are unpredictable. So the
lack of useful predictions of the financial crisis is evidence in favor of the
EMH.

[edit: Note, in response to Dn_Ab that 3SAT, the problem considered by the
paper, is NP complete.]

~~~
Dn_Ab
But he is not, in that quoted statement saying much. For example, if the
problem is NP-Hard but not NP-Complete then we will not even be able to tell
how well we are doing.

Or for markets, aspects of it may invovle solving NP-Hard problems with
efficient approximations that are themselves NP-Hard (you are better placed to
opine on whether such a possibility is likely).

------
realize
The ability of some traders to make consistent significant profits should be
an "existence proof" of the inefficiency of markets. The EMH is just wrong.

~~~
tatsuke95
Now you just need to point to some traders who are able to make "consistent
significant profits".

~~~
rafcavallaro
And not just "some" but a proportion of profitable traders greater than what
we would expect due to chance - i.e. the existence of some lottery winners
does not prove that playing the lottery is in general profitable.

------
michaelochurch
Efficient Market Hypothesis isn't very well defined. It's more like a class of
assertions, some of which are demonstrably true and some of which are false.
Loosely speaking, it says the market price of an asset (or exchange rate
between two assets) will be fair, which means that it corresponds to the
expected value of its basic value. If you're talking about a bond, you can
look at the (known) payment stream, discount the cash flow, and compute a fair
value based on known market conditions. For an equity, there is no fair value
other than "the expected value of its price in the future". Some stocks pay
dividends, but many don't, so their values are based on something other than a
current dividend stream, and largely that "something" is: what is the expected
value of the thing in the future? The question is: what is the timeframe?

In the very short term, EMH is false for computational reasons. Efficient
markets require someone (i.e. an arbitrageur) to keep them efficient. The
profits of arbitrage are the incentive for people to keep markets as efficient
as possible, and they tend to have this effect. This is an extremely
competitive business, and in this day in age, it often comes down to
_microseconds_ , but it can be done and billions of dollars are made every day
by people who are doing it (and, contrary to popular depictions, arbitrage is
actually _good_ for the markets and economy).

In the very-long term, EHM is probably also false. By very-long, I'm talking
about 10+ years. The reason for this is rooted in the scarcity of money: there
are a lot of projects and companies that will produce and capture economic
value in the future, but people don't have enough money to fund them all.
Hence, stocks are "cheaper" than they should be, and risky growth stocks (much
less illiquid private equity) especially so. ("Equity premium.") There are
some people who (demonstrably) "get" value investing and are better at
predicting long-term corporate futures than others. The issue here is the long
feedback cycle. If your frequency of investments is that low, you have no way
of knowing if you're _actually_ good, or just getting lucky, especially in the
context of the non-normal (i.e. fat-tailed) distribution of equity returns
driven by "black swan" events.

EMH is true enough that if you don't have the technical machinery to trade at
microsecond latency, nor the reputation that will allow you to invest for the
very long term despite market caprice-- i.e. even if the market tanks, people
will trust Warren Buffett's judgment-- you probably can't reliably make a
better profit on the stock market than you'd get if you invested in an index
fund.

What does EMH rely upon? Ultimately, it says that if there is _expectancy_ to
be made selling or buying a security at a price other than P, it will be sold
or bought until the price reaches P. This assumes an infinite amount of
capital ("smart money") that people are willing to deploy in order to exploit
pricing inefficiencies or inconsistencies. This is an obviously false
assumption, but for liquid securities of _known_ expected value, it's close
enough. Leverage (borrowing) generates a lot of "additional" smart money, so
that even a 20bp (0.2%) discrepancy can be levered up into a 10% gain. (If you
have $100, borrow $4900, and turn that $5,000 into $5,010, your equity
position has gone from $100 to $110.) Microprofit opportunities will be
exploited so long as there's sufficient leverage to make them worthwhile, but
the willingness of lenders is not infinite.

EMH is usually used to make mathematical analyses work. It's a guideline, but
no one who understands financial markets believes it to be literally and
universally true. No one can actually predict the future or human behavior,
but the (false) assumption that there is no arbitrage produces closed-form
numbers that are often very close to the real values.

Also necessary is the distinction between smart and dumb money. The latter
isn't a pejorative; "dumb money" means that there are incentives other than
informed speculation. For example, when you buy a house because you want to
live somewhere, that's dumb money. Or when an index fund buys stocks because
of its chartered requirement to do so, that's "dumb money", not because the
buyer is an idiot, but because his purchase doesn't convey information about
the stock's real value in the way that smart money would. Markets are
efficient when there's enough smart money to keep the dumb flow from pushing
the price around. This is going to be true of highly liquid stocks, currency
rates, and commodities, but not true of assets like real estate. Financial
engineers tend to discount "dumb" activity as harmless Brownian motion, but
the 2008 subprime mortgage meltdown established that not to be always wise.

~~~
confluence
What happens when the smart money realizes that shorting a massive amount of
dumb money would leave them freight trained by the crowds?

Smart money can be dumb as well - because it pays to do so. I'm quite sure
that the number of rational investors/capital are greatly outnumbered by the
irrational investors/capital such that many correct trades become essentially
insolvent before the market becomes rational again. See value shorters for the
last two booms.

~~~
mason55
_such that many correct trades become essentially insolvent before the market
becomes rational again_

Keynes was quoted as saying "Markets can remain irrational longer than you can
remain solvent."

