

AI That Picks Stocks Better Than the Pros - pier0
http://www.technologyreview.com/blog/guest/25308/

======
rauljara
My understanding of why the stock market is supposed to be good for society is
that it encourages investors to invest in companies that need more capital. It
has the nice side effect of allowing people to buy high value assets that are
likely to increase in value over time, letting people save for their
retirements, while simultaneously providing capital to companies that may need
it. I fail to understand how a computer that holds on to stocks for < 20
minutes forwards either of those goals. Maybe someone can explain why this is
a good thing to me, but this really seems like a perversion of what free
markets are supposed to be good for. And again, maybe someone can explain why
this would be a bad idea, but I'd really like to see a system where if you buy
a stock you are not allowed to sell it for a fixed period of time that would
be measured in months, and not seconds.

~~~
hugh3
Market making. Suppose I decide I want to buy 3000 shares of IBM right now.
I'll have made the trade within minutes, because I'll probably have bought it
from a computer which holds shares for twenty minutes. Four years later, when
I want to sell, I can sell it within minutes too, because I'll be selling it
to another computer that will probably sell 'em off to a series of other
computers, before eventually winding up in the pocket of another long-term
investor.

If long-term investors were the only ones buying and selling shares it would
be much more difficult to buy and sell shares when you wanted to. Net result:
shares are a less attractive investment, IPOs raise less, economy suffers.

~~~
rauljara
Thank you. That was the explanation I was looking for.

My next question then, is whether this flexibility to sell instantaneously is
worth the risk of near instantaneous drops of 700+ points in the dow, as
happened a couple of weeks ago. My gut tells me that letting computers have a
free run of it is not good. But I'm definitely open to being convinced
otherwise. And I'm definitely open to a better solution than you have to hold
on to stocks for a month or more.

~~~
patrickk
The high frequency trading (HFT) systems that seemed to cause the drop you
mention also make it more expensive for ordinary investors to actually buy
stock in the first place, thereby minimizing your potential return (as Buffett
says, "you profit when you buy" or something to that effect).

There was an article on this a few weeks back where a guy in the know was
explaining how such systems are a bad thing. Say you bid a price of $2 per
share for a share that is currently at $1.92. Since the advent of HFTs,
there's a much higher probability that the price you eventually pay will be a
lot closer to $2 than to the current $1.92, (perhaps $1.98 or $1.99) whereas
before HFTs you could get it closer to the current price. This matters
particularly when you buy a lot of shares, and when you hold them for a long
time, as the couple of cents difference compounds a lot over time.

I'm not an expert on the subject, that was my takeaway from it.

~~~
SkyMarshal
Basically what the HFT algorithms do is probe orders for their market limits.

Say you want to buy your 3000 shares, and you don't want to spend more than
$20/share, so you enter your bid with a $20 limit. Your limit is supposed to
be invisible to the market, only you and your dealer know it. But the HFT
algorithms have a way to probe your order and uncover your limit.

The stock is currently trading around $15, so your order hits the exchange,
and in the first few milliseconds, a HFT computer makes a micro-offer for 10
shares at $16, then 10 shares at $17, then 10 shares $20, then 10 shares at
$21.

When the last offer is rejected b/c it's over your limit, the computer then
guesses your limit is $20 and offers to sell you 2960 shares at your limit of
$20.

All this happens in just a few milliseconds, and it just cost you roughly
(2960x20 + $20 + $17 + $16 + $15) - (3000 x $15) = 59,268 - 45,000 = $14,268.
That's 30% more than you would have paid had you been able to execute at $15.

It's basically cheating, and should be made illegal. Just one more example of
how our markets are rigged.

~~~
jrockway
Of course, if your limit was $15, it would have executed at $15. Or not at
all, in which case $15 was not the asking price.

Remember, the price is a two-way street. It's what you are willing to pay, and
it's what the seller is willing to sell for. If you say, "I would like to buy
at $20" and your partner says "$20 is fine with me too", and the trade happens
at $20, then the market is working perfectly. You could have gotten a better
deal, but it's not what you asked for.

~~~
jfager
Let's say you're an institutional investor with an obligation to maintain a
particular investment profile, and there's some event that forces you to
rebalance your portfolio in a fairly dramatic way. You're used to working with
stable portfolios with more or less predictable returns, and this might be a
bit much for you to take on, so you decide to blow some of your management fee
on some really smart consultants who can do the rebalance for you. They, in
turn, have to figure out how to actually execute the plan that they come up
with - there are going to be some pretty big orders involved, and if the
market gets tipped off that this is going to happen, people are going to start
hoarding their shares to drive the price up to screw you. So they start
chopping the order up into much smaller orders and dribbling them out slowly,
or looking for baskets of stocks that act like the stocks you really want, or
trying to fill from dark pools, all kinds of crazy tricks (and they have to do
the same kind of things on the sell side to finance all this).

Meanwhile, some even smarter guys at an investment bank just down the road
know all these tricks, too, and have come up with a way to algorithmically
spot them happening more accurately or more quickly than anyone else. So they
start jumping in front of your trades, not because they want the stock, but
because they know you want it and are probably willing to chase a rising bid
for a while. Then all the other algorithms start to notice, and they start
doing the same thing.

You, the friendly investment manager, are now financing both sides of what
really amounts to one of the coolest deathmatches going, a bunch of math and
computer geeks playing on your quarter and keeping a penny for the honor of
letting you watch them do it. But you actually don't care, because it wasn't
your quarter in the first place, and now that your fund is back in order,
you're not going to get sued, and your customers, who had no idea that any of
this was going on, and who may not even realize they're your customers, are
back to paying 2% fees for (statistically likely) below-market returns.

In the past, these kinds of large trades would usually end up splitting the
difference, with 'real' sellers earning a bit more than what they would have
otherwise settled for and 'real' buyers paying a bit less than their actual
ceiling. But now 'real' buyers end up paying close to their high bid, and
'real' sellers end up earning their low ask, with the bots fighting it out for
the very thinly sliced middle territory.

In one sense, I do agree that this is an example of a market working
perfectly. Algorithmic traders are exploiting an arbitrage opportunity between
mainstream industry practice and cutting edge technical ability, and just like
theory would suggest, they're making bank from it.

~~~
jrockway
_In one sense, I do agree that this is an example of a market working
perfectly. Algorithmic traders are exploiting an arbitrage opportunity between
mainstream industry practice and cutting edge technical ability, and just like
theory would suggest, they're making bank from it._

I think this is all there is to be said. Your trading has a pattern that makes
them money. The algorithmic traders are going to use that information
advantage to make money. That's the idea of the market; trying to extract as
much value from securities as possible.

Even if you impose arbitrary trading limits, the big banks are still going to
find a way to make money off of you.

~~~
jfager
There are lots of ways to make money that would ultimately harm the market
that have been made illegal or have been banned by exchanges. Nobody has a
problem with people getting rich from providing value, but there's a long
history of patching the system when it looks like someone's starting abuse an
advantage that doesn't return any value to the market.

------
theli0nheart
> Using data from five non-consecutive weeks in 2005, a period chosen for its
> lack of unusual stock market activity

Doesn't this completely invalidate the results of this test?

Compare with: I've developed a system for testing whether or not any given
number is prime. Using data from the numbers 5, 7, 11, and 17, ...

~~~
jgoewert
That was my first thought as well.

Actually my first thought was: In the other consecutive weeks, the system lost
its shirt and jumped out the window.... twice.

------
breck
> Using data from five non-consecutive weeks in 2005, a period chosen for its
> lack of unusual stock market activity, here's how AZFinText performed versus
> funds that traded in the same securities (which were all chosen from the S&P
> 500):

This is silly. It's quite easy to generate an algorithm that would perform
similarly. I wonder how it would have performed had it been run across the
entirety of 2005 (or any year for that matter), and not just during cherry
picked weeks.

~~~
hugh3
Yes, I'd like to see it run across the entirety of the available data.

Also, if I came up with a method like this, and I actually had evidence that
it worked, I'd be patenting it and selling it to an investment bank for _very
large_ sums of money before I published the details.

~~~
khafra
Anthropic bias: The other thousand programmers that came up with such an
algorithm used it or sold it to an investment bank; this is the first one that
published instead.

------
obsaysditto
The research paper, which is a pretty good read, can be found here
[http://www.robschumaker.com/publications/IEEE%20Computer%20-...](http://www.robschumaker.com/publications/IEEE%20Computer%20-%20A%20Discrete%20Stock%20Price%20Prediction%20Engine%20Based%20on%20Financial%20News.pdf)
[pdf]

------
kurumo
I work in this field. That is not the first paper written about this system;
the previous was "Sentiment Analysis of Financial News Articles", Schumaker
et. al. This is not the only academic project describing something like this,
see papers by Mittermayer and Lavrenko for other examples. Short summary of
the results is basically this: you would have to be insane to trade purely on
output of a system like this. There are multiple issues with this, the primary
being that most of these systems do not and cannot distinguish documents
leading the trend from those lagging the trend. Furthermore, to actually be
able to trade on the indicators derived from news you need to encode the
market expectation into your model, which is a separate (and difficult)
problem. The margins these systems produce in simulated trading are small, and
for some reason nobody takes transaction costs into account. This is not to
say that systems like this do not exist in the real world - they certainly do,
but the people that build and/or use them generally will not discuss the
details, for obvious reasons.

------
Eliezer
Shouldn't there be some way to flag stories as "headline is obviously false"?

~~~
rms
I believe it has been discussed by pg as part of a proposed revamp to the
flagging system. It'll happen eventually.

------
bh42
Don't index funds and completely random selections ALSO outperform most
professional stock pickers?

~~~
GFischer
Here's an answer:

<http://www.studyfinance.com/jfsd/pdffiles/v8n1/liang.pdf>

Apparently the random stocks outperform the "pros" picks after 6 months (the
pros have the edge in the short term)

Actual quote from the study: "Results suggest that the pros selection
statistically outperforms the random selection only in the one-week period.
Over a six-month holding period, the random stocks perform better than the
pros recommendations."

------
Periodic
I read the title as "AI that picks _socks_ better than the pros". I was very
intrigued. As I waited for the link to load I mused about what a damning
report this must be for the fashion industry. Can a computer really out
fashion the fashionistas? Alas, it was not so. The fashion elite are still
unassailable in their fortress of subjectivity.

~~~
sp332
>I mused about what a damning report this must be

I think you mean _darning_.

------
cameldrv
These types of systems are now common in the market, and you need more
sophisticated algorithms to make any money. They said that they picked 2005
because it had a lack of "unusual market activity." I bet they picked it
because their system didn't work in later years, because there's too much
competition to make money with their system now.

------
mkramlich
here's one in Python:

import random

stocks = (ibm, apple, ...)

stock_to_buy = random.choice(stocks)

------
PassTheAmmo
So if I made a system that traded automatically and I was convinced that it
would beat the market, how would I go about connecting it to the stock
exchange? What costs would there be?

~~~
desponsible
Something like this - <http://www.interactivebrokers.com>

------
jrockway
I'd like to see what this looks like when run on bond derivatives from the
early 2000s. I wonder if it would have been able to foresee the subprime
crisis better than the big investment banks.

(My theory, as someone involved in the industry, is that emotion dictates 90%
of whether or not someone is going to make a trade. This is why nobody noticed
how bad the subprime bonds / CDOs were -- they bought them, so they must be
good. I feel the same way about my telephone and keyboard ;)

------
b0b0b0b
They don't detail the number of trades nor do they account for commissions or
slippage.

~~~
larsberg
And it's all simulated. Last I heard, placing orders affects the market,
especially if you're working with more than a toy account.

~~~
joshu
It totally affects the market.

~~~
mikeklaas
As in, 'it affects the entire market' or 'all of it affects the market'?

~~~
joshu
I was just agreeing vigorously with him.

------
jheriko
"Computer programs can gamble better than gamblers"

Sorry to say, but I'm completely unsurprised by this - statistical analysis is
the most sensible approach to the problem. Gamblers tend to be less sensible
and calling a gambler a stock market trading pro doesn't stop him being a
gambler...

I'd also be wary of labeling pros in a system which will naturally produce
"lucky" and "unlucky" individuals regardless of skill... at least some
proportion of successful traders are just lucky. I can't measure it, but it
seems very heavily implied...

------
MikeCapone
I wonder how well covered their downside is. Will a black swan make it blow up
spectacularly, negating any previous gains?

------
Maven911
I have a question for all of you...how do so many of you know a lot about
financial markets and finance...I made an Ask HN about it:

<http://news.ycombinator.com/item?id=1422453>

------
naner
Not really surprising. This been done before to some degree by Kurzweil and
Schmidhuber and I'm sure others.

~~~
desponsible
Any pointers?

~~~
naner
Be really smart? No way in hell could I do this. You need a strong math,
statistics, and machine learning background to begin with.

------
aresant
Catch 22 - if this system is popularized, it will change the way stocks are
traded, and in the process will damage its own value by destroying the control
index.

------
bmf
nit picking, but many of the words listed (such as "smaller" and "crude")
aren't verbs.

~~~
georgieporgie
From the article:

> (In his work, 'verb' is a technical term, and does not exactly correspond
> with the conventional definition of the word.)

