
I might be a complete failure (after 8 years of work) - chasingsparks
http://pathdependent.com/2010/06/27/i-may-be-a-complete-failure/
======
mbateman
A lot of people are leaving comments critical of his goal of beating the
market as playing a zero-sum game, just pushing wealth around, having no value
for society, etc.

This strikes me as dubious and high-handed, to say the least.

First, as to the value of the endeavor. He is trying to, in his words, "come
up with an algorithm that finds pockets of profitability in a cloud of
probable randomness." If he could really do this, it would create capital
flow, the ability to borrow, and so on, for people that could turn it into
wealth that would not otherwise have access to that wealth. Financial markets
serve a real function, and beating the efficient market hypothesis would
improve their ability to fulfill that function.

Second of all, he's extremely interested in the problem. He's obsessed with
modeling complex systems and sees this as a sort of holy grail. Surely the
fact that he's interested in it and values it counts for a lot, especially if
you grant the above point, that there is a value to what he's doing. He has
his reasons for being interested in it, and it's a productive endeavor.

I think it's false that his lack of progress, or the reasons that he's
frustrated at his lack of progress, have to do with these aspects of his goal.
It's more that he's working on an extremely hard (arguably insoluble) problem
where it's hard to see incremental gains. This is a problem that ambitious
people in many different fields face.

~~~
dmfdmf
I agree. Disparaging speculation as "zero-sum" and therefore inherently bad is
non-sense. In almost all mature industries such as grocery stores, auto sales,
lumber, appliances, etc., any individual seller's gain is some other seller's
loss. So what? They are providing a valuable service and if profitable in the
long-run are doing so at lowest cost, highest value to their customers.

Financial speculation has a long history of being disparaged by various anti-
capitalist demagogues. Speculation is a zero-sum activity _between_
speculators but, from an economy-wide perspective, speculators nevertheless
perform a valuable function in the dissemination of new information and the
formation of prices. This latter function is why it is so hard -- the
identification of an arbitrage opportunity and acting in the markets to make
profits means that price differentials fade away and thus the potential for
profits. A speculator has to continually find new opportunities in order to
make money over the long term.

~~~
jeromec
> Speculation is a zero-sum activity _between_ speculators

I agree.

> any individual seller's gain is some other seller's loss

I disagree. A farmer can sell to a grocer and both can gain.

~~~
chc
I think you don't disagree. The grocer is not in a seller in the same
ecosystem as the farmer. The grocer is one of the external people who benefits
from the zero-sum game the farmers are playing.

~~~
wooster
Farming is pretty much the opposite of a zero sum game.

~~~
chc
The growing part isn't, but the selling part can be seen that way in the right
conditions. I agree, though, it isn't the example I would have chosen.

------
edw519
_After eight years, I have nothing concrete to show for my efforts._

So sad, so true. This is what happens when you play a zero-sum game. Even if
you win, someone has to lose.

Here's an idea. For the next 8 years, why don't you do something that helps
others by making the pie bigger for everyone instead of just trying to game a
bigger piece of a smaller pie for yourself.

When you do for others, you _always_ have something concrete to show for your
efforts. It may not be what you expected, but it will definitely be something
worthwhile, I promise you.

~~~
jeromec
_> This is what happens when you play a zero-sum game. Even if you win,
someone has to lose._

How is it always a zero-sum game to win? Fundamentally, trading stocks is not
that at all. If I buy one of the first issued shares of "Tasty Ice Cream" for
$1 because I want the company to spread their goodness as well as get a return
on my money, no one has lost. If I lose my job the next week and need to sell
my share which may be valued at $1.10 the person that buys has not lost,
especially if he receives a dividend long enough to cover the $1.10.

~~~
lowkey
I think this is the way capital markets were originally supposed to work;
matching companies in need of growth capital with investors seeking to earn a
risk-adjusted return on their long-term savings.

Unfortunately, the financial markets have morphed over time such that now,
traders trade for the sake of trading. Instead of our work being our wealth,
trading securities somehow became an end unto itself. What's worse is that
there really are two markets today: one for the individual retail investor and
a second for institutional investors.

Retail investors are taught to 'invest for the long-term', and 'not try to
time the market' meanwhile hedge-funds and high-frequency traders play a
different game with different rules. Their profits come primarily at the
expense of retail investors and in many cases investment banks 'front-run'
their own clients taking advantage of their customers in a way that would be
illegal in any other industry.

John Stewart explains it better than I ever could in his face to face
indictment of Jim Cramer of CNBC:
[http://ccinsider.comedycentral.com/2009/03/13/jon-stewart-
an...](http://ccinsider.comedycentral.com/2009/03/13/jon-stewart-and-jim-
cramer-the-extended-daily-show-interview/)

(Link for the Canadians: [http://watch.thecomedynetwork.ca/the-daily-show-
with-jon-ste...](http://watch.thecomedynetwork.ca/the-daily-show-with-jon-
stewart/exclusive-interviews/the-daily-show-with-jon-stewart---jim-cramer-
interview-uncut/#clip149637))

~~~
mistermann
>Retail investors are taught to 'invest for the long-term', and 'not try to
time the market' meanwhile hedge-funds and high-frequency traders play a
different game with different rules.

Exactly....that the financial services industry constantly advises the retail
investor to "invest for the long term", when they do the exact opposite (while
generally using the funds provided by these exact same retail investors), to
me is a pretty strong indicator that the markets are something entirely
different than how they are sold to people.

The markets were created to provide funding for companies in need of capital,
and a branch of the financial services industry was legitimately required to
facilitate this process between companies and individual investors, but now
almost everywhere I look people are focused on finding new ways to skim money
out of the system, under the guise of "providing liquidity", as if there was
some problem with liquidity 10, 20, 30 years ago.

------
10ren
Question: has _anyone_ managed to do this? (eg. do hedge funds do this, or do
they combine algorithms with their own ideas and quasi-insider knowledge, plus
massive amounts of timely data - inaccessible to an individual).

My old supervisor (ACM fellow) comforted my failure to create AI by saying
that lots of other clever people hadn't done it either. I have a feeling that
you are attempting something _more difficult_ that AI - ie, algorithmic
inference _without the data that your competitors have_. Because of their
better information, they will find a regularity (eg. 0.01% dip on Wednesdays)
quicker than you - and trade it away.

It would be awesome if you could do it though - forget the philosopher's stone
aspect of gold-for-nothing, this would be a very cool intellectual
achievement.

Idea: focus on small markets/exchanges (unpopular metals, unpopular countries,
even non-financial markets, like eBay) that don't attract the attention of big
players. Easier to find arbitrage there; you can make some actual money, and
_then_ move onto a bigger market with your learnings, earnings, and
confidence. Sort of a niche strategy.

 _Personal Note_ : My AI research depressed me, and I (wisely) decided to
shelve it. It's been incredibly more satisfying to work with others, to
achieve doable goals, that benefit others. Fortunately, AI comes up in
everything; and my current approach (of not doing AI directly) may well be the
most effective way to approach the goal. It might be the same with you - both
markets and inference are universal.

 _EDIT_ So, Tantalus = low-hanging fruit + moving-goalposts
<http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus>

~~~
pradocchia
Simply put, if anyone has managed to do it, you aren't likely to hear about
it. That said, I know of two:

First was a math prof that would play the markets every couple years when he
needed a little extra cash.

Second was someone who had isolated patterns of market intervention and
figured out how to profit from them. But that's more like insider trading than
beating the market algorithmically.

------
drewcrawford
You can keep adding data points to your spline/polynomial interpolation, but
that doesn't mean the model will be any better. In fact, for the purposes of
predictive power, it will probably be worse.

I'm not extremely well educated on the subject, so somebody correct me, but
these predictive models have always struck me as just complicated splines. Add
a thousand data points, and the "predictions" for old datapoints look really
good. The problem isn't with the math, it's with an inappropriate application
of the math based either on fundamental misunderstanding of extrapolation or
on a fundamental failing of human psychology in not realizing that the model
was built on the data, and not the other way around.

~~~
kiba
_Taleb's empiricism implies resisting generalization from data and limiting
the derivation of general rules from particular observations as one can be
missing hidden properties. Thus he believes that scientists, economists,
historians, policy makers, businessmen, and financiers are victims of an
illusion of pattern. They overestimate the value of rational explanations of
past data, and underestimate the prevalence of unexplainable randomness in
those data._ \-- Wikipedia on Nassim Nicholas Taleb

People project patterns onto data when in fact, they don't exists.

~~~
mikecane
"Fooled by Randomness" is a great book. But I sometimes feel about it the same
way I feel about people who say lotteries are a waste of time: some people
_do_ win. And sometimes, despite everything, a transient pattern _does_
repeat. How frustrating!

Edited to add this, which is sometimes also brought up:
<http://en.wikipedia.org/wiki/Illusion_of_control>

~~~
Tichy
Yeah, now Taleb is betting on the big random event, but what if the unexpected
thing is that the unexpected thing never happens?

------
cjy
You've learned a lot, so you're obviously not a failure, but I think your post
illustrates how draining a winner-take-all field like quantitative trading can
be. You can never be sure if your wins or losses are real or just mirages
caused by noise. Humans need the emotional encouragement that incremental
success brings. Nothing get's me excited to work more than a good day of sales
or a thoughtful comment from a customer. It takes a special type of
personality to keep on persisting in the face of repeated failure. Certain
start-ups also have winner-take-all properties. Anything that relies on
network externalities or advertising revenue is more risky. The rewards can be
big, but I prefer markets where the product's value isn't a function of the
customer base and success is more incremental.

------
aspiringsensei
My mother was down on the floor of the NYSE the other month, and she got the
opportunity to corner a trader who she noticed was relying heavily on
algorithms.

She asked "What's the difference between a good trader and a good algorithm?"
The trader said "nothing.If you're buying 1000 shares, it's probably better to
simply put in a market order. A large hedge fund seeking to buy many hundreds
of thousands - where the difference between paying 8.30 and 8.31 a share
matters - algorithms can actually be helpful."

The issue here is the definition of "trader." For this guy, trader is a person
who executes large trades for an investment manager. A trading decision is not
the same as an investment decision, and in that vein many trader's motivations
are simply to buy a desired quantity of a financial instrument as cheaply as
possible.

If you're buying 1000 shares, it's probably better to simply put in a market
order. A large hedge fund seeking to buy many hundreds of thousands - where
the difference between paying 8.30 and 8.31 a share matters - algorithms can
actually be helpful. If you're buying 1000 shares, it's probably better to
simply put in a market order. A large hedge fund seeking to buy many hundreds
of thousands - where the difference between paying 8.30 and 8.31 a share
matters - algorithms can actually be helpful.

Humans are probably better computers for figuring out which stocks to buy and
sell to my mind, but there's certainly a place for algorithms if they are not
too high-minded.

~~~
cowmoo
You are right that for a individual investor focused on the long term, a algo-
order probably wouldn't make a difference in comparison to a market order with
your E-Trade/Schwab account (most likely, your order-flow won't go directly to
the market anyways; it is either crossed internally, or re-routed to a
broker/dealer that's paying retail brokers for the order flow such as
Timberhill).

However, I respectfully disagree that humans are better at making trading
decisions than computers. The world of algo trading can be divided into two
sides, a) high frequency traders, who through fast cancel-and-replace limit
orders and colo with the market centers, try to act as virtual market-makers
(or scalpers, depending on your perspective), b) buy-side institutional fund
managers who want to complete their orders, without HFT predators and negative
market pressure. Large block orders are spliced into small lots (i.e., VWAP)
and sent to the market using intentional limit price over time to hide the
movement of a huge buy or sell order on the market.

Human beings might be better than machines at picking single stocks for long
term investing (although most people are probably still better off investing
in an ETF). But that's not how sell-side traders make money in the first
place. Guys like GS/MS/Timberhill make money by having the unfair advantage of
faster execution speed, more capital and specialized trading algo's against
the small retail investors.

~~~
aspiringsensei
Has there been a good writeup of what HFT "Is" at any point?

What is your impression of them as a pernicious/positive force? Are they
essentially market making?

This is the kind of stuff I really should have an opinion about, but I haven't
found anything to lean on in the public domain.

~~~
cowmoo
Has there been a good writeup of what HFT "Is" at any point?

<http://vimeo.com/6056298>

What is your impression of them as a pernicious/positive force? Are they
essentially market making?

Open for debate. Depends on what you mean as a pernicious/positive force. Good
for retail investors, institutional investors, stability of the market, or the
sell-side? All of these are conflicting sides. It is generally SEC's mission
to protect the small individual investors' fair access to the market, while
trying to walk the fine line of not disrupting the big institutional
investors/sell-side brokers' way of doing business (and their political
lobbying groups).

Pro HFT argument: HFT are virtual market makers that through the use of
technology and arbitraging through multiple ECNs, are decreasing the bid/ask
spread of the traditional market makers and providing more liquidity to the
market. They serve as stabilizing force during irrational exuberances.

Con HFT argument: HFT are bad predators who through technology, jump ahead of
institutional investors' block orders and in term pass on higher priced
liquidity to retail investors that no one needs. They don't serve as
stabilizing force, as they stop trading as soon as they stop making money and
in fact may fan the fire by employing high frequency short selling in a flash
crash.

------
erikpukinskis
If you're doing what you love, you've already won.

I feel a similar way pretty often. I have a couple of visions of software I
want to build that are far out of my technical reach. I've been prototyping
them, on and off, for about eight years too, depending how you count.

They've all failed. I start with a flash of optimism, and code until I realize
I was wrong. I then stop, and rethink things until I think of another guess.

This is exactly how great accomplishments happen. You may be working on an
impossible problem, or you may be working on a 20 year problem. Or maybe
you're lucky and it's a 10 year problem.

But even if your goal is impossible, it's very likely that if you work on it
for 20 years, you _will_ solve a 20 year problem. It might be slightly askew
to what you thought you were solving, but it will happen. It's almost
inevitable.

Regardless, I wholly support what you're doing. Keep going. And read widely...
explore... make sure you're playing in the margins of your field, and
exploring odd projects far outside the scope of your work. It's those places
that you will make that accidental discovery... that in all these years of
trying to "beat the market" you made a fundamental discovery in artificial
intelligence, or network theory, or whatever.

And yeah, if you're doing what you love, you've already won.

------
malloreon
It sounds like despite your ideas of what failure is, by age 25 you've learned
and applied more than most people ever will.

~~~
singular
Agreed. I'm 28, soon to be 29 and I have achieved a great deal less. That kind
of experience, though sour, is pretty awesome, at the very least you should be
able to get yourself into a highly paid quant position at a finance firm.

------
motters
Many smart people have tried to "beat the markets" and failed. From what I've
seen of it, this sort of activity is at best a form of numerology. Also, in
trying to move money around slightly more efficiently than the next market
analyst you're not really generating wealth as such, and your ideas and
activities will leave no lasting value for society.

You may not be a failure, since the skills acquired in analysing trends can be
utilised in other areas. The process of having promising looking preliminary
results, only to find that they are illusory is familiar to many scientists.

Ultimately it comes down to what you really want to do with your life. Is your
life only about moving money, or something else?

~~~
Kaizyn
If I'm not mistaken, I believe the idea is to set up some automated trading
bots that just simply make money for you on a continual basis. This constant
revenue stream allows you to forget about making money and allowing you to use
those resources to either fund a life of ease and luxury or other pursuits
such as "making the world a better place".

------
jey
Seems like it'd be a better bet to apply all your skills to creating value
(wealth) instead of just trying to push it around a smidge more efficiently.

In other words: find a real problem and solve it, increasing global wealth in
the process, and collect the economic reward.

~~~
mwerty
The median startup does not offer a better outcome.

~~~
jey
If you're only going to get the median outcome, don't do a startup. If you
have a real chance at making it, do a startup.

Yes, people mis-estimate their ability all the time, but that's part of life.
If you want a safe good outcome, become a doctor or something.

------
danbmil99
I suspect you could get a pretty good job as programmer for some hedge fund.
Not only would it leverage your command of this domain, you might learn a few
things that will explain why it is difficult to impossible for you to win in a
vacuum.

There are social engineering aspects to this game that you have not been
exposed to yet. Also, you may just not be sophisticated enough, or you have
pursued a dead end approach. This is a very rich field of study.

If it's your passion, I say stick with it, but expand your circle of knowledge
and influence.

And screw all this 'zero sum' crap, capitalism is the least zero-sum game
humanity has ever come up with. And speculation, market-making, betting on
winners & losers is one of the most important driving forces of capitalism.

------
chegra
"He[Dan Zanger] holds the unofficial record in trading stocks by turning
$11,000 into more than $18 million in 18 months in 1999-2000. He grew that to
an incredible $42 million in less than two years and has the tax receipts to
prove it." <http://www.investopedia.com/articles/trading/04/082504.asp>

"Williams won the 1987 World Cup Championship of Futures Trading from the
Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in
a 12 month competition with real money. Ten Years later his daughter Michelle
won the same contest" <http://en.wikipedia.org/wiki/Larry_R._Williams>

Somebody has some sort of algorithm.

------
arthurdent
_I find myself excited over the preliminary results of my increasingly
sophisticated simulations_

This. The vast majority of the money being made in the market is being made by
people with what you may consider a laughable lack of "sophistication".

Start simple. Complex modeling of stock market behavior only makes sense if
you think the stock market has a complex underlying structure.

~~~
chasingsparks
By sophisticated, I meant refined and cultivated, not devoid of simplicity. It
was a poor choice of words. At some point in my development, I started to
eschew more complicated forms of AI and machine learning, in favor of simpler
agent-based models with "laughably" simple assumptions. I think it's the
proper direction, then again, the title I chose for the post was not
arbitrary. ;)

------
Loy
The term "complete failure" is more revealing about your draconian idea of
success. You should take a look at your own words : "they have never found
their Holy Grail" [therefore, they joined the ranks of Academics as a
testimony of failure].

If you're running after the Holy Grail, you have no choice but to be a hero.
What kind of pressure is that ? In fact, most of the answers are in front of
you, in the way you express yourself.

For instance, the myths of the Holy Grail and Tantalus are ones of insoluble
challenge and dilemma. Therefore, when you are wondering if the Holy Grail
exist, where it is, and if you'll ever reach it... You are probably asking the
wrong questions.

Instead, you should ask yourself : "Who am I ?". Do you want to be rich, or do
you want to "beat the markets" ? That's 2 different things, one is a mere
consequence of the other. Where is your priority ? As it's not clear from your
writing, I suppose it's not clear in your mind either.

~~~
chasingsparks
The title was unnecessarily provocative, a reflection of my mood at the time
of writing it. (It's Sunday; I'm hung-over; and I was looking at the results
of a four day simulation run that were ambiguous.)

My priority is in "beating the markets" because it's a fascinating challenge.
I also have interests that lie beyond satisfying my intellectual curiosity but
require wealth. I find the idea of using my own wealth to solve problems (e.g.
avenues of cancer research) far more appealing than fund-raising or requiring
public funds.

------
noonespecial
Look on the bright side. Many people dedicated their lives to the search for a
process to turn lead to gold. This seemingly fruitless search lead to all
kinds of neat discoveries. You've learned lots of valuable things that you
wouldn't have learned otherwise as well. Keep searching if you must but start
benefiting from that knowledge.

The real secret about turning lead to gold is that it _is_ possible, but by
the time you have mastered the skill to do so, you no longer need to or want
to.

~~~
gvb
Alchemy and transmutation of lead to gold is a pretty apt analogy for John
Nelson's quest to turn his model of the market into gold.

a) _The alchemical belief in transmutation was based on a thoroughly wrong
understanding of the underlying processes._ John's (and other's) lack of
success indicates a faulty model.

b) _Ironically, it transpired that, under true nuclear transmutation, it is
far easier to turn gold into lead than the reverse reaction, which was the one
the alchemists had ardently pursued. Nuclear experiments have successfully
transmuted lead into gold, but the expense far exceeds any gain._

<http://en.wikipedia.org/wiki/Nuclear_transmutation>

~~~
chasingsparks
If you look at the first sentence of the article, it is linked to the
subheading of a blog post that explained my intellectual evolution -- the
subtitle of which was "The Contemporary Philosopher’s Stone" which is how I
refer to this problem. (<http://en.wikipedia.org/wiki/Philosophers_stone>)
You're not the only one drawing the comparison ;)

~~~
khafra
Sounds like in that frame, most of the consoling messages here are along the
lines of "our gold is not the common gold."

------
daylast
I hate to be the bearer of bad news but, based on the comments you've made in
your post, you are most likely a smart fool (as you have so aptly put). In
essence, you don't even know what you do not know. I've worked at top-tier
Wall St institutions and have had first hand experience working with several
of the largest quant hedge funds out there. Here's my advice: if you truly
enjoy markets and are sure you want to take this path, you will be much better
served by spending several years working at an established quant hedge fund.
This will open your eyes to the theory, processes, data, and techniques
required to earn alpha consistently and in a risk efficient manner. It's a
long road and that road is full of extraordinarily hard working and
intelligent people that one has to constantly compete against. If you can
handle this though, you will learn a bunch and at some point in the future you
can strike out on your own again. At that point you will at least have a
chance of success. Good luck!

------
narrator
I tried to beat the market using all kinds of AI stuff off and on for a number
of years. Learned a lot about AI techniques, and became a better programmer,
which helped in my day job, but never got anywhere. It's a fool's game.

Besides, most of these HFT systems hold positions for only a few minutes or
less. There's no way someone without a huge account, to cover commissions from
trading so much, and access to a lot of expensive real-time data streams could
possibly make any headway at it.

Not to say there aren't people who do make money at trading, but they are
swing traders and/or people who've figured something out about the market that
the market doesn't know yet, and thus has not revealed at all in the price
movements of the stock. Since price movements are the primary input into a lot
of trading algorithms, most trading algorithms are blind to these
developments.

------
MLnick
Only very few people have commented that, you are only 25! (I guess given the
nature of the HN community :)

I don't think anyone can consider themselves a failure at 25, having spent 8
years learning a lot of programming, algorithms, data mining / machine
learning and complex systems modelling.

It would be easier to think that if you hated what you were doing. But it sure
doesn't sound like you do. So you just need to decide what you really want to
be doing.

If it's algo trading, I concur with another poster that says go work for a
hedge fund (or bank, or prop trading shop). It is super competitive, but they
have the technology infrastructure and most importantly the capital. Getting
in is not easy, but simply show them all your work (it doesn't work anyway,
but is indicative of your skills and way of thinking). You will learn a lot,
you may hate the people and environment, or you may love it even more. And
then yes after a few years of experience you will most certainly be in a
better position to go off on your own again (or do something totally
different, by then you will really know if you like it or not!). Many top
hedge fund managers / traders only started their own thing at 30, 35, 40, even
50... I know a dentist who became a prop trader. Anything is possible.

A PhD would be a great option IF it's for the right reasons. But if you want
to do a startup (sounds like you might quite like the idea and you posted here
on HN, so...): \- you already live on ramen, so no lifestyle change there; \-
bootstrapping something can't cost more than losing money with trading
algorithms; \- you already have many of the requirements: coding / technical
skills, low-cost living circumstance, love to solve tough problems and a huge
amount of tenacity in the face of failure and overwhelming odds; \- bonus:
your interest in social systems modelling etc ties in pretty nicely with
what's big right now and in the near future.

So if that is what you really want to do, go for it either alone (or find a
co-founder), or find a small/medium startup to work for. To make the
transition a bit more natural perhaps focus on ones that are data-driven and
have machine learning / modelling at the core of their business. Think
recommendations, systems modelling (www.flightcaster.com) and weather
(www.weatherbill.com). There are many many examples of YC and other startup
companies of this nature (many focused on the social network space).

Good luck in whatever you do decide to do with the next 60+ years of your
life. On your deathbed you can post about whether you think you are a complete
failure or not.

------
noahc
John,

95% of people would agree with you. But they (and you) are wrong. What is your
end game? Do you want to be worshiped on HN or do you want a six figure job
some where?

If you want either of those, you could just as well write up all you know spit
it out in blog posts and free eBooks. Someone is going to recognize your
skills and take it from there. At this point the worst thing you can do is
keep all your failures to your self, because then no one can benefit from
them.

------
retube
His approach to this - the goal of beating the market - has been totally
wrong. It sounds as if he's been at this completely solo - he mentions in the
comments that he worked alone as he didn't want to share his algorithms.

He would have benefitted enormously from working in an institution that
specialises in this kind of work (hedge funds etc). Being largely self-taught
and having worked alone since he was 17 pretty much guarantees he doesn't know
anything institutional trading firms won't know or have studied - employing as
they do advanced mathematicians, physicists etc to work at the cutting edge of
algo research and trading.

Not only has he missed out academically, he's missed out operationally.
"Playing-the-game" is as big a part - if not bigger part - of trading than
pricing/valuation. With the resources of a large institution you can learn to
play the market in a way you can't possibly by yourself. This is in fact
largely how they make money - consistently, year on year.

Thirdly - even if he did discover a pricing inconsistency or whatever, it's
doubtful he'd be able to leverage it in as profitable a way that he would be
able to as a firm.

------
dunk010
Well, he might just not be smart enough. Large financial institutions and many
smaller hedge funds reap countless millions from the markets and much of this
comes from proprietary trading (well, all of it in the case of hedge funds.)
There's a massive brain-drain into financial institutions of many of our best
and brightest, all in search of making a fortune by the time they're 30. So on
the one hand you have top students from top institutions applying their
considerable intellect and training into applying advanced mathematical
methods against an extremely complex problem. On the other you have a guy who
tinkers around himself hoping after hope that the amateur (for that is by
definition what he is) attempts that he makes at beating the market are going
to some day work out and make him his fortune.

Unfortunately the odds are stacked against you, and those are compounded by
fact that you have little or no opportunity to employ any serious leverage.

"I would have probably, or at least possibly, been wealthy by other means by
now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant
statement to make.

------
muhfuhkuh
Look to none other than Long Term Capital Management[1] back in the 90s to see
how dangerous a game it is to either algorithmically predict or automate a
thing as emotionally wrought and, at times, institutionally manipulated as the
market.

[1] <http://en.wikipedia.org/wiki/Long-Term_Capital_Management>

~~~
jrockway
What? Their time-tested bond arbitrage algorithms worked great. They beat the
market month after month.

Only when they tried speculating on equities did they fail. And even then,
their positions only took short-term losses that they would have recovered
from had they not been leveraged 100-to-1. Their algorithms were correct, but
they didn't have enough capital to see their positions through to the end.

The LTCM story is about greed and leverage, not algorithmic trading. Plenty of
money can be made trying to beat the market: it pays the salaries of the
hundreds of thousands of investment bank employees around the world. The
problem is, you personally can't do the same things they do, because you don't
have enough capital.

Raise a billion dollars and your days of commuting to work are over.

------
stretchwithme
The most important thing in your life is you and the brain you are carrying
around in your head. If your experiences have made you smarter, you DO have
something concrete to show for the years you've put in.

Entire countries have had everything destroyed, but have come back to be very
successful. And this happened because of what they carry around in their
heads.

------
krosaen
I find it inspiring that he's stuck with it, and wonder how's he's made a
living in the meantime

~~~
chasingsparks
I take short-term jobs and contract work for a few months, then quit
everything for full-time research. Usually the jobs I take happen to require
learning something tangential but possibly useful. It's a happy coincidence
that I am perfectly happy with the absurd level of frugality required. (I live
in a place that most people call the "House of Squalor. It's really not dirty,
but it is semi-dilapidated.)

------
soyelmango
It sounds like a trolling headline - he himself acknowledges the value of what
he has learnt.

Some suggested in his comments that he should join a firm so that knowledge
and experience could be shared towards finding the algorithm[s]. His reply to
Henrik, "I’ve been wary about taking jobs at firms like that because of IP
concerns. I always believe I am on the cusp of something great, and wouldn’t
want to share my algorithms", is a factor in his progress, or lack of.

By working alone, he's more likely to earn 100% of nothing, when he could
collaborate and earn a small % of a very large number.

That said, if he finds the algorith alone, then big congratulations and
respect are in order for sticking to his convictions.

------
brianobush
negative findings (and publishing those results) are just as useful as
positive findings.

~~~
pbhjpbhj
If that's true, I think it needs a lot of thought, then we at least know that
negative findings are valued far less. I found that my TOE was wrong is not
likely to win me a Nobel unless there are associated "positive" findings ...

~~~
brianobush
I was obviously considering only negative findings with actual thought,
research plans and analysis.

------
Tichy
I've met people who claimed to make money with their algorithms, which really
surprised me. Your post gave me an idea, though: What if instead of running
your algorithm on the whole set of stocks, you create several random subsets.
One of them has a good chance of being successful, at which point you can
claim that your algorithm works and sell it for $$$. (I am not entirely
serious, of course - but maybe some of the successful algorithms work that
way?).

------
pvdm
<http://en.wikipedia.org/wiki/Monroe_Trout>

------
wangwei
Even people with very high level math knowledge fails to predict the market
(remember the two noble prize scientists who failed?).Since you're out of
school when you were 17, I doubt you'll able develop complex schemes (unless
you are a very very rare genius).

~~~
chasingsparks
I was not out of school at 17. What caused you to infer that? (I'd like to
correct it if it sounds that way.)

I am actually starting a masters/Ph.D. program in the fall in computational
social sciences. Like I mentioned in the post, I am interested in social
systems in general, not only markets. (However, I do find markets to be the
most interesting.)

------
rjett
"Every few months, I find myself excited over the preliminary results of my
increasingly sophisticated simulations, only to be disappointed a few short
weeks later to find that I was simply wrong."

Isn't this why algorithmic traders "refit" their algorithms on a daily basis?

------
bbers
To beat the market you HAVE to predict what trades others will make in the
future. It is easy to use math to make it appear to be a numbers game but that
is just abstraction. The reality is that people buy and sell stocks for 1000s
of different reasons.

~~~
chasingsparks
I use much less math than your average quant. I prefer simulations with _a
lot_ of agents. Think Game of Life, not partial differential equations.

------
WalterBright
Any algorithmic method of beating the market can only be successful if it is a
secret and if its trades are of insignificant volume. Otherwise, knowledge of
this algorithm will be factored into the prices, defeating it.

~~~
arthurdent
this is really not entirely correct.

when things started melting down in 2008, it was largely attributed to the
fact that hedge funds basically all had the same strategies in their book.
When one started liquidating, many started losing value and liquidating in a
disastrous feedback loop.

There are plenty of strategies that are very robust, and in hedge-fund land,
its not uncommon that many players are playing very similar hands.

In fact, there even exist strategies that get better as more people run them.

------
ctingom
Remember, no man is a failure who has friends. -- Clarence in It’s a Wonderful
Life

------
rortian
An interesting post. I'd be interested hear what kind of background you've
got.

I think your remark on finance professors is quite wrong. It takes quite a lot
of dedication to do a Phd. I'm not great respecter of academic prestige, but
its a bit insulting to imply that people fail their way into a professorship.

I'd be more than happy to share some pointers on financial papers I've read
and such. Just talk about your background and what you know. If you think 'a
random walk down Wall Street' is correct, it isn't. Have you seen the
derivation of Black-Scholes? Do you know what most high frequency trading is
based on?

I'd also be interested to see what type of stuff you like to read.

~~~
chasingsparks
I didn't mean it in an insulting way. I'm actually starting a Masters/Ph.D.
program in computational social sciences in the fall, hoping to fully immerse
myself in an environment that also finds markets fascinating.

I have derived Black-Scholes; I study market micro-structure; I read as much
as I can on anything related to social systems from volatility surfaces in
options pricing to more generic machine learning methods such as PSO.

(I'd like to respond more, but I just noticed I am 20 minutes late for my
fathers birthday party.)

~~~
rortian
Cool, definitely get back to me. Kinda random but I am very interested in
complex networks as well as some finance stuff.

------
naner
This type of thing has been done before:

<http://en.wikipedia.org/wiki/FatKat_(investment_software)>

<http://www.idsia.ch/~juergen/finance.html>

<http://www.technologyreview.com/blog/guest/25308/>

I'm sure there are other examples. I think part of the problem is this guy
thinks he can get rich quickly with just the right set of programming
instructions.

~~~
ruang
Nice quote from one of the links' links, from NY Times:

But as these new techniques proliferate, some worry that promotion is
outpacing reality. These techniques may be better for marketing than stock
picking.

