This strikes me as dubious and high-handed, to say the least.
First, as to the value of the endeavor. He is trying to, in his words, "come up with an algorithm that finds pockets of profitability in a cloud of probable randomness." If he could really do this, it would create capital flow, the ability to borrow, and so on, for people that could turn it into wealth that would not otherwise have access to that wealth. Financial markets serve a real function, and beating the efficient market hypothesis would improve their ability to fulfill that function.
Second of all, he's extremely interested in the problem. He's obsessed with modeling complex systems and sees this as a sort of holy grail. Surely the fact that he's interested in it and values it counts for a lot, especially if you grant the above point, that there is a value to what he's doing. He has his reasons for being interested in it, and it's a productive endeavor.
I think it's false that his lack of progress, or the reasons that he's frustrated at his lack of progress, have to do with these aspects of his goal. It's more that he's working on an extremely hard (arguably insoluble) problem where it's hard to see incremental gains. This is a problem that ambitious people in many different fields face.
Financial speculation has a long history of being disparaged by various anti-capitalist demagogues. Speculation is a zero-sum activity between speculators but, from an economy-wide perspective, speculators nevertheless perform a valuable function in the dissemination of new information and the formation of prices. This latter function is why it is so hard -- the identification of an arbitrage opportunity and acting in the markets to make profits means that price differentials fade away and thus the potential for profits. A speculator has to continually find new opportunities in order to make money over the long term.
> any individual seller's gain is some other seller's loss
I disagree. A farmer can sell to a grocer and both can gain.
How do you figure? Generally, the way one seller beats out another is by providing some combination of increased value or decreased cost.
There are certainly plenty of talented people working in finance. That talent is just completely wasted.
Now, this isn't to say that paper money shouldn't exist, but with its benefits come some serious risks and responsibilities.
So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose.
Here's an idea. For the next 8 years, why don't you do something that helps others by making the pie bigger for everyone instead of just trying to game a bigger piece of a smaller pie for yourself.
When you do for others, you always have something concrete to show for your efforts. It may not be what you expected, but it will definitely be something worthwhile, I promise you.
* Accuracy in pricing shares (and it is important to know the actual value of a company for many reasons)
* Enable market participation. In many illiquid OTC markets, it is all but impossible for anyone outside large institutional players to participate. More trading = lower transaction costs, more transparency, and ability for the little guy (me) to join the action
* Liquidity. And yes, this is very important. It is a common answer because it is a good one
* The same security can be worth different amounts to different people. By definition this precludes it from being zero sum (this statement is of course subject to what school of financial theory you subscribe to)
* Performance measurement. Stock options have gotten a lot of bad press, but compensating a CEO according to lets say, revenue, is much less unbiased and can lead to worse outcomes (e.g., grows business in unattractive segments or over-invests to boost top line)
The above are in no way MECE and overlap in many ways. Markets can definitely be better, but calling it a zero-sum game or completely value-less to society is a bit too far.
Define "accuracy"? Preferably in such a way that helps me reconcile it with events like the flash crash.
One of my biggest problems in understanding the financial markets is how you separate the signal (events reflecting actual change in company value) from the noise (speculation, flash crashes, etc.).
While I realize that there is real information (e.g. earnings reports) that goes into the pricing, it seems like there's also a hell of a lot of worthless noise (i.e. I see the flash crash as a massive noise spike).
It seems to me that separating the two is how people make money long-term.
If smaller differences between current price and opinions can lead to action--because of lower transaction costs--the market is better.
That's why I quit playing poker, even though I was a winning player after a lot of study. Zero sum games are no good. Creating things makes more winners and the capability for an unlimited amount of winning.
 Well, that and the culture with all the cigarette smoke, alcohol, late neon lights screwing up your sleep schedule, and general degeneracy. But that stuff didn't bother me before so much before I put it in greater perspective.
When you play football too, if you win someone has to lose. And yet there are positive externalities, to wit millions of people watching the World Cup.
In trading, the externalities might not be so obvious but they do exist.
Please see my other comment below and (especially) the linked article.
Another positive externality of trading is lowering the cost of funding for enterprises. Companies go through a lot of trouble to make an IPO; they do so because getting listed increases their sources of funding.
A longer answer would probably require a blog post, so that we could discuss whether I'm "cheating" -- after all don't the points above boil down to providing liquidity? Am I really "cheating" or maybe your condition is unreasonable? Et cetera. If I write such a post I'll post it to HN, though I doubt I could do better than Larry Harris.
There is an economic progression from the switch from barter to money, to lending of money, to spread of risk by enabling minority ownership of multiple ventures (early corporations), to easier transferal of ownership in smaller pieces (allowing more of society to participate), to large-scale, low-friction, highly-liquid markets used by everyone, directly or indirectly. Each step further reduces economic friction, which lowers the threshold for economic activity to take place, causing more of it to occur, increasing the prosperity of the whole society.
Even if you lose money personally in the "zero-sum game" with unfortunate investments, you're still probably losing only a portion of the money you made working at a company that wouldn't exist if not for large, liquid, low-friction markets.
How is it always a zero-sum game to win? Fundamentally, trading stocks is not that at all. If I buy one of the first issued shares of "Tasty Ice Cream" for $1 because I want the company to spread their goodness as well as get a return on my money, no one has lost. If I lose my job the next week and need to sell my share which may be valued at $1.10 the person that buys has not lost, especially if he receives a dividend long enough to cover the $1.10.
Unfortunately, the financial markets have morphed over time such that now, traders trade for the sake of trading. Instead of our work being our wealth, trading securities somehow became an end unto itself. What's worse is that there really are two markets today: one for the individual retail investor and a second for institutional investors.
Retail investors are taught to 'invest for the long-term', and 'not try to time the market' meanwhile hedge-funds and high-frequency traders play a different game with different rules. Their profits come primarily at the expense of retail investors and in many cases investment banks 'front-run' their own clients taking advantage of their customers in a way that would be illegal in any other industry.
John Stewart explains it better than I ever could in his face to face indictment of Jim Cramer of CNBC:
(Link for the Canadians:
Exactly....that the financial services industry constantly advises the retail investor to "invest for the long term", when they do the exact opposite (while generally using the funds provided by these exact same retail investors), to me is a pretty strong indicator that the markets are something entirely different than how they are sold to people.
The markets were created to provide funding for companies in need of capital, and a branch of the financial services industry was legitimately required to facilitate this process between companies and individual investors, but now almost everywhere I look people are focused on finding new ways to skim money out of the system, under the guise of "providing liquidity", as if there was some problem with liquidity 10, 20, 30 years ago.
For algorithmic traders, it is indeed a zero-sum game. For the rest of us "investors", as I like to call it, to distinguish it from trading, there is potential for long-term good.
Maybe 1 year later Bob will sell GOOG to Alice at $200, and the stock will close at $201. Again Alice won $1 and Bob lost $1.
Of course Bob made a lot of money in the process, so the zero-sum game definition might sound arbitrary. However, as far as trading is concerned, the definition as zero-sum game is useful. If Bob were a better trader (more skilled or luckier) perhaps he could have saved $2.
See The Winners and Losers of the Zero-Sum Game by Larry Harris.
I don't see the causality. He has yet to succeed because it's an incredibly difficult problem domain. There are problem domains of similar difficulty that benefit people that he could have chosen and still be in this position.
For example, ESR derives a development methodology for open source software using Austrian economics.
Then again he/she is creating software and algorithms.
Incidentally, a recent study claims that 60% of UK workers feel that their work is of no use to society. Financial Services, Telemarketing, Spammers, High-Frequency Traders and many other 'professions' would fit this description.
Here is the link: http://www.reddit.com/r/Economics/comments/cjgx7/60_of_uk_wo...
My old supervisor (ACM fellow) comforted my failure to create AI by saying that lots of other clever people hadn't done it either. I have a feeling that you are attempting something more difficult that AI - ie, algorithmic inference without the data that your competitors have. Because of their better information, they will find a regularity (eg. 0.01% dip on Wednesdays) quicker than you - and trade it away.
It would be awesome if you could do it though - forget the philosopher's stone aspect of gold-for-nothing, this would be a very cool intellectual achievement.
Idea: focus on small markets/exchanges (unpopular metals, unpopular countries, even non-financial markets, like eBay) that don't attract the attention of big players. Easier to find arbitrage there; you can make some actual money, and then move onto a bigger market with your learnings, earnings, and confidence. Sort of a niche strategy.
Personal Note: My AI research depressed me, and I (wisely) decided to shelve it. It's been incredibly more satisfying to work with others, to achieve doable goals, that benefit others. Fortunately, AI comes up in everything; and my current approach (of not doing AI directly) may well be the most effective way to approach the goal. It might be the same with you - both markets and inference are universal.
EDIT So, Tantalus = low-hanging fruit + moving-goalposts http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus
First was a math prof that would play the markets every couple years when he needed a little extra cash.
Second was someone who had isolated patterns of market intervention and figured out how to profit from them. But that's more like insider trading than beating the market algorithmically.
I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints look really good. The problem isn't with the math, it's with an inappropriate application of the math based either on fundamental misunderstanding of extrapolation or on a fundamental failing of human psychology in not realizing that the model was built on the data, and not the other way around.
Regularization is some means of complexity control. You switch form trying to find the model that has minimal error on your data to finding the model that minimizes "error(model, data) + C*complexity(model)". There are many ways to measure the complexity, and usually the value of C is pretty unimportante (it is empirically found that it only affects the solutions when you change it in orders of magnitude). Keeping a test set is always testing your predictions on held-out data that you didn't use to train the model and that is hopefully independent from your training data.
Still, if you come up with a huge number of models and test them on the same test data you're statistically bound to find a few that seem to work very well, until you try it in different data.
My feeling with most of these market-beating schemes is that just the price history of stocks shouldn't be that predictive of future prices (but aggregating real-world information should). After all, usually what creates spikes and valleys in the prices are some relevant news or decisions.
To wit, P(model|data)=P(data|model)P(model) / P(data), which is to say that you calculate the probability a model is true given the data you see based on the probability you see the data you see given a certain model, multiplied by the probability of the model you are using.
I guess the problem in all this is we want* the model to be very complex, but we can't tell that complexity from overfitting (and it may be that no model exists so the only thing you can do is overfit.)
Being bayesian gives you an extra capability that is model averaging, and this does usually improve the behavior at a high computational cost.
I really like bayesian models, and right now I'm experimenting with one that should do unsupervised sentiment analysis without a priori knowledge of word polarity or things like that (yes, I'm a phd student in machine learning).
It also goes the other way, and some formulas are first proposed in a more bayesian-ish context and then extended to some simpler-looking empirical formulas (for example, the jumps from hidden Markov models to max-ent Markov models to conditional random fields to max-margin Markov networks to structured SVMs).
There are more approaches to machine learning, and in John Langford's blog there is a nice table showing the merits and flaws of many of them: http://hunch.net/?p=224 . But you must keep in mind that you can find many equivalencies between these approaches (boosting for example can be seen as a loss minimization with regularization, and max-ent can be seen as a special case of a bayesian model, etc).
People project patterns onto data when in fact, they don't exists.
Edited to add this, which is sometimes also brought up:
So, the patterns we see in the Ulam spiral can be completely random?
We observe patterns. Human brains are really good in finding them. The problem is finding patterns in past data may or may not provide useful insights into future data.
With the resources being dedicated to this problem, I find it highly unlikely I could be able to offer any helpful advice that would prove more helpful than what already exists.
It would be hard enough to predict the market if the only factors influencing it were signals coming out of the companies we trade. Predicting what other algos may me doing in the space we are trying to examine and the influence their trades exert in its future is... Well... I won't try.
She asked "What's the difference between a good trader and a good algorithm?" The trader said "nothing.If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful."
The issue here is the definition of "trader." For this guy, trader is a person who executes large trades for an investment manager. A trading decision is not the same as an investment decision, and in that vein many trader's motivations are simply to buy a desired quantity of a financial instrument as cheaply as possible.
If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful.
If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful.
Humans are probably better computers for figuring out which stocks to buy and sell to my mind, but there's certainly a place for algorithms if they are not too high-minded.
However, I respectfully disagree that humans are better at making trading decisions than computers. The world of algo trading can be divided into two sides, a) high frequency traders, who through fast cancel-and-replace limit orders and colo with the market centers, try to act as virtual market-makers (or scalpers, depending on your perspective), b) buy-side institutional fund managers who want to complete their orders, without HFT predators and negative market pressure. Large block orders are spliced into small lots (i.e., VWAP) and sent to the market using intentional limit price over time to hide the movement of a huge buy or sell order on the market.
Human beings might be better than machines at picking single stocks for long term investing (although most people are probably still better off investing in an ETF). But that's not how sell-side traders make money in the first place. Guys like GS/MS/Timberhill make money by having the unfair advantage of faster execution speed, more capital and specialized trading algo's against the small retail investors.
What is your impression of them as a pernicious/positive force? Are they essentially market making?
This is the kind of stuff I really should have an opinion about, but I haven't found anything to lean on in the public domain.
Open for debate. Depends on what you mean as a pernicious/positive force. Good for retail investors, institutional investors, stability of the market, or the sell-side? All of these are conflicting sides. It is generally SEC's mission to protect the small individual investors' fair access to the market, while trying to walk the fine line of not disrupting the big institutional investors/sell-side brokers' way of doing business (and their political lobbying groups).
Pro HFT argument: HFT are virtual market makers that through the use of technology and arbitraging through multiple ECNs, are decreasing the bid/ask spread of the traditional market makers and providing more liquidity to the market. They serve as stabilizing force during irrational exuberances.
Con HFT argument: HFT are bad predators who through technology, jump ahead of institutional investors' block orders and in term pass on higher priced liquidity to retail investors that no one needs. They don't serve as stabilizing force, as they stop trading as soon as they stop making money and in fact may fan the fire by employing high frequency short selling in a flash crash.
I feel a similar way pretty often. I have a couple of visions of software I want to build that are far out of my technical reach. I've been prototyping them, on and off, for about eight years too, depending how you count.
They've all failed. I start with a flash of optimism, and code until I realize I was wrong. I then stop, and rethink things until I think of another guess.
This is exactly how great accomplishments happen. You may be working on an impossible problem, or you may be working on a 20 year problem. Or maybe you're lucky and it's a 10 year problem.
But even if your goal is impossible, it's very likely that if you work on it for 20 years, you will solve a 20 year problem. It might be slightly askew to what you thought you were solving, but it will happen. It's almost inevitable.
Regardless, I wholly support what you're doing. Keep going. And read widely... explore... make sure you're playing in the margins of your field, and exploring odd projects far outside the scope of your work. It's those places that you will make that accidental discovery... that in all these years of trying to "beat the market" you made a fundamental discovery in artificial intelligence, or network theory, or whatever.
And yeah, if you're doing what you love, you've already won.
You may not be a failure, since the skills acquired in analysing trends can be utilised in other areas. The process of having promising looking preliminary results, only to find that they are illusory is familiar to many scientists.
Ultimately it comes down to what you really want to do with your life. Is your life only about moving money, or something else?
In other words: find a real problem and solve it, increasing global wealth in the process, and collect the economic reward.
One should not confuse the agency issues, borderline-legal information advantages, or even outright fraud in the markets with the basic activity that sustains them. Think about the alternative: Would you like to have to sit with a dimwit credit officer (= bureaucrat) from a big bank who can then effectively veto your plans? Or would you rather have a lively market that decides on such things? Do you want to do your IPO in a public equity market or would you prefer to try to break into an insider club of old rich people who may never come to understand your funny newfangled business model ("what's internet?!").
German sociologist Georg Simmel wrote an interesting book about money, one of the ideas I liked is that its fungibility dissolves the need for relationships, making us free to choose them, and thus effectively increasing personal freedom. The same, I'd venture, is true for capital markets. By spreading transactions around, they increase your options and freedom as entrepreneur. And for markets to work, you need people actually _in_ the market, like it or not.
Figuring out value (as opposed to the going price), identifiying and exploiting inefficiencies, trying to make more intelligent forecasts of likely developments, all these _are_ "real problems" in a market economy. Having intelligent, hard-working people working on them makes the system as a whole more intelligent.
It's just a crapshoot unless you have some real information asymmetry you can exploit to make money. This could be in the form of noticing some sort of systematic inefficiency, having a unique and effective strategy of gaining market information to identify mispricings, etc. Otherwise you'd be better off finding some unmet need to creatively address and avoiding the cognitive trap that leads from "I want money" to "I should go into finance".
Yes, people mis-estimate their ability all the time, but that's part of life. If you want a safe good outcome, become a doctor or something.
There are social engineering aspects to this game that you have not been exposed to yet. Also, you may just not be sophisticated enough, or you have pursued a dead end approach. This is a very rich field of study.
If it's your passion, I say stick with it, but expand your circle of knowledge and influence.
And screw all this 'zero sum' crap, capitalism is the least zero-sum game humanity has ever come up with. And speculation, market-making, betting on winners & losers is one of the most important driving forces of capitalism.
"Williams won the 1987 World Cup Championship of Futures Trading from the Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in a 12 month competition with real money. Ten Years later his daughter Michelle won the same contest"
Somebody has some sort of algorithm.
This. The vast majority of the money being made in the market is being made by people with what you may consider a laughable lack of "sophistication".
Start simple. Complex modeling of stock market behavior only makes sense if you think the stock market has a complex underlying structure.
If you're running after the Holy Grail, you have no choice but to be a hero. What kind of pressure is that ? In fact, most of the answers are in front of you, in the way you express yourself.
For instance, the myths of the Holy Grail and Tantalus are ones of insoluble challenge and dilemma. Therefore, when you are wondering if the Holy Grail exist, where it is, and if you'll ever reach it... You are probably asking the wrong questions.
Instead, you should ask yourself : "Who am I ?". Do you want to be rich, or do you want to "beat the markets" ? That's 2 different things, one is a mere consequence of the other. Where is your priority ? As it's not clear from your writing, I suppose it's not clear in your mind either.
My priority is in "beating the markets" because it's a fascinating challenge. I also have interests that lie beyond satisfying my intellectual curiosity but require wealth. I find the idea of using my own wealth to solve problems (e.g. avenues of cancer research) far more appealing than fund-raising or requiring public funds.
The real secret about turning lead to gold is that it is possible, but by the time you have mastered the skill to do so, you no longer need to or want to.
a) The alchemical belief in transmutation was based on a thoroughly wrong understanding of the underlying processes. John's (and other's) lack of success indicates a faulty model.
b) Ironically, it transpired that, under true nuclear transmutation, it is far easier to turn gold into lead than the reverse reaction, which was the one the alchemists had ardently pursued. Nuclear experiments have successfully transmuted lead into gold, but the expense far exceeds any gain.
Besides, most of these HFT systems hold positions for only a few minutes or less. There's no way someone without a huge account, to cover commissions from trading so much, and access to a lot of expensive real-time data streams could possibly make any headway at it.
Not to say there aren't people who do make money at trading, but they are swing traders and/or people who've figured something out about the market that the market doesn't know yet, and thus has not revealed at all in the price movements of the stock. Since price movements are the primary input into a lot of trading algorithms, most trading algorithms are blind to these developments.
I don't think anyone can consider themselves a failure at 25, having spent 8 years learning a lot of programming, algorithms, data mining / machine learning and complex systems modelling.
It would be easier to think that if you hated what you were doing. But it sure doesn't sound like you do. So you just need to decide what you really want to be doing.
If it's algo trading, I concur with another poster that says go work for a hedge fund (or bank, or prop trading shop). It is super competitive, but they have the technology infrastructure and most importantly the capital. Getting in is not easy, but simply show them all your work (it doesn't work anyway, but is indicative of your skills and way of thinking). You will learn a lot, you may hate the people and environment, or you may love it even more. And then yes after a few years of experience you will most certainly be in a better position to go off on your own again (or do something totally different, by then you will really know if you like it or not!). Many top hedge fund managers / traders only started their own thing at 30, 35, 40, even 50... I know a dentist who became a prop trader. Anything is possible.
A PhD would be a great option IF it's for the right reasons. But if you want to do a startup (sounds like you might quite like the idea and you posted here on HN, so...):
- you already live on ramen, so no lifestyle change there;
- bootstrapping something can't cost more than losing money with trading algorithms;
- you already have many of the requirements: coding / technical skills, low-cost living circumstance, love to solve tough problems and a huge amount of tenacity in the face of failure and overwhelming odds;
- bonus: your interest in social systems modelling etc ties in pretty nicely with what's big right now and in the near future.
So if that is what you really want to do, go for it either alone (or find a co-founder), or find a small/medium startup to work for. To make the transition a bit more natural perhaps focus on ones that are data-driven and have machine learning / modelling at the core of their business. Think recommendations, systems modelling (www.flightcaster.com) and weather (www.weatherbill.com). There are many many examples of YC and other startup companies of this nature (many focused on the social network space).
Good luck in whatever you do decide to do with the next 60+ years of your life. On your deathbed you can post about whether you think you are a complete failure or not.
95% of people would agree with you. But they (and you) are wrong. What is your end game? Do you want to be worshiped on HN or do you want a six figure job some where?
If you want either of those, you could just as well write up all you know spit it out in blog posts and free eBooks. Someone is going to recognize your skills and take it from there. At this point the worst thing you can do is keep all your failures to your self, because then no one can benefit from them.
He would have benefitted enormously from working in an institution that specialises in this kind of work (hedge funds etc). Being largely self-taught and having worked alone since he was 17 pretty much guarantees he doesn't know anything institutional trading firms won't know or have studied - employing as they do advanced mathematicians, physicists etc to work at the cutting edge of algo research and trading.
Not only has he missed out academically, he's missed out operationally. "Playing-the-game" is as big a part - if not bigger part - of trading than pricing/valuation. With the resources of a large institution you can learn to play the market in a way you can't possibly by yourself. This is in fact largely how they make money - consistently, year on year.
Thirdly - even if he did discover a pricing inconsistency or whatever, it's doubtful he'd be able to leverage it in as profitable a way that he would be able to as a firm.
Unfortunately the odds are stacked against you, and those are compounded by fact that you have little or no opportunity to employ any serious leverage.
"I would have probably, or at least possibly, been wealthy by other means by now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant statement to make.
Only when they tried speculating on equities did they fail. And even then, their positions only took short-term losses that they would have recovered from had they not been leveraged 100-to-1. Their algorithms were correct, but they didn't have enough capital to see their positions through to the end.
The LTCM story is about greed and leverage, not algorithmic trading. Plenty of money can be made trying to beat the market: it pays the salaries of the hundreds of thousands of investment bank employees around the world. The problem is, you personally can't do the same things they do, because you don't have enough capital.
Raise a billion dollars and your days of commuting to work are over.
Entire countries have had everything destroyed, but have come back to be very successful. And this happened because of what they carry around in their heads.
Some suggested in his comments that he should join a firm so that knowledge and experience could be shared towards finding the algorithm[s]. His reply to Henrik, "I’ve been wary about taking jobs at firms like that because of IP concerns. I always believe I am on the cusp of something great, and wouldn’t want to share my algorithms", is a factor in his progress, or lack of.
By working alone, he's more likely to earn 100% of nothing, when he could collaborate and earn a small % of a very large number.
That said, if he finds the algorith alone, then big congratulations and respect are in order for sticking to his convictions.
I am actually starting a masters/Ph.D. program in the fall in computational social sciences. Like I mentioned in the post, I am interested in social systems in general, not only markets. (However, I do find markets to be the most interesting.)
Isn't this why algorithmic traders "refit" their algorithms on a daily basis?
when things started melting down in 2008, it was largely attributed to the fact that hedge funds basically all had the same strategies in their book. When one started liquidating, many started losing value and liquidating in a disastrous feedback loop.
There are plenty of strategies that are very robust, and in hedge-fund land, its not uncommon that many players are playing very similar hands.
In fact, there even exist strategies that get better as more people run them.
I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship.
I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your background and what you know. If you think 'a random walk down Wall Street' is correct, it isn't. Have you seen the derivation of Black-Scholes? Do you know what most high frequency trading is based on?
I'd also be interested to see what type of stuff you like to read.
I have derived Black-Scholes; I study market micro-structure; I read as much as I can on anything related to social systems from volatility surfaces in options pricing to more generic machine learning methods such as PSO.
(I'd like to respond more, but I just noticed I am 20 minutes late for my fathers birthday party.)
On another note, betting markets are (or were) apparently far less efficient than financial markets, in that the true likelihood of a team/horse/camel winning in a competition may be different to the money which punters are willing to bet on it. I know of a guy who made his fortune after many gruelling years by finally coming up with prediction models which worked in these kind of gambling scenarios.
I'd love to read it, but the easiest way to answer your second question is to point you to 'A non random walk down wall street.'
One huge assumption that Black-Scholes makes is that returns are normal distributed. This is not done for empirical reasons rather mathematical ones, namely it is so convenient.
Edit: Pdf http://www.newyorkfed.org/research/conference/2007/liquidity...
Some meta data and such if you don't feel like going with a pdf:
I'm sure there are other examples. I think part of the problem is this guy thinks he can get rich quickly with just the right set of programming instructions.
But as these new techniques proliferate, some worry that promotion is outpacing reality. These techniques may be better for marketing than stock picking.