Hacker News new | past | comments | ask | show | jobs | submit login
The Day I Lost a Shit-ton of Money, Part I (ptotrading.blogspot.com)
279 points by jonnyy on Nov 12, 2014 | hide | past | web | favorite | 163 comments

Guys, you are missing the mark on technical analysis. It's not about forecasting where the prices are going to be, it's about forcing yourself to follow a set of rules instead of following your emotions.

There was an article on HN sometime ago about the practices of some old tribe to choose where to sow the crops for the next year. The practices were totally random, like watching the clouds, where a bird would fly, etc. Well, researchers eventually realized that those practices actually ensured a truly random selection and that was the best strategy. A non random selection, that is any possible bias, could have exposed the tribe to the possibility of a negative bias in selecting terrains and the risk of multiple years of bad crops which would have led to extinction. A random strategy would led to a bad year here and there and offer better chances of survival. But humans cannot make decisions truly randomly so they need a rationale system that helps them make random decisions. We are not that different from computers in this respect, but I am going astray now, so back to topic.

The markets are full of people trading based on their emotions. A stock is going up, greed and fear of missing out kick in and people buy high. A stock is going down, panic and fear of losing kick in and people sell low. People are psychologically wired to make bad decisions in the stock market, they have a negative bias. If you can find a rationale system to follow you will make better decisions than the crowd following their emotions and take their money.

Traders know very well that the first rule in the market is that everything can happen. They also know well that for any chart there are TA "rules" that say buy and other that say sells. They also know that a method will beat the guy with no method.

This at least is my theory, I have never practiced TA but I saw my father throw away tons of money with it and invariably the losses were caused by a few trades where he did not follow his rules but convinced himself to bend them a little. And he preached all the time that following the rules was the only way to win.

You are your worse enemy in the markets.

Trading with emotions is definitely bad (and all too human) --- but it's best not to trade at all. Technical trading in particular is highly irrational if you know what your up against.

Except that all TA 'rules' are based essentially based on numerology. The MACD camp worships 12, 26 and 9. RSI does 14, 70 and 30. They all also worship past values that wre present in the market in the past, which apparently have some special significance in the future. For example, they may be connected by a shooting star or a hanging man or three white soldiers. Either that, or they are constellations used in astrology. Same difference.

> There was an article on HN sometime ago about the practices of some old tribe to choose where to sow the crops for the next year.

Really interested in this...do you have a link to/name of/keywords in said article?

You do realise that the human mind is incredibly good at "recognising/identifying" "patterns" in random data.

I lost a shit ton of money at the firm where I work right now almost a year ago. I wrote software that began issuing trades outside of where it was expected to and did not stop. In a panic, I forced the server it was running on to terminate its process but trades were still open in the market. They had to be manually closed. I feel sick to my stomach even writing this right now, that hour I was trying to fix the problems sticks out as the worst moment in my life. I am glad whoever wrote this can be so nonchalant about it, if you have a strategy for forgetting I would love to hear it.

Yeah, I think all of us HFT programmers have war stories like this. I got a friend that had his program take out the entire market...

Usually it's best if the programmer is separate from the ops guy, but in this case it sounds like you got stuck doing both. That sucks. I've done a bit of trading myself and it's both boring and terrifying. The two worst emotions.

Your friend wasn't at Knight, was he? I'd love to have an insider's tale as to what went on that day.


To me there seems to be a certain sort of karma in this.

HFT doesn't really add value to anything in my opinion (cue the arguments that HFT somehow adds real value to our society). Yet HFT creams a profit by shuffling money around very quickly. So if there are sometimes big losses like this from a bug, then it seems to even things out somewhat.

To me there seems to be a certain sort of karma in this.

Momentum prop trading doesn't really add value to anything in my opinion (cue the arguments that momentum prop trading somehow adds real value to our society). Yet some momentum prop trading creams a profit by shuffling money around very quickly. So if there are sometimes big losses like this from a bug, then it seems to even things out somewhat.

"Anything of value" is a subjective measurement - you could say that stock markets don't add anything of value, or you could say that they enable more investments, since you're able to liquidate your investment whenever you want.

Trend following, and mean-reversion following (what this article describes) are techniques that counteract the basic human irrational/emotional biases. Ideally, they should prevent bubbles and stop crashes. They make markets more rational.

"They make markets more rational."

Said during the greatest economic downturn since the great depression.

They don't help with long-term behavior, but they definitely can make markets more rational on a shorter time scale. You can think of HFT market-makers as adding friction to a system. They will generally take positions against the trend, reducing (or maybe just delaying) price impact of the trend traders.

EDIT: I should point out: _all_ market-makers provide this benefit to the market. The difference now is that HFT is automated, and like most forms of automation it has out-competed most manual market makers, for better or for worse. AFAIK, the last bastion of manual market-making is NYSE, where the humans have information and discretionary powers that are not granted to any of the robots.

Disagree. Momentum prop trading means that there's a ton of liquidity available to trade against when you actually do have new information.


The profit to the society is from regular guys like you and me not partaking in the scam anymore. Not spending money on it. Distrusting the game. Long term it makes sense: people will start investing long term based on fudamentals vs wanting to get rich quick on hocus locus TA. Short term trading is losers game. You will always loose against HFT.

So instead of doing that, read the companies fundamentals. Learn about their debt levels. Earnings. Read their books. And only then invest with 1-2 year time horizon minimum.

HFT is valuable to the society as it makes the society understand that treating Markets like casino gambling isn't working.

Cue the argument: there are many flavors of HTF trading. Some funds exploit the latency of different trading platforms - I agree they don't add anything to the society. Others offer liquidity - if the bid-offer spread is $1, they will insert bids above the other bids and offers below the rest of offers, making trading less expensive for investors. Yet others specialize in execution - your "dumb" pension fund manager wants to sell $100M of a particular stock, he'll route the order through a HTF broker who knows the "best strategy" to execute the order without affecting the market price too much, so that the fund gets a better price.

"Some funds exploit the latency of different trading platforms - I agree they don't add anything to the society."

Even those are pretty easy to make an argument for. Very specialized firms spend vast sums of money, to make very thin margins, to provide me with a very valuable service. That is, I don't have to venue shop based on pricing oddities, they will arbitrage those away. I can shop purely on fees and features, the things that actually will impact me in the long term.

Not really - that's ordinary arbitrage, not HFT arbitrage. No "actual" investor will profit from the fact that prices converge in 0.1s instead of 1.1s. They just wast hume sums of money building fiber optics on the direct line from Chicago to NY.

> that's ordinary arbitrage, not HFT arbitrage.

As it turns out, the "HFT" version of any trading strategy is _just like_ the "ordinary" version, but faster. You could say that they "disrupted" the older arbitrageurs due to their collective technology R&D.

There are 1000's of trades that occur in that second and not only HFT. Someone will benefit if they just happen to cross the spread in that .1s. If you are unloading 10,0000 shares the speed arbitrage could save you real money.

Well, for starters it sounds like you didn't act maliciously. You acted based on the sum total of your training and experience to that point. If your training or experience was lacking then this is an issue for which the company needs to take some responsibility.

FWIW: Whoever allowed you do to that unsupervised is just as guilty as you. Separation of duties and all that.

Were you able to keep your job after that? (If you're willing to talk about it; if not, I understand.)

From his phrasing ("firm where I work right now") it sounds like he was :)

If the firm did not go under as a result, it doesn't make sense to fire the individual. An event like that is a valuable (and expensive!) training lesson, and a mistake that probably won't be made twice.

Meh. You're still working there. If you forget, you'll get dumber.

Reading this sort of stuff from manual prop traders makes me laugh. It sounds so amateur hour. How in the world can manual traders ever compete against a short-term stat arb or HFT strategy? It just sounds like pure luck that any of them will make money.

Also, is 130k really a huge loss? I run HFT strategies, and while it would definitely be a big loss even for one of my strategies, it wouldn't be a phenomenal outlier. How small was this guy trading before?

Hey Consz,

I'm glad you laughed, as I tried my best to make it humorous. You're absolutely right, it is total amateur hour. Sometimes I can't believe this is how I make money! Price-action based trading can be so intellectually vapid. And that is why I feel so blessed to have this job and to be able to make it work. I don't try to pretend to be anything more than what I am. I don't make 7-figures but I make consistent, steady, low drawdown thousands that add up to a 1%-er income. I'm only 25 so I like to think there's room for improvement too. If you're doing better than that, congratulations.

130k is just a number in the middle of the drawdown that I arbitrarily referenced to give some context. You'll have to read the rest to see what the final number was when the full loss was realized.

Well, I'm getting "stop submitting too quickly" msgs and I'm sure there will be no shortage of people telling me this will never work, it's all random, no edge when backtesting, etc. So I'm done debating for the night, best of luck to you guys, hope you read part II!

I'm in the "technical analysis is nonsense" camp but enjoyed your story. You write an entertaining line and I look forward to reading the rest of it.

I'm not sure that's completely true.

1) Are there places / markets that ST Arb and HFT won't go because it's too illiquid or too idiosyncratically volatile?

2) Are there edges so small that you won't chase?

3) Are there one-off opportunities (2010 flash crash, UST flash crash recently, etc.)?

4) Is there a set of strategies that manual traders know (which they should probably transfer to HFT strategies) that the HFT/ST arb shops don't know?

I think the answer to all of those questions is "yes". They probably take on a lot more risk per trade. They would also be better served by automating their strategies. But I think it's not as cut and dry as you state.

Each individual trader might be "amateur" -- but the firm as a whole might end up with a decent return / risk.

#4 is the only one which I would say does not hold. I'd be very surprised if a firm of manual traders could consistently outperform the market (or beat a Sharpe of 1).

I think 4) holds.

We have 2 spectrums for all known strategies (the amount of gold in the veins is another question):

* Strats that are automatable vs non-automatable.

* Strats that are unique (known to a few) vs well-known strats


Well-Known - This is where HFT becomes an engineering problem. It's also the easiest business decision to make. I hire some engineers + buy some computers and I can mint some money for a time. This is also a race to the bottom -- where there will be some easy money to be made at equilibrium, but it'll be unsustainable for HFT firms at the margin. I would say manual trading has 0 impact here -- and they'll likely get crushed.

Unique - These are strategies that only a small set of firms know. Eventually these strategies become well-known as people move around companies. There must be some secret strategies, though. Otherwise, why would Rentech & Citadel sue their own employees (unless we are to believe Simons & Griffin are simply vindictive). I think manual trading could work here if there is a strategy that simply haven't been discovered by the HFT & quant. It's might be hard, but not impossible. These secrets aren't anything you'll find online or in some trading forum, though. Or perhaps they are, but no self-respecting quant would go there (lest they become laughing stocks in their fund).


Well-Known - PE is an example of one. In terms of liquid assets, maybe penny stocks are another area. If a group of "manual" traders couldn't get permission to delve there... it's probably even harder for a quant fund. It's also the territory of insider info and rumors.

Unique - Maybe being there for the liquidity crashes I stated earlier is a good example. But there are certainly non-automatable unique strats around.

Depending on how much gold you believe is in the veins, all strategies are heading towards automatable & well-known. However, in the interim, I can see manual traders still making money (even with a sharpe > 1). In fact, I would say there are well-known strats you can execute manually (in say, your PA, that return sharpe ~0.8ish).

They aren't competing against it though, they are riding the same wave. Competition would assume that HFT's or stat arbs are market makers, doing pump and dumps, which is not the case based on HFT reserve holdings - the antithesis of arb.

Is it less efficient? Yes. It is also orders of magnitude easier to get into. You need PhD mathematicians, computer scientists/engineers to build real hardcore HFT systems. The manual stuff takes what, a high speed internet connection as its infrastructure.

Stat arb is usually not a market maker. The simplest stat arb trade is a pairs trade, so let me use that as an example.

Suppose you do want to market make on a pair A,B. You notice they are out of whack so you open an ALO order to sell A and buy B. The market moves and both stocks go up in sync. You've now opened a short position on A with no offsetting position in B. Oops.

If you want to save part of the spread, you can often do add liquidity, then the minute you get a fill you can cancel and take liquidity on the other side.

I've tried this but I haven't made it work, though my strategies are pretty amateur (I only trade as a hobby).

It sounds so amateur hour.

Amateur hour? The traders could be monkeys, since it's only the salesman who matter.

When you bet other people's money, the strategy is fairly simple: Go long (or short). Somebody at another company goes short (or long). One wins, one loses (it's guaranteed) and the winner gets a percentage of the "win". This is why they love volatility so much, as it determines the size of the profits, even in a zero-sum game.

Sure, mathematicians can beat the house. Doesn't matter to these guys. Other people's money? Flip a coin.

Prop shops don't have salesmen (at least, none that I've ever seen).

Prop stands for "proprietary," it means that they trade their own money.

Correction: it means they trade private money. This could be their own money, but more often than not, it's the owner's/partner's money.

The exact definition of proprietary trading is trading with the firm's own money. Public/private is an entirely different dimension.

There are no sales reps with prop trading, it isn't an agency.


I also think it's an important lesson to learn... I'd rather gain that young then older.

statistical arbitrage should come with a general surgeon warning. "may cause hair loss, not actually risk free"

Now you are just bullshitting. A 130k is a huge loss and a phenomental outlier. You've never had even a 100k loss on your strategies. And by current standards your strategies are not even HFT. I've seen you talking milliseconds. Nowadays people are talking sub-microseconds.

Sub-microsecond transactions seem impossible unless you're physically jacked in to the trading datacenter's network. If you try to transfer a message (like a string buffer) as quickly as possible from program A running on core 0 to program B running on core 5 on your server-grade computer, the best benchmarks I could achieve were "99% of measurements executed in fewer than 150 nanoseconds." And I worked hard on this problem. It seems like maybe you could execute a trade in ~400 nanoseconds at best, if you have a server that's connected with 10GigE directly to a NASDAQ (or whoever) box and you're bypassing how Linux normally does packet handling, but I wouldn't be surprised if the overhead of the other party's server pushes that to at least 1 microsecond in all cases.

But... now that I've examined the facts, I admit what you say may be true. Are HFTs really operating in sub-microsecond time for trades nowadays?

The goal is to be faster than the other guy - HFT is a race, thanks to the subpenny rule. So if network latency is 2ms, you still care about making your trades happen in 2,050us vs 2,100us.

Tl;dr even if trades happen in ms, you still care about us.

Yes, you are correct. On an average Linux box to have a single cache line data transfer between the cores under 150ns 99% of the time is about the best that you can get. Especially if you are running stock kernel that eats this 1% and creates huge outliers ;). There is some talk though, about crazily-expensive switches with integrated FPGAs...

These switches aren't really a secret.

And the crazy expensive switch is guaranteed to be the cheap part. Now add the all the ip required to make that fpga smart enough to place orders, and you're talking huge bills in dev hours and third party licensing.

There are HFT trading systems implemented in FPGAs. They're written in Verilog. Here's one system:


Want to work in that field? Know VHDL/Verilog, Linux, networking down to the wire level?


Light moves at 1 ft per nanosecond... I sure hope you're close to the NASDAQ data center :)

It's 1ft/ns in a vacuum, but my understanding is that it's closer to 0.5ft/ns in a wire, so what you say is doubly true. But I think HFTs are running on servers close to the NASDAQ datacenter, and that people pay buckets of money for such privileges.

Pretty sure NASDAQ rents servers directly on site for serious dough, so distance is not that much of an issue.

Yes and No. Nearly every electronic exchange now offers co-location services (including NASDAQ).

The "serious dough" part is a little harder to quantify. I've not looked into NASDAQ specifically but server colocation is usually on the order of a couple of thousand dollars a month. This is a drop in the bucket compared to the real costs of a professional trading outfit (namely employees and margin/risk costs).

Anecdotally, it's also almost exactly what I paid for a tier 1 co-located server at my first job in a startup during the first dotcom boom.

Wire? It's infiniband and other bufferless network protocols over fiber.

Fiber is still only about 66% of c.

I worked at a prop shop where my teams worst 2-day losses totaled more than $2MM. The worst period for any team on the firm was a $30MM loss with an additional $60MM of risk over the course of 5 days.

No one was happy about it, but it was part of the job and considered reasonable volatility for a portfolio of our (middle) size. While I wasn't a fan of Consz's tone, his numbers check out from my personal experience at a trading firm. Not HFT, but in the end it's all about return on capital and the P&L's should be similar.

One of the big problems about conversations around "HFT" is that there is no definition of what that actually is. I like to put computerized trading systems on a scale of 3 different things: 1) trade volume 2) latency sensitivity and 3) human interaction.

So for instance, I've worked on systems that were "fairly" latency sensitive (~tens of micro seconds in latency budget), not very high volume (100s of trades a day) and had virtually no human interaction. I've also worked on systems that were "not very" latency sensitive (co-lo'd but never actually measured tick to trade times, which were assumedly in the tens of milliseconds), high volume (10s of thousands of trades a day) and had a team of clerks looking over it.

I would call both of those systems HFT systems, but some people wouldn't consider either of them HFT.

In both of those cases a 130k loss would be an outlier, but not a phenomenal one. I've worked with people who have lost millions of dollars in seconds and it blew up the group. I've met others that million dollar swings on a given strategy was par for the course (and there hedge funds doing manual trades that won't notice that trade in a graph).

That is, the world of computerized trading is pretty varied. If you are going to say someone is or is not HFT, let them know what your definition is first, as that is the only way the discussion can move forward.

People are downvoting you, but as someone who only worked on a FIX application in a tiny finance shop, microsecond was barely acceptable. The comment rings false with me as well.

What do you mean that 'microsecond as barely acceptable'? That was too long for what?

I'm skeptical that there are really that many players out there who are in a position where a single microsecond would really make a difference. There are definitely companies out there in this situation, but not many.


Duh. Common. Describe me just one automated strategy for which $130k would not be a phenomenal outlier. And you were talking multiple strategies...

I was talking about 130k in any one strategy. I'm sorry you misunderstood me.

A single strategy is easily capable of losing that much. Any bond futures or cash bond strategy post-FOMC. Any equity index futures strategy during Twitter "flash crash". Any strategy taking large size in a big future during early-mid October this year. Any strategy taking large size in a big future during August 2011. Any Nikkei futures strategy right after the recent QE announcement from Japan.

All of those are easily capable of dropping 130k in a day.

I can't help but think of confidence and the illusion of control:


"Mutual funds are run by highly experienced and hard-working professionals who buy and sell stocks to achieve the best possible results for their clients. Nevertheless, the evidence from more than 50 years of research is conclusive: for a large majority of fund managers, the selection of stocks is more like rolling dice than like playing poker. At least two out of every three mutual funds underperform the overall market in any given year.

More important, the year-to-year correlation among the outcomes of mutual funds is very small, barely different from zero. The funds that were successful in any given year were mostly lucky; they had a good roll of the dice. There is general agreement among researchers that this is true for nearly all stock pickers, whether they know it or not — and most do not. The subjective experience of traders is that they are making sensible, educated guesses in a situation of great uncertainty. In highly efficient markets, however, educated guesses are not more accurate than blind guesses. "

To add to that, a friend 'got into being a trader' as a friend of his simply needed a large number of bodies to manage trades.

Of a year the majority would fail, but would would do great - he would then tout all the 'good' traders out as being new super star traders, and use them to get new business. As long as he had a large enough pool he'd be sure to have a few that became multi-year profitable, and now command large fees.

Basically they are able to make it seem like they know what they are doing, when really they are just hiding failure.

This reminds me of Derren Brown's awesome "The System" special:


More info for anyone who hasn't had the pleasure of knowing who Derren Brown is:


Derren Brown is a British illusionist, mentalist, trickster, hypnotist, painter, writer, and sceptic. He is known for his appearances in television specials, stage productions, and British television series such as Trick of the Mind and Trick or Treat. Since the first broadcast of his show Derren Brown: Mind Control in 2000, Brown has become increasingly well known for his mind-reading act. He has written books for magicians as well as the general public.

Though his performances of mind-reading and other feats of mentalism may appear to be the result of psychic or paranormal practices, he claims no such abilities and frequently denounces those who do. Brown states at the beginning of his Trick of the Mind programmes that he achieves his results using a combination of "magic, suggestion, psychology, misdirection and showmanship".

TL;DW for The System (courtesy of Wikipedia):

The System, a Channel 4 special in which Brown shared his "100 percent guaranteed" method for winning on the horses, was first shown on 1 February 2008.

The show was based around the idea that a system could be developed to "guarantee a winner" of horse races. Cameras followed a member of the public, Khadisha, as Brown anonymously sent her correct predictions of five races in a row, before encouraging her to place as much money as she could on the sixth race. To demonstrate the system to the viewer, Brown tossed a coin showing ten heads in a row to prove it was not impossible, just highly improbable.

After Brown had placed a bet of £4,000 of Khadisha's money on a horse in the final race, he explained that "The System" did not really exist. He had started by contacting 7,776 people and split them into six groups, giving each group a different horse. As each race had taken place 5⁄6 of the people had lost and were dropped from the system. Brown had a different person backing each horse in each race, and one individual, Khadisha, won five times in a row. This was similar to the coin flipping earlier: rather than having a predictive technique, Brown had tossed a coin repeatedly until ten heads had come up in a row, taking over nine hours to produce the required film. Brown expressed the opinion that the principle behind "The System" (essentially confirmation bias or survivorship bias) is what is behind belief in spiritualism or homoeopathic and alternative medicine.

After the selected horse in the final race lost, and Khadisha was convinced that she had lost all her borrowed money, Brown told Khadisha to look again at the betting slip in her hand. The ticket showed the winning horse's name, meaning Khadisha kept her stake and received winnings of £13,000. Brown claimed that he had decided to bet on a different horse when he got to the booth.

At the end of the show, a title card explained that "at each stage of the process, participants who did not make it to the next round were offered a complete refund of any bets they had placed."

Didn't someone do something very similar in the early 1900s with newspaper adverts? I have a definite vague memory of a binary "keep the winners" scam.

To me this is depressing as my future pension is invested in funds. Given the rest of the comments: how best to invest it for long-term grown above the rate it'd have in a savings account?

1. Minimize fees a. This means staying far away from advisors. Either do it yourself somewhere like Vanguard or let something like betterment.com do it for you. 2. Minimize taxes a. Max out your 401K, make sure you're using your IRA to full advantage 3. Diversify a. Own a domestic and international index and that's it. Don't focus on market sectors. 4. Set it and forget it — namely don't let your emotions make you do stupid things 5. Rebalance every year or so 6. Consider tax loss harvesting each year 7. Perhaps add two types of funds which have a low correlation to stocks - REITS and precious metals. Each should be under 5% of your portfolio, though.

It's all quite easy, actually. For pretty much all you need to know in condensed form read Allan Roth's "How a Second Grader Beats Wallstreet" or Bill Bernstein's "If You Can."

I've done some recent reading from financial independence evangelists (ex. Mr. Money Mustache, etc[0]) who advocate low-fee index funds like Vanguard's. The idea (which generally makes sense to me) is that the overall stock market has historically significantly beat inflation over basically any period over 10 (20?) years, so an investment that has that overall profile is fairly low-risk. There are various things you can do with 'target retirement' funds or other methods of allocating more towards bonds as you age to reduce your short-term volatility risk as you near the age you want to use the funds.


Go with fund managers who genuinely invest over the long term, there are surprisingly few, and over 10+ years it is possible to generate better returns. Picking companies you think will beat the market over 10+ years is about fundamentals and research, not "playing the market".

Too many fund managers aim for consistent shorter term performance, which is getting into dice rolling territory.

Have a look at the fund managers that big pension schemes use. Some of them do genuinely perform well over the long run by not being distracted by short term issues.

Invest directly in stock and bonds. And if that is not available, than invest in low fee funds.

"Invest directly in stock and bonds."

That is pretty much the opposite advice you should take from those findings.

In general, investors now have the widest array of low cost, diversified instruments available at any point in history. Most advisors who know what they are talking about, will tell you to take one of these options (a non-managed index fund, a highly diversified etf) and invest in that. Rebalance once a year and don't worry about beating the market.

> Most advisors who know what they are talking about, will tell you to take one of these options (a non-managed index fund, a highly diversified etf) and invest in that.

Thanks, do you have any references for this, or more details on such options?


Also "A Random Walk Down Wall Street" is a central book in this genre.

I will say, you can't generally control how your pension is run (and they run the gamut from wonderful to nearly criminal). It's one of the reasons I highly discount pension funds as a benefit.

In the UK the latest hot thing is Self-Invested Pension Plans (SIPPs) and being self-employed all of mine is in that. Hopefully better performance in the long run, but a lot more faff/stress making good decisions :)

A Random Walk is great, as is Bill Bernstein's the 4 Pillars of Investment.

Well I do not advocate stock picking to beat the market. Just to construct the index tracker yourself instead of paying a fee for someone else to do it.

That is still phenomenally bad advice. You aren't paying index funds or etfs for their stock picking, heck most of that is in publicly available documents. What you are paying them for is their operational capabilities and economies of scale.

I would be amazed if a retail consumer could put together a balanced, diversified portfolio with the same cost as an actively managed fund. To try and compete with an etf or non-managed index fund is crazy.

The expense ratio of VTSAX is .05%. It's ludicrous at that price to do it yourself.

I work in a small prop-shop doing HFT. Most of the guys there are manual traders, and I've seen the same guys there for several years making steady money. I don't think this is bs or amateur hour, let me explain why. It is essentially the small (independent) traders that can make a win when the big guys (hedge funds, pension funds) are moving their positions around. Imagine a dude on a surfboard enjoying the wake of an oil tanker and you get the idea. Just don't get too close :-)

Exactly, small moves that George Soros wouldn't blink at are the bread and butter of these guys. I think of it like a sea of fish, these guys are almost the smallest tuna out there and they find the food the bigger guys don't want.

Any suggestions for HFT programming reading material and background knowledge for that kind of work?

I learned most of it on the job, from the masters. So I don't really know about the literature. The non-finance part of it involves alot of real-time stuff, which is found in game programming, and also audio software. Also, you could read up on networking, TCP/IP, etc.

As for risk control (by risk I mean bugs) I often wonder if there are lessons from eg. the nuclear industry, on how to keep complex, highly-strung systems on track. That's another thing I learned on the job, and not always the easy way :-)

The problem with using nuclear industry systems lessons is that the nuclear industry moves exceedingly slowly (for good reason) but trading needs to move fast. Opportunities exist for a very short time, so being able to find them, exploit them, and not blow yourself up in the process is the trick.

A trick not many folks have proven they can do over the long haul.

Cool, how'd you get into the industry if you don't mind me asking? Math background and/or degree I'm guessing by the username? Thanks for the response.

The book Flash Boys provides a solid understanding of how HFT got started. http://www.amazon.com/Flash-Boys-Michael-Lewis/dp/0393244660

No it doesn't. Everyone I know with HFT industry experience that has read it (including myself) has panned it. It is possibly the worst thing you can read if you are interested in how HFT works. "Dark Pools" has it's own faults but for narrative non-fiction it is the only game in town.

And here I was hoping the meat of the blog post would actually be in the post. Nope, just a teaser for part two.

Also, why is this submission marked "2011"? It was apparently written today.

They removed it now... must have been a typo

I worked at an investment bank writing software for traders like the author. I never quite knew how it worked, but it somehow paid my salary. (It seemed a lot like crazy kids doing whatever they wanted. That might not be far from the truth.)

What I always thought was interesting was that when the traders blew up like this, they became the PMs or support for the software :)

Technical analysis doesn't work. Big surprise you lost money. Those guys that you thought were good at it? They lost their shorts at some point too.

My first job out of college was building a TA package at a then-small Bloomberg competitor. I knew basically nothing about any of this and just approached it as a technical challenge, and I had a generally ok time just getting paid to write software that was vaguely mathy. This was software that was actually used by people trading, who were paying on the order of tens of thousands of dollars a year per seat.

The first signs that something was amiss arose pretty early. I kept some printouts of what the various metrics were in my desk. Occasionally a support employee would come over and say so-and-so at some bank wants to know how (say) Bollinger Bands are defined. I'd hand them a copy of my definitions sheet covered in capital pi's and sigmas. They'd look at it, their eyes would glaze over, they'd wander away, and I never got a single followup question.

About four years later a client discovered that there'd been a bug in a few of the calculations that made them wildly "incorrect." Many people used the product for years as happy customers without realizing this.

At that point I must have googled "technical analysis wtf" or something, and realized that basically what I had been spending my time on was the financial equivalent of biblical numerology. So it goes.

I'm wrong all the time and lose money all the time. The big difference here is I let one get away from me instead of keeping it small/manageable like I always do. It was a situation where I could have controlled it and I didn't. I'm still up more than 4x what the final realized loss was in my trading career.

I used to think TA was a joke. I was very skeptical before using it. I have never bothered to explain why it works intellectual to non-believers. But I do think someone who understood probability and saw the compiled statistics of practitioners would concede that there's 0 chance of non-randomness.

Yeah, no. That's called anecdotes, because when you do understand probability and backtest these strategies, they are known not to work in the large.

Stat arb is what happens when you actually do statistics, and these days it's got only a slim to nil advantage.

Trading is like the Israeli nuclear program: those who talk about it don't know about it, those who know about it don't talk about it.

If technical analysis actually worked, you'd be able to find the "Technical Analysis Toolkit" for on GitHub, and... technical analysis would no longer work. A little bit of grepping around yields: http://ta-lib.org/ (Technical Analysis Lib).

Ergo: if technical analysis ever worked (there is some evidence it did before about 1990 when personal computers became ubiquitous) it almost certainly does not today.

The only way market timing approaches can work is if they embody significant information that is not generally available. A successful market timer has to be smarter than everyone else in the market at the time of each trade.

Anyone claiming technical analysis works is claiming that there is an inefficiency in the market that has been well-documented in public for decades, to the extent that an open-source BSD-licensed tool for identifying the inefficiency exists, and yet the inefficiency still exists.

This is simply contradictory.

An economics professor is walking down the street with a student. The student sees a $100 bill on the ground and tells the professor. The professor says, "Nonsense! If there were a bill on the ground, someone would have picked it up already!"

I actually think that in this case you are probably right, though. But I wouldn't bet my life on it.

I love this joke, because on the surface it's making fun of economists whose theories blind them to an obvious reality. But how often does anyone actually find a $100 bill on the ground?

I found a 20 on the ground by a gas station pump. Its not improbable that someone taking their keys or wallet out would accidentally drop any loose bills they had.

My wife found one once on a very windy day in Denver.

Of course not every idea ever has been tried. But there's a lot of incentive.

Statistically it would be ok to bet your life on it, though.

Did you hear the one about the economist who drowned crossing a river that was three feet deep on average?

Haven't heard that one until now that you mention it, nice.

You sound like the people I've seen on gambling forums postings about their unbeatable roulette system that they've "really won with in the long term".

There are actually ways to beat roulette.

1. Observe the motion of the ball and the wheel between the time they start moving and the time betting is closed, and use physics to calculate where the ball is likely to land. Bet accordingly. This was proven effective in the early '80s [1].

2. Record a lot of outcomes on a given wheel to learn its biases. Bet accordingly. You might think it would be hard to surreptitiously gather all the data you would need, but it is made considerably easier because you do not have to be surreptitious. Casinos love people who are taking notes--99.999% of the time it means that is someone who has some idiotic system that they think will let them beat the house, and casinos want to encourage such people.

There was an episode of the wonderful documentary series "Breaking Vegas" [2] that covered this method, focusing on a family that practiced it. They were making a lot of money until the Las Vegas and Atlantic City casinos banned them. American casinos can ban you for winning too much, even if you are not in any way cheating. They switched to Europe, where the casino regulators prohibited the casinos from banning them without cause, and "winning to much without cheating" is not cause. The casinos tried moving the wheels to different tables to confuse the family, but the family had spent so much time looking at each wheel they could recognize individual wheels be wear patterns and place the appropriate bets. It was pretty cool.

3. My favorite, even though it was flat out cheating. This was also covered on a "Breaking Vegas" episode. Before I explain this cheating method, we need to take a look at an older cheating method that is no longer effective. That was "past posting" or "late betting". In past posting, you place a bet AFTER the outcome of the event you are betting on has been determined. So in roulette that would be placing a bet after the ball has fallen into its final slot.

Of course, that would be hard to do. The risk would be too high that the operator might have noticed that there were no bets on that particular number when betting closed. So what you actually did was modify a winning bet after the fact to make it bigger. You put, say, a stack of 3 of the lowest value chips on a number. If that number loses, you lose those chips. If it wins, the rest of your team distracts the operator, and you swap that stack of 3 low value chips for a stack that consists of 2 low value chips on top and 1 high value chip on the bottom.

There was a guy who was doing a lot of past posting on the roulette tables, back before they had constant recorded closed circuit TV surveillance. The casinos suspected he was doing something to cheat, because he was winning more than they would like (but not enough to get banned), but they couldn't catch what he was doing.

Then surveillance came, and when he would win with a stack of 2 low value and 1 high value chip instead of paying right away they would detain him while they pulled the tape and reviewed it to see if they could see any shenanigans. He had to stop past posting. His career as a roulette cheat seemed over.

And then he had a brilliant idea. Instead of cheating by converting low value winning bets into high value winning bets, why not go the other way? Convert high value losing bets into low value losing bets!

The new plan was to place bets consisting of a high value chip on bottom with some low value chips on top, with the low value chips positioned so that the operator would not see the high value chip. It would look to him like a stack of low value chips.

If their bet lost, they would swap the stack for a stack of all low value chips. If their bet won, they would not touch it, and point out to the operator that there was a high value chip there so they would get the right pay off.

The casinos were, of course, very suspicious. They recognized that they were dealing with a known past poster, so every time he won they pulled the tape...and the tape showed that no one had come near fiddling with the winning bet. The high value chip had been at the bottom of the stack from the moment the bet was placed.

He got away with this for a ridiculous amount of time, with the head of security from one of the major casinos reviewing every win and getting quite frustrated at not being able to catch how the cheat was working. I forget how he finally got caught.

BTW, I highly recommend "Breaking Vegas". They profile some remarkable people, some who cheat, and some who won legitimately. A couple neat examples.

• They have an episode on dice dominators. These are people who, through great practice, can roll the dice at craps with such control and consistency as to get the outcome they want much more often than by chance.

• They have an episode on a man who took up counterfeiting slot machine tokens. The casinos detected this because they were getting high token counts at the end of the day, but they could not tell which tokens were counterfeit. They sent batches back to the manufacturer, and the manufacturer said that all the tokens that were sent were real.

His downfall was interesting. He was playing a slot machine that took something like $25 tokens, and the machine jammed. He moved over to the next machine and continued playing. Security happened to see that on camera, and that made them suspicious. The normal behavior when someone loses a $25 token to a jammed machine is for that person to get mad, and go find casino staff to seek a refund. No one just moves over to the next machine and keeps going--unless that $25 token isn't worth $25 to them. Hence, the guard thought he might be looking at the infamous token counterfeiter. They followed him to the parking lot when he left, and when he opened the trunk of his car that he had boxes full of tokens for all the major casinos.

[1] http://en.wikipedia.org/wiki/The_Eudaemonic_Pie

[2] http://en.wikipedia.org/wiki/Breaking_Vegas

Wow! Fascinating comment. I'm going to have to check out "Breaking Vegas".

Yeah, I alo made a fortune in the market except for the money I lost.

I recall seeing a graph of a stock index in which all but the 10 (e.g.) best days within a very long time span (~years) were removed - to mock up a trader who removed his money at a few unlucky times. You ended up losing huge.

The lesson was exactly what you're saying -- you also have to count that one huge loss. You can't fence it off as an exception and remove it from the accounting.

Wealthfront posted a graph on that. I called them out on it on Twitter which they didn't appreciate.

Url please?


Not sure how to find the Twitter interaction as Twitter isn't big on history as far as I understand it.

Thanks for finding the plot.

Yep, I tried to find the graph but failed. It would be interesting to see again. I have a professional interest in probability and stochastic processes, so it stuck in my mind.

I have seen the graph before, I was curious about the response on Twitter.

No surprise to me that these guys win some and lose some. All they are doing is trying to predict the future of a stock with absolutely no knowledge other than a graph of price movement. If it's going down, it will continue to go down. It it's going up, it will continue to go up. It's a fools game.

Hey Jbuzbee,

This is what I thought as well when I was initially introduced to the idea of technical analysis and day trading. I was very skeptical.

I don't want to call myself a probability expert but after studying poker theory and reading mainstream works like Fooled by Randomness, I bought into the idea that it was just a bunch of a guys throwing darts and the "winners" whom were trying to sell all their BS were simply benefiting from survivorship bias.

But after seeing the numbers themselves first hand, there's just no way it's random chance. Being net positive 80% of all days traded with all your winners and losers falling in a relatively tight distribution -- luck can't create highly specific, repeated outcomes like that. There is virtually zero tails risk since 99% trades are intraday only.

My loss was a failure of discipline. I froze like a deer in the headlights, failed to execute like I normally do and paid the price. Hope you keep reading to find out what happened!

> Being net positive 80% of all days traded with all your winners and losers falling in a relatively tight distribution -- luck can't create highly specific, repeated outcomes like that.

Yes, it can. E.g. you could sell deep out-of-the-money puts and collect a $1 premium day after day, say 99% of all days. Until one day a {terrorist attack in the US, humongous earthquake in Japan} happens and you lose more money than you ever made.

I just can't stay out of the debate no matter how much I try to.

I am as well-rehearsed in financial history as anyone. I have read about LTCM, Nick Leeson (I even watched the movie), etc. In these situations there was so much size being used that if any unexpected squeeze scenario occurred, those guys would move the markets and cause a horrible chain reaction.

I understand on the surface this totally looks like I'm trading in a "eat like a chicken, shit like an elephant" type of fashion. If I were in your shoes and I read "guy made x, x, x consistently and lost 20x one time!!!" I'd be thinking along the same lines. The difference is, I had more control of my outcome. It's hard to prove this and you won't totally believe me unless you are also a day trader who grinds it out and has a feel for intraday liquidity and slippage (particularly on the otc/pink sheets). I wasn't trading such a large size where I would move the market if I was squeezed out. The risk distribution of intraday scalps is not at all similar to writing naked options with unlimited loss.

It's like this: normal trade: entry signal occurs. get in. exit signal occurs. get out. the trade in particular: exit signal occurred and i chose to ignore it and keep scaling in. would there have been awful slippage? yeah but it would have manageable. in the heat of the moment on the largest loss ever, 10-15c slippage on a $4 stock massive size feels like the end of the world but it's better than riding it down 50c or a point.

Obligatory case where exactly that happened, at a cost of $1.4 billion: http://en.wikipedia.org/wiki/Nick_Leeson#Downfall

Which is why it's good to have net-short deltas (when beta weighted against the SPY), to protect against tail risk.

It's also why it's important to trade small, and have a large number of uncorrelated positions.

You can go on believing that nobody makes money doing this sort of thing over the long term. That's fine. But it's definitely not true.

Derivatives have no more risk than the underlying. What makes them more risky is the leverage. Selling 1 option w/ a 0.5 delta is no more risky than selling 50 shares of the underlying.

My comment is an example of why the reasoning below incorrect. My comment does not say "it is impossible to make money by trading stocks/derivatives".

> Being net positive 80% of all days traded with all your winners and losers falling in a relatively tight distribution -- luck can't create highly specific, repeated outcomes like that.

But you don't understand. My special variety of the Martignale system works to beat the wheel in the long run. Guaranteed!

We call it picking up pennies off a train track.

There is virtually zero tails risk since 99% trades are intraday only.

Unless something meaningful happens to the business while you're exposed, no?

I suppose you're right and that's why I wrote "virtually zero" and not "no chance whatsoever".

Not only does significant breaking news have to happen while you are in the position, which is already very rare, it has to happen in a way in which you cannot respond and cut it off -- like the stock getting halted. Most intraday stock halts that occur are not exactly a shock (like a biotech or a stock under investigation of any kind) so if you are highly leveraged on that stock to the point where your entire equity can get hurt, that's on you.

If something big like GE or AAPL halts, the move proably won't be extraordinarily huge like 20%+, unless it's an Enron scenario.

I feel like you're misunderstanding Fooled by Randomness. The results are supposed to look like steady gains in a tight distribution, but the super far out tails are fatter than the average person thinks. Therefore, typical traders take average risk with below average returns (relative to the risk one bears) whereas Taleb takes high risk with supremely above average returns (again, relative to the risk he bears)

When's part 2 slated for? That was annoying to just get into the meat of the story and then...

He got me reading the whole thing and then it stopped abruptly.

The guy basically just posted the introduction.

Wow. I just saw my pageview count go up like a rocketship and saw that HN was the reason why. Kinda cool.

Get out now! It'll crash any second now.

Nah, just short the server!

Hey guys! Thanks again for all the interest. Since this site was responsible for more than 90% of my traffic, I thought I'd address some of the comments in this thread. Hope you all enjoy.


Long story short:

Guy makes living watching dead cats bounce, and correctly realising when it's bouncing and when it's peaking.

One day, he misses the peak, due to some SNAFU.


I made a living doing exactly such for a little while, until I lost the whole damn lot due to having to reboot in the middle of a big position. Own damn fault, should have had another screen spare.

Every time I try to read about trading, a little Baron Munchausen in my head starts going "Your reality, sir, is lies and balderdash, and I'm delighted to say that I have no grasp of it whatsoever." :-/ I dunno.

(This goes double for HFT.)

I feel the same way every time I read a story about a startup trying to put a web front end on some service industry.

This goes double for when I'm reading about their valuations.

Anybody have a glossary?

http://www.investopedia.com/terms/h/hitthebid.asp - there's one term from the article defined for you. The site has lots more.

If there are still terms you can't find, post them here and I'm sure someone can explain.

You can use investopedia for almost anything finance/trading related.

What a tease! I want part 2

Do you think technical analysis works? Look at the mountains, try drawing resistance lines, my god, you are able to predict when the mountain ridges are going to break through right past 9000! Are you Jesus?

Don't forget the magical term 'support'. Or 'Fibonacci levels'. Or 'resistance zones'. If I see another blog post by the brokerage firm I'm with about freaking 'supports' I'm going to hurl. Technical analysis puts everything from astrology through religion on to homeopathy to shame in its bullshittery.

The most entertaining part is when they go on about the support levels between the currencies. "NOK-NZD is about to break the 5.30 support level and this chart indicates that NOK-NZD is going to go up." It just leaves me thinking WTF, how do you go from here to there? I think, more honestly, the actual example I saw was CHF to JPY.

'Butterfly patterns'? You sure you're not reading tea leaves?

That said, I'm sure technical analysis works. Something like 0.5+ε of the time. And, if you find the right sort of TA, if it's just RSI or some MACD crossover with specific sets of parameters x or y for that stock you might beat buy and hold or whatever. 50% of the time, if you're lucky, before commissions.

But I'm not even convinced, nor should anyone with half a brain be, that 365 days is somehow a magical interval over which we should compare in the long term or anything.

Just buy and hold and index fund, people. Or something very managed with a very low level of turnover and low fees, keeping in mind an appropriate diversification period. When to buy? Today, or gradually over the next few months with dollar cost averaging. Holding period? Ideally forever. Simple.

When I saw the post I was worried that HN would eat this crap up. But this is the appropriate level of revulsion. Whew.

I'm glad you're here to back up the other revolted skeptics.

I've just had a revelation.

MACD has three days as parameters, 12, 26 and 9 being commonly used for completely arbitrary reasons.

Biorhythms use 23, 28 and 33 for similarly arbitrary reasons.

Maybe if we found the right magic numbers we can establish a link between these two highly reputable sciences.

I figured that stuff was determined by looking at orders to buy at a certain price? If there is a large number of orders to buy currency at 5.30 from your example you would say that it is at least likely to stay or bounce until all the orders are run through?

Correct me if I am wrong, don't trade myself.

Nope, technical analysis of the usual sort doesn't look at the book at all. It just looks at the past prices and, if you're really lucky, past volumes.

Going past what's traditionally technical analysis, in that example at 5.30 it's possible that there's a wall of people. However, the trading in any intermediate currently pair between those, NOK-USD-JPY, for example, will negate that wall in no time at all for sheer volume alone and any artificially imposed walls based on this kind of numerology would be chopped down by HFTs looking for arbitrage opportunities in milliseconds.

If you're looking at a big pair like EUR-USD with a hypothetical wall at 1.25, for example, it's possible it might last for a little longer. But again, arbitrage across currency pairs - EUR-JPY-USD or EUR-GBP-USD or whatever - will wipe out any numerical niceties in marginally longer than no time at all.

That said the more TA traders are out there the more the actual investors (and with SPY having an annual volume 8.5 times its number of shares there seem to be too few of those) can get themselves a long-term good average of "Be fearful when others are greedy and greedy when others are fearful." just through regularly putting money aside and investing through dollar cost averaging in index ETFs/mutual funds.

But people ARE emotional? So if a stock was trading at a level, then dropped significantly, it makes sense that it sees "resistence" as it climbs back to that level. There are real people there who are happy to be back at $0 or back where they started so they sell.

I believe stock prices are random, but it doesn't mean that market phenomena doesn't exist.

If you don't think about technical analysis as charts and lines, but rather patterns or cycles of market (and human) behavior (which is of course what the charts and lines represent) you might see how technical analysis could have some validity.

Are there patterns in markets? Are there patterns in crowd behavior? I think so. Others may disagree.

Another point... if enough people believe and act on technical analysis signals, the prophecy becomes self fulfilling to an extent.

This is not to say a consistent simple mathematical formula involving Bolinger Bands or the like can be developed with any certainty. I think prediction based on patterns is art as much as science and involves a lot of probabilities rather than exacts.

You know what else has patterns? The weather.

Not saying TA is legit, but uh, we can predict the weather with better than random probability.

You're kidding, right?

But Josh, don't you understand that 120 days in a past market is just the same as 120 days in the future market? As we all know the stock market is innately tied to certain magic specific numbers of days that are ever unchanging. Like 14, 30 and 70 will always be relevant when you're RSIing.

Incidentally, your biorhythm cycle may be at a low point atm, so take care. Especially as you were born under the astrological sign of head and shoulders. Meanwhile, however, I've heard about this guy who tells me that at least on the Australian ASX you can't even rely on the number of shorts having been placed to predict whether a stock will go down. But to that I say poppycock, there's no way, after all, that the stock prices reflect the currently public knowledge and thus have already assigned a reasonable price taking into account all that is reasonably known about the future.

Never forget - the average market participant is always smarter than the market and therefore TA works for them /s

Good point.

(Ps: joshua not josh)

Not sure why the title has (2011) in it. This was posted today, November 11, 2014

Just for clarity, the loss in question occurred March 2014.

Whoops—not sure what happened there. Fixed.

I think this is the part of the story about his experiences in 2011.

Nah, the story starts in 2013

I guess this is only written for other traders? I couldn't make much sense of it and got bored pretty fast.

That's how most of us feel when we see articles about SF real estate or politics :-)

Why are you telling me? I don't post articles about SF real estate or politics, I just like to comment on them. In fact, I haven't posted anything so far :-)

Relevant to your comment:

"I guess this is only written for other San Franciscans? I couldn't make much sense of it and got bored pretty fast."

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact