
Someone got the natural gas report 400 ms early - HockeyPlayer
http://www.nanex.net/aqck2/4090.html
======
apaprocki
[http://invezz.com/news/alternative-investments/625-uk-
report...](http://invezz.com/news/alternative-investments/625-uk-report-
advises-against-more-hft-regulations)

"Veteran traders would usually wait in anticipation for the weekly report of
gas-inventory figures by the U.S. Energy Information Administration released
on Thursday at 10.30 AM and then dive into the busiest trading window of the
week. This is no longer true as most traders are now staying out of the market
due to the HFTs new strategy - sending floods of orders in an effort to
trigger huge price swings just before the data gets released, also known as
“banging the beehive”."

edit: Fast algo puts orders out on multiple equity exchanges and then hedges
itself with a slow algo in the futures market.

~~~
lifeisstillgood
At what point will HFT drive out the proper functioning of a Market?

Have there been any studies on this? If human traders mostly reacted to "real"
news (the Orange juice crop is bad this year) then human trading was mostly
linked to actual changes that affect the price mechanism

But if large volumes of trades are speculative, or even worse, are directed at
affecting the behaviour of other large Market players, is there not a point
where the selling of actual OJ is just noise in the Market?

One can anticipate HFT bots acting like the strange Amazon pricing of obscure
books in the millions of dollars, bots competing against bots to acquire more
of the rent redistribution. If we can imagine that we can imagine a failed
crop with prices being driven down in a frenzy.

Surely some research exists on this? Where is the price signal tipping point?

~~~
runT1ME
I've been wondering if an exchange that prevented HFT would prosper in the
current climate. I'm sure that plenty of companies aren't a fan of their
market cap being at the whim of an algorithm and the large number of swings it
would undergo.

Wouldn't they prefer an exchange that offered liquidity in minutes or even
hours, opposed to fractions of a second?

~~~
prostoalex
What is your definition of H though? If someone decides to offer a large block
of shares for sale, and they break it into 100 lots, are they a high-frequency
trader under this definition? What if it's 10,000 lots? 10,000,000?

This is a data structure problem at its core - the fact that someone entered
one billion BUY orders at $15.51 should not prevent me from seeing that
there's an outstanding order at $15.52. The stock market employs the queue
only for legacy reasons.

~~~
KMag
"This is a data structure problem at its core - the fact that someone entered
one billion BUY orders at $15.51 should not prevent me from seeing that
there's an outstanding order at $15.52. The stock market employs the queue
only for legacy reasons."

(1) I'm not sure I understand what you're saying about price information. You
seem to be saying that level 2 quotes don't exist. I assure you that for the
major exchanges, level 2 quotes do exist, but they're more expensive than what
most discout brokerages/Yahoo/Google give you.

(2) I'm not sure what you mean about queuing at price levels being a legacy
artifact. There needs to be some objective rule for deciding who trades with
whom when there are multiple participants at a given price level. In most
markets, it's first-come-first-serve (a queue). I read that MS POOL was going
to try prioritizing based on size, but I haven't heard anything since. So, if
it's not first-come-first-serve, and there are two offers at $5.43 and a
trader comes in and lifts one of the offers (not enough size to lift both),
which of the two offers should get lifted?

~~~
MrMan
several futures markets queue partially or only by order size.

------
lrm242
The most likely explanation: no one got anything early.

Venue timestamps can often disagree by a significant amount. It is very likely
that SIAC (distributors of CQS and CTS) simply are not well synced to the
reference clocked used to distribute the report.

Nanex spends a lot of time doing analysis based on precision timing without
providing any sort of error analysis as to how well timestamps produced from
different references synchronize. It would lend credibility to their
hypotheses if they either provided their own reference or provided some
measure of margin of error to the timestamps they so heavily rely upon.

For example, if Nanex had stated that they measured the time of release and
the time of trades themselves as they appeared in CQS as observed from their
machines then they could make a much more concrete statement as to whether:
(a) the report was "early" or (b) the trades were "early" relative to one
another. Most likely they would observe that the report appeared "early" as
measured to their system clock (or possibly some other reference), but the
trades did not appear before the report. Of course, Nanex would be smart
enough to account for transit latency and other what-not.

~~~
JumpCrisscross
The data are printed with the CME's and NYSE's official timestamps. It would
not be unprecedented for the CME's timestamps to diverge from, say, the
NYSE's. That would present itself as a consistent temporal dislocation between
the CME and NYSE, but would be invisible within each data-set (sort of like
your calendar putting itself into the wrong time-zone where - the times are
off, but they're consistently off, i.e. spacing between events is preserved).

Instead, we see a signal _within_ each dataset and symmetry _between_ them. It
could still be a data aberration, but that would implicate the CME and NYSE of
having irregular clocks, a far more serious problem than some twat front-
running a government report.

Most likely is that this is a real trading signal, but not one based on
insider information.

~~~
lrm242
Relative timestamps between venues don't matter. More importantly, using
timestamps without a common reference point makes it impossible to assert what
happened first, because the absolute value of the timestamp does not matter.
What matters is what "really" happened first. Of course, everyone's view of
"first" can differ, but that's a different issue.

Whether there is signal here or not wasn't my point. There may well could be.
My point was simply that the analysis provided by Nanex is spurious without a
definition of the timestamps, where they came from, and their relationship to
known reference points. Taking three events with a timestamp from each of:
CQS, CME, and an arbitrary point in time such as 10:30:00 and then asserting
that events observed within a CQS or CME event stream happened "before" the
arbitrary point in time, 10:30:00, is impossible.

Unless either (a) one observes CQS, CME, and the event scheduled for 10:30:00
from a known reference and accounts for all variables (differing transit
latencies, etc) or (b) each of the generating events are known synchronized
and the current state of that synchronization is observable (CQS lags CME by
475us, government event source leads CME by 1.2ms, etc). Clearing (b) is not
happening today. (a) can be accomplished by Nanex, if they so choose.

~~~
niggler
"(a) can be accomplished by Nanex, if they so choose."

Doing so doesn't give them a sensationalist headline that finds its way to the
top of HN.

------
smcl
Or someone got the report a few hours early, didn't want to be seen to jump
the gun (insider trading, and all that) and kicked off a new trading strategy
400ms earlier than they intended...

~~~
rikacomet
and of course that also is only possible, when they know the exact time of
when the news would go out. So that, they can time their activity just
milliseconds before that, using tech.

~~~
tazzy531
For a lot of these economic numbers and earnings report, they are provided to
news organizations prior to the announcement with instructions that the
information cannot be released prior to a certain time.

So, for this example, that time was 10:30AM. The moment it hits that time, the
reports get pushed out into the newswires and reported on-air.

This allows the reporters to digest the numbers and figure out how to report
it.

~~~
analog
So news organizations get this financially sensitive information before
traders? Seems an almost too obvious way to make a quick buck.

~~~
ig1
Yes, but they have strong motivation not to abuse it. I've seen one case where
an news org broke an embargo due to fat fingering a release, that news org
lost early access privileges for a year as did every sister company that
belonged to the same news group.

If you specialize in financial news losing numbers is a huge deal, you'll lose
thousands of customers over it. And when those customers are often $1000/month
subscribers it can easily mean a financial loss of tens of millions of
dollars.

Within news organizations the information is typically restricted to 1-2 named
individuals who have restrictions on trading. If they leak the information
they can generally be criminally prosecuted.

~~~
niggler
R.R. Donnelly is still kicking even though they released google's earnings
early: [http://www.reuters.com/article/2012/10/18/us-google-
results-...](http://www.reuters.com/article/2012/10/18/us-google-results-
idUSBRE89H14Q20121018)

~~~
ig1
I'm talking about government figure releases (employment, inflation, etc and
the like).

In that case it was probably just a contractual issue between Google and their
printer, to the wider market it would have been the same as if Google had
accidentally released their numbers early.

(Actually even in the governments case it's still a contractual issue, it's
just that governments have huge power by withdrawing early access rights)

If RR traded on those numbers they would have violated criminal insider
trading law.

------
aleyan
Some back of the envelope calculation for UNG (ETF in the top Nanex graph)
profits for the early mover.

In the second before the announcement UNG was trading at $18.72 High and
$18.55 Low. According to some minutely data I saw 400,000 shares where traded
between 10:29:00 and 10:30:00; I believe this ties out with the first Nanex
chart where there are 490,000 shares traded, with majority of it coming in 400
ms before the 10:30:00 mark. 4 Seconds after the 10:30:00 report release the
price had stabilized at around 18.51. I will consider this $18.51 the fair
price with all resonably fast algos having made their post release moves.

Let us assume that there was a single trader/algo who got the report early and
executed all of the 400,000 share sells 400ms before 10:30:00 and all other
market participants only bought. Additionally assume that the average fill for
these sell trades was the average of the High and the Low at (18.72 +
18.55)/2= $18.635. I believe this fair because looking at the first Nanex
graph the early trades are somewhat uniformly distributed between the high and
the low. In a simple arb on UNG, where the trader went short 400ms before the
the announcement and closed the position at the fair price a few seconds after
the announcement he stands to make a profit of (18.635 - 18.51) * 400,000=
$50,000.

For a trade that lasts 5 seconds, making $50,000 is nothing to sneeze at, it
is not that much in grand scheme of things. Additionally other ETFS and
futures were impacted and could have made more or less money.

TL;DR: If one guy captured all the profit from the early UNG trade, the max he
made was roughly $50,000.

~~~
elemeno
If the total value of any trades that might have been made in the claimed
400ms window was only $50k, it would suggest to me that there was nothing
untoward going on, and that this story is most likely nothing more than a
clock error. For any of the HF firms, prop houses and Hedge Funds that I've
run into (which is not a short list, but my no means exhaustive, given that I
work in finance) $50,000 is not interesting action - especially since
transaction fees will eat away a reasonable chunk of that.

~~~
niggler
It is a clock issue, as observed by others in this thread.

I would point to <http://news.ycombinator.com/item?id=5146571> because its
clear from nanex's response that he hasn't fully thought through the clocking
issues with using CQS data without observing it directly himself.

~~~
apaprocki
Or, it's not a clock issue: [http://live.wsj.com/video/banging-the-beehive-
explained/A943...](http://live.wsj.com/video/banging-the-beehive-
explained/A943CC10-8E9D-45E6-B986-A73352A7CF03.html)

~~~
niggler
Have you looked at the raw data in this particular case? The particular
deviation that nanex points to isn't borne out in the exchange raw feeds ...

------
niggler
After seeing some of their posts earlier and comparing it to live data I
record at the colocations, I've concluded that they have clock issues which
makes these types of anomalies appear frequently. Or they have a bad data
vendor.

Interestingly enough, even the regulators don't have good (only millisecond-
resolution) trade data.

~~~
nanex
These are charted with CQS (official) timestamps. Try again.

~~~
niggler
CQS has had issues in the past. You should know this.

More importantly, why wouldn't you invest in colocations and collect the data
yourself using direct feeds (with GPS clock synchronization etc to validate
the data)?

~~~
nanex
We do. You are grossly misinformed. The charts correctly show the sequence and
times of this event. I'm not going to engage this discussion further, though
pmail is fine.

~~~
scott_s
As a bystander who does not have the background necessary to evaluate either
of your claims at face value, I would really like to hear your explanation for
why you disagree with his explanation.

~~~
niggler
A simple analogy: you are an advertiser using Facebook. You tell them you want
to get 1000 impressions, and facebook gives you a report at the end of the day
saying that they gave the number of impressions you paid for.

So you look at yesterday's view count and today's view count and notice that
somehow the view count only increased by 900. Something is afoot!

Nanex's argument is tantamount to saying "that means we must have lost 100
organic views today"

My argument is tantamount to saying "I actually bothered to look at our access
logs (which we record on our servers) and only saw 900 that we could
definitively attribute to real Facebook users. Is it possible that the report
is incorrect or falsified?"

Now I bring up this example because this actually did happen with facebook.
Quick HN search revealed one such discussion:
<http://news.ycombinator.com/item?id=667308>

Back to the current situation. There are many sources of market data. Each
individual exchange generates its own feed, and with the major exchanges
(NYSE, NASDAQ, BATS, DirectEDGE, ...) you can colocate in the exchange data
centers (NYSE and ARCA are in Mahwah NJ, NASDAQ is in Carteret NJ, and various
other exchanges are located in New Jersey and Chicago) and record the data
yourself. There is a unified tape (CQS/CTS) which combines and disseminates a
combined record (across all exchanges). This is used to determine the
"national best bid/offer" -- the prices people are willing to buy/sell at.

The process of CQS generation is fraught with problems, but lots of older
traders and academic types use CQS data because its much cheaper to get that
data than to get data from individual exchanges directly. However, you are
subject to the quirks of the combination process, including subtleties
regarding timestamping data (since this data includes trades and quotes from
Chicago and from New Jersey, the sequence of events may appear different if
you record from chicago or new york or philadelphia or some other place; if
you ask the exchanges to timestamp directly, you have to worry about clock
delay and skew between the exchanges' servers).

nanex is saying that it is acceptable to depend on that data and any anomalies
must have occurred outside of the recording process. I am saying that the
recording process can create the types of anomalies that nanex is showing, and
that the only way to be sure is to record the data directly and carefully
synchronize your recording machines. _AND_ when you do that you see that there
really is no anomaly.

Just to emphasize how sloppy the exchanges are with regards to timing: on the
BATS exchange they use multiple servers to run trades and generate quotes, and
every once in a while you see messages appear to be out of time order because
the individual machines weren't properly synchronized (although, if you filter
for a single ticker, messages are always in chronological order)

~~~
noname123
There's quote feed solutions that do co-location at the exchange servers and
deliver the quotes to you in individual channels (ARCA/BATS/EDGX-A etc) and
also SIAC feeds.

<http://www.limebrokerage.com/services/marketdata/citrius>

~~~
niggler
That looks like a hardware product. You still need to purchase market data
access to the relevant exchanges to get that data, and those are expensive
(NASDAQ costs, for example, run upwards of 20K/mo to get the lowest-latency
data)

They should just spring for the data before making accusations -- the problem
is that when you cry wolf all the time no one will take them seriously when a
real case comes around.

------
zaphar
This has nothing to do with the main subject matter of the article but those
graphs are almost unintelligible. I didn't even try to decode what they were
trying to communicate.

~~~
JumpCrisscross
The graphs are brilliantly clear to me, but I have a background in electronic
derivatives trading and so am used to staring at wiggling psychedelic surfaces
for hours on end. Nanex's audience is algorithmic securities traders. Imagine
trying to explain a circuit diagramme to a room of uninitiated economists ("is
there a reason the wire goes all squiggly right there?").

~~~
cgcdesign
"electronic derivatives trading" is the phrase I was looking for. Thanks a
bunch

------
mixedbit
I don't think that increased trading activity supports a conclusion that the
report leaked.

People may have anticipated increased trading after the rapport release and
may have prepared algorithms to try to gain during this event. The algorithms
may have started working before the release.

I'm not saying this was actually the case, but my theory is as well supported
as the claim in the post.

~~~
incision
>...my theory is as well supported as the claim in this post.

Are you familiar with Nanex beyond this article?

~~~
mixedbit
No, I'm not, but based on the quality of this post alone, I don't value them
much. A respectful source would present a compelling argument for such a
claim, whereas this post looks like a marketing trick to attract attention.

------
petercooper
_It is worth pointing out that the EIA Natural Gas Report comes out weekly
(every Thursday at 10:30) and the market reacts within a few milliseconds._

Since it's not the report producer's job for investors' computers to rapidly
parse it, they should have some fun in phrasing and presenting the information
in different ways each time. If nothing else, it could lead to an explosion in
NLP and content parsing technology ;-)

~~~
peejaybee
Not as long as they are presenting it in JSON format.

<http://ir.eia.gov/ngs/wngsr.json>

~~~
VikingCoder
Haha - yeah, they still could: they could change the schema weekly.

------
jmix
Since it takes a while to digest the report after having seen it, chances are
that they were in possession of the report far earlier than T-400ms but waited
until they were in a time window where they knew the regulators would not come
after them.

This is how fortunes are made. By taking advantage of loopholes in the
regulatory mechanism.

~~~
homosaur
I don't know the way these reports are structured, but is it regular enough
where there's even a possibility that a bot could digest, analyze, and act on
the information there in near real time?

~~~
lotyrin
The thing is (supposedly and based on this chart) that this order was placed
faster than that, even.

Edit to clarify.

~~~
yebyen
How can you read this chart, and infer from it that the activity is due to
having already read the report?

Surely everyone knew that the report would be released at 10:30, and they had
strategies (or hedged positions) that were not as likely to be influenced by
the contents of the report as by the market forces surrounding their orders?

~~~
lotyrin
I'm not, I'm not certain of that at all.

~~~
yebyen
I don't have any idea, I don't want to sound snarky,

I read the headline and jumped to the same conclusions as everyone else.
Jumped on Twitter and re-tweeted, all I had to say was "incredible"

After having some time to think about it, all I can say is...

Can you imagine how many people would be fired already if their dots were 5
seconds on the wrong side of 10:30, instead of being 400ms early?

------
thechut
Can someone explain what this means in practice. As in how much money did
these people make or stand to make because of being 400ms early?

------
zwischenzug
Sorry if this is a question with an obvious answer, but how is the report
delivered/obtained?

~~~
crypteasy
By parsing this JSON file <http://ir.eia.gov/ngs/wngsr.json>

~~~
nisa
Do they hammer the webserver for this? Is the JSON pushed somehwere? I'm in
Europe, so numbers are probably too high, but httping results for the file
suggest this is not how it's done:

    
    
        % httping http://ir.eia.gov/ngs/wngsr.json
        PING ir.eia.gov:80 (http://ir.eia.gov/ngs/wngsr.json):
        connected to 205.254.135.25:80 (286 bytes), seq=0 time=459.92 ms 
        connected to 205.254.135.25:80 (286 bytes), seq=1 time=230.15 ms 
        connected to 205.254.135.25:80 (286 bytes), seq=2 time=232.81 ms 
        connected to 205.254.135.25:80 (286 bytes), seq=3 time=235.23 ms 
    

I would love to have some background information about how this trading works.
Are there any technical details available how this works?

~~~
jcromartie
HFT traders are actually physically positioning their systems to be as
unhindered by physics as possible. They likely have servers in DC just to
check ir.eia.gov, and definitely have servers at (or as near as possible) to
the exchange on direct lines with trading systems.

~~~
davidmr
I'd be shocked if anybody big was doing that. There are news agencies
(Reuters, Bloomberg, etc.) that get the reports early and sell exchange-
quality news feeds. People subscribe to those news feeds, and at the appointed
time, messages go out to the subscribers.

------
unreal37
This shows that not all investors acted on the information at once. But
doesn't show that the report got out early.

To the point of "private investors", if you are expecting to be able to see a
news release the second it drops, spend 10 seconds reading it, make quick
predictions on stock movements, and then buy/sell equities based on that...
computers have had you beat for a long time on that game. Find another or
automate that.

------
vincefutr23
"There is the old story about the market craze in sardine trading when the
sardines disappeared from their traditional waters in Monterey, California.
The commodity traders bid them up and the price of a can of sardines soared.
One day a buyer decided to treat himself to an expensive meal and actually
opened a can and started eating. He immediately became ill and told the seller
the sardines were no good. The seller said, "You don't understand. These are
not eating sardines, they are trading sardines."

------
homosaur
If we can't solve this problem, with the rise of machine driven microtrading,
is there really any reason to place any faith in the stock market as a private
investor?

~~~
vannevar
We could significantly alleviate the problem by limiting by law the trading
frequency. Traders ultimately depend on the law to recognize the validity of
their transactions. There is no value to society in high frequency trading.
Mandating a full second in a market that operated quite well when slow-
reacting humans conducted all the transactions should be more than sufficient.

~~~
jrockway
Will laws work? I imagine the big investment banks will just set up dark pools
in countries with favorable laws and just trade there instead. Added benefit:
no more taxes!

The solution is to realize that high-frequency traders are playing a different
game than you, even though they're on the same playing field. They do weird
things but it's probably not hurting your returns. (It wasn't HFT that
imploded the big banks, Enron, and Worldcom, right?)

~~~
vannevar
_They do weird things but it's probably not hurting your returns._

I'm not so sure. The financial system is a nonlinear dynamical system. The
hallmark of such systems is that small local perturbations can lead to very
large changes in system-wide state. High-frequency trading vastly increases
the number of small perturbations, and while most remain local, there is a
finite probability that some will percolate upward in scale. So micro-scale
trading may increase our exposure to catastrophe.

~~~
MrMan
i think that is nonsense.

------
danielweber
Every sell is someone else's buy, and every buy is someone else's trade.

If you saw a bunch of activity happening milliseconds before it should, why
would you be the other party to someone you suspect is committing fraud? You
will be the primary victim of the fraud.

~~~
vannevar
_If you saw a bunch of activity happening milliseconds before it should, why
would you be the other party to someone you suspect is committing fraud?_

If no such report had come out milliseconds later, the activity wouldn't be
suspect. How could a person (or in this case, given the timeframe, an
algorithm) possibly distinguish this spike from a 'legitimate' spike? It
doesn't make sense to blame the victim of a fraud when the victim has no way
of knowing at the time that they are being defrauded.

~~~
danielweber
I presume the report didn't just randomly come out at some random time. That
is, everyone knew exactly when it was going to come out.

~~~
vannevar
To the millisecond? I suppose it's possible. Even so, rumors drive spikes all
the time. Someone could put out a false report, committing fraud in the
reverse direction. I still doubt such a fraud could be detected at the time in
any remotely reliable way.

~~~
sirclueless
The report is a government report, in machine-readable format, that is taken
from a government server. If you wanted to put out a false report, you would
have to hack the server or the connection to the server.

~~~
vannevar
No, you'd just have to convince people you were releasing inside information,
like this kind of thing: [http://edition.cnn.com/2012/11/28/tech/web/google-
icoa-fake-...](http://edition.cnn.com/2012/11/28/tech/web/google-icoa-fake-
news/index.html) .

------
Zenst
Remind me of the film Trading Places and the Orange Juice crop reports.

A pico or nano second is in this digital age is an advantage - it is the
fairness of that advantage be it routing or some more underlying more sinister
aspect that is the real issue here.

------
CarlTheAwesome
Maybe that news was published 400ms or even a little bit earlier. Such little
difference human beings cannot notice. Those who trade early were those who
utilized A.I. systems to watch and analyze the report and trade accordingly,
400ms is long enough for big expensive machines to accomplish that task. To
add even more conspiracy into the story, you can imagine someone was paid to
publish the report just about 400ms earlier than supposed to by those who use
computers to trade, and not very many people will notice, or just like post
said feds don't even think this matters.

------
unclebucknasty
There is a lot of philosophizing over whether HFT harms or helps the market,
etc. Much of the pro camp centers around liquidity, but as someone else
mentioned, much of that liquidity is absorbed by offsetting HFT.

Rather than get lost in all of the gnarly details, however, I think it is
easier to simply look at the purpose of the market and ask whether HFT serves
or harms that purpose. IMO, it is pretty clear that it represents a hijacking
of the market's true purpose and functioning in the service of that purpose.

For example, is it helpful in setting a price which reflects true supply and
demand that we have algorithms designed specifically to manipulate the pricing
mechanism by creating artificial supply and demand. These algorithms place
phony orders, never intended for execution, but instead merely to trigger a
move from the other side. How can that possibly be helpful to such a
fundemental market mechanism as pricing?

HFT uses the market for an entirely different purpose. Anyone who defends it
must acknowledge this point and argue that the purpose is good if they wish to
defend HFT honestly. Otherwise, to couch pro HFT arguments in terms of it
being supportive of the market's true purpose and functioning is to mislead.

~~~
varelse
So what _is_ the market's true purpose?

~~~
unclebucknasty
I assume that such a simple question is intended to allow its author to make a
point. So, perhaps we can skip the dance and get right to what your point is?

Else, if you really are unaware of the market's purpose, perhaps you can spend
a little time on Google. There is much available information online that you
might find helpful in answering your question.

~~~
varelse
What perplexes me here is the enthusiastic response received for the
development and deployment of bitcoin mining hardware (for which the motive is
giving a hardware edge to people in order to make lots and lots of money)
versus the condemnation of HFT (for which the motive is giving a hardware edge
to people in order to make lots and lots of money). Both cases require a
significant outlay of resources and talent and I have respect for the work
that went into each.

And this gets at a bunch of people, including yourself, claiming that HFT
somehow defeats the "purpose of the market(tm)." But if I look this up on
Wikipedia, I get the notion that it exists to help corporations raise money
and secondarily to provide an indicator of the general economic mood. I fail
to see how HFT goes against this. That said, I'd support everything Mark Cuban
suggested short of the trading tax (because I believe (without proof) that it
would have unintended negative consequences far beyond HFT).

Mark Cuban aside, I mostly see various people who, to quote an Amazon review
of "Broken Markets: How High Frequency Trading and Predatory Practices on Wall
Street are Destroying Investor Confidence and Your Portfolio", sound like the
reviewer's grandparents explaining why they don't use computers. No hard proof
- just lots and lots of fear-mongering.

So much so that I'm starting to believe that the underlying issue here is
resentment and jealousy that they didn't pull these hacks off themselves and
walk away with the big bucks.

Finally, how is HFT worse than the "good old days(tm)" where major brokerages
all had seats on trading room floors from which they could manipulate the
bid/ask spread with impunity? I'd counter that it's actually better now that
_anyone_ capable of getting the cash and talent upfront can get into the same
game that's been played since the very beginning, except that the pace is
continually accelerating and those left out in the cold are succumbing to
future shock. Buffett knows how they'll deal with the events preceding an
actual Singularity(c) if one occurs.

But that's just my opinion, feel free to disagree.

~~~
unclebucknasty
Ah, now there's an honest response. Thanks for the thoughtful reply.

I'm not sure why the comparison between bitcoin and HFT, however. My
admittedly limited knowledge of the former informs me that the purpose of
using compute intensive operations to mine bitcoins is to prevent arbitrary
creation by limiting the number and means of creation, thereby attributing
value to bitcoins.

I will allow that my understanding could be off by some degree but, in any
case, there is no comparison. There is nothing in the design, purpose, or
function of the market that is served by accruing advantage to whomever simply
has the best hardware. Hence, I am really missing the leap that "well, if it
works for bitcoin it should work for anything". The market is designed for
economic purposes such as raising capital and allowing broader participation
in the economic output of society via investment opportunities. To see how
counterproductive HFT is to these purposes, one need only take it to the
extreme. That is, what if all transactions were performed by HFT algos? Would
the market continue to serve its purpose and function the same? Or would it
simply become some strange self-serving system that benefits an entirely
different group with entirely different agendas and objectives? How would an
IPO even work in this scenario?

Now, going back to the point in my original post, one can argue that the new
system is better or fairer, or whatever. But one cannot honestly argue that it
is the same or, worse, better serves its current purpose due to HFT.

Your argument that HFT is no worse than the good old days is a straw man, so
there's not much I need to say there. I will add though that whether spreads,
prices, etc are manipulated by major brokerages or "anyone capable of getting
the cash and talent up front", it does subvert confidence in the market and
its core mechanisms.

~~~
varelse
Let me see if I understood what you wrote...

Summarizing:

1\. I don't understand bitcoin mining but that's no obstacle to rendering a
negative judgment on a comparison between building bitcoin mining farms and
building an HFT firm.

2\. The market is designed for raising capital and HFT is deleterious to that
purpose.

So, looking at the market today, GOOG is 772, AMZN is 264, FB is 29, ZNGA is
3, and LNKD is 126. Sounds like they're raising capital to me. What am I
missing?

3\. I'll just dismiss your comparison of HFT profiting from bid/ask spreads to
the previous roll of pit bosses doing the same thing as a straw man. And then
I'll close with an unsubstantiated claim that HFT has reduced confidence in
the market because it's obvious(tm).

Now I'm assuming that because Mark Cuban has wisely decided against investing
in markets he doesn't understand that you're reasoning that _no one_
understands the market?

Which to me is as much poppycock as the belief that the market is %100
efficient 100% of the time (for if so, there would be no housing bubbles, no
dotcom booms, and more recently, no fiscal cliff chaos and witness
<http://en.wikipedia.org/wiki/Renaissance_Technologies> which whose
performance would be effectively impossible if so).

Might I propose an alternate path from futilely fleeing the event horizon of
this mini technological singularity? Instead of saying "I don't understand HFT
so it's bad" why not use the same physics currently in use to understand the
ensemble behavior of 10^23++ molecules to derive higher-level trading
strategies? It's worked in the past
(<http://en.wikipedia.org/wiki/Didier_Sornette>) and I have no reason to
believe it cannot be made to work again (which means to me that smart people
are already working on it or have already figured it out).

Finally, I'll close with evidence that HFT does reduce bid/ask spreads:
[http://www.tradersmagazine.com/news/hfts-spreads-credit-
suis...](http://www.tradersmagazine.com/news/hfts-spreads-credit-
suisse-110365-1.html) and a statement by a day trader that he's benefiting
from it:

From the final comment on
[http://www.amazon.com/review/RNIOIX766KXWG/ref=cm_cr_dp_cmt?...](http://www.amazon.com/review/RNIOIX766KXWG/ref=cm_cr_dp_cmt?ie=UTF8&ASIN=0132875241&nodeID=283155&store=books#wasThisHelpful)

" So if I can make a dollar or more per share with a .16 risk, why do I care
about HFT's? They might even help me if they decide to move the price up a few
pennys when Im close to my profit target and they may hurt me if I was a few
pennys away from the stop.(all my stops are in my head, never manually till
close to the target or stop. And even then I may sell at market if last 2
moves were upticks. Ithink HFTS hurt the other greedy scalpers and market
makers and specialists and whatever goniff's are swimming around looking for a
quick edge. But how would they hurt the buy and hold value investor who lets
say buys AAPL at 100 and knows its going way higher and doesnt sell till its
at 500. Who got hurt. Maybe a penny in slippage somewhere. Irrelevent to the
long term player and even to me, the shorter term player. Its the scalpers
that get hurt and they supposedly are providing liquidity anyway so their job
is done. How do I get affected?"

~~~
unclebucknasty
It's hard to get to the meat of your arguments because they are riddled with
logical fallacies. You seem to be trying to wedge as many in as possible.

Regarding your itemized "summary":

1\. You seem to forget that YOU are the one asserting the validity of YOUR
comparison between HFT and bitcoins. So, the onus is on YOU to prove it. So
far, you have failed. Beyond that, I simply stated that I am no bitcoin
expert. But, I believe my summary of bitcoins was sufficient and relevant to
draw the distinction. If it wasn't, then you should have explained how I
missed the mark and why that miss is relevant to your argument.

2\. Are you kidding? So the fact that there are people who have cancer but not
yet died from it proves that cancer is good for you? I mean your argument here
is literally: HFT exists; there are stocks that are doing well; therefore HFT
cannot be harmful. Wow.

3\. You introduced the straw man. Don't blame me for noticing. As for much of
the rest of the stuff you wrote, I never said it.

Ditto the Mark Cuban stuff. Not sure why you invoke him or the question of
understanding the market for that matter. Likewise with the 100% market
efficiency comment. I never referred to any of that. How many straw men are we
up to now?

You do a really good job of arguing with yourself, but you fail to address my
original point. Your talk about bid/ask spreads does at least address market
mechanisms, but even if we were to accept all of the great wonders that HFT
does for spreads at face value, that is a far cry from rebutting my OP, as
spreads are but a small part of the picture. In fact, it is so insignificant
to this discussion, let's call that one a red herring.

I also noticed that you left my example in the extreme completely untouched.
Wise decision.

In any event, you joke about my not offering evidence, but the irony is that
much of this actually is self-evident. For instance, who do you think is
sitting on the other side of the trade when HFTs profit? I won't ask you to
buy my trademarked brand of common sense this time though:
[http://www.nytimes.com/2012/12/04/business/high-speed-
trades...](http://www.nytimes.com/2012/12/04/business/high-speed-trades-hurt-
investors-a-study-says.html?ref=business)

That's an actual study, not an Amazon review from some random "day trader".
BTW, it is slightly hilarious that you referred to that review as "evidence".

------
benpbenp
A fun thought experiment. Suppose someone invents a time machine that gives
the correct price of all securities at all future points in time.

1) Would it be against current rules to trade on this information?

2) If someone did use this machine surreptitiously to their own gain, how
quickly will they approach owning 100% of everything?

3) If the entire data set of future prices were made publicly available, what
would happen to markets? I mean, exactly, what would happen to stock prices?

~~~
dclowd9901
I'll bite.

Chances are, the existence of a perfect trader would cause enormous shifts in
how a market handles trading. This trader would, by default, be barred from
trading, because her influence on the market would be staggering to the effect
that it would influence the market itself. Even Warren Buffet has to temper
things he says, so as not to accidentally push the market around.

 _But_ , assuming the FTC just decided to take a nap on the whole thing, I
imagine those shifts in trading would so quickly dilute value that you'd end
up with a system very similar to one we see now: machines doing the majority
of the trading, but rather than on algorithms, more on observing Perfect
Trader X.

~~~
alexmchale
If I were the trader with perfect information, I'd program my trading
algorithms to lose (or at least not be guarenteed to win) 40% of the time.
This would make you look merely like a incredibly fortunate trader, and not a
perfect one. Just make sure that your algorithm guarantees an X% return per
year.

------
achompas
Is there any way to subscribe to these Nanex posts? Couldn't find an RSS read
but I'd like to slowly dip into HFT, and they like a good source of trading
news.

------
parfe
Why would trade relevant data be released while trading is open?

Seems like it could screw tons of people with open orders who can't react
within seconds of new information.

~~~
tedunangst
If the release of information could affect you, don't trade until after its
released. You are free to pretend the market is closed whenever you like.

And it's not like this risk isn't there anyway. Earthquakes rarely coordinate
their timing with market hours, to name one example.

~~~
incision
>Earthquakes rarely coordinate their timing with market hours, to name one
example.

When speaking of trading and earthquakes, it's hard not to mention Nick
Leeson.

 _The beginning of the end occurred on 16 January 1995, when Leeson placed a
short straddle in the Singapore and Tokyo stock exchanges, essentially betting
that the Japanese stock market would not move significantly overnight.
However, the Kobe earthquake hit early in the morning on 17 January, sending
Asian markets, and Leeson's trading positions, into a tailspin. Leeson
attempted to recoup his losses by making a series of increasingly risky new
trades (using a Long-Long Future Arbitrage), this time betting that the Nikkei
Stock Average would make a rapid recovery. However, the recovery failed to
materialize._

 _Leeson left a note reading "I'm Sorry" and fled Singapore on 23 February.
Losses eventually reached £827 million (US$1.4 billion), twice the bank's
available trading capital. After a failed bailout attempt, Barings was
declared insolvent on 26 February._

<http://en.wikipedia.org/wiki/Nick_Leeson>

------
eksith
Delay every trade by 10 seconds. Not only does it prevent spikes like this,
but it will ensure trades done with thousands of transactions per second by
software are brought down as to not cause stupid crashes :
<http://en.wikipedia.org/wiki/2010_Flash_Crash>

------
mildavw
To make shenangins more obvious, what if 1 minute were the maximum resolution
that any trade could happen? Say, every order gets a random number of seconds
between 0 and 60 added to it before it is executed. Or even longer. What would
happen if everyone gets 10 minutes to digest any news?

~~~
gamegoblin
This sounds like the type of regulation that people outside of an industry put
on the industry with good intentions but really no idea what the consequences
would be.

Consider a company who holds a press conference announcing something huge
(either positive or negative). Anyone wanting to buy or sell in this tiny
window pretty much gets shafted by such a system.

~~~
svachalek
I think that's exactly the point. So the whole world gets to trade after
they've digested the news, not the guy with the fastest computer or the
shortest wire to the exchange.

~~~
varelse
And why exactly _shouldn't_ the guy who's invested in the best hardware have
an edge? Should perhaps we also mandate that all software engineers use
exactly the same 2004-era Acer Pentium 4 laptop so that any difference in
productivity is strictly due to programming skills?

~~~
jellicle
Because the purpose of the stock market is not to give the guy with the best
hardware an "edge".

~~~
prodigal_erik
This. The entire reason for having this market is to reward making _better_
decisions about allocating society's resources, not the _same_ decisions
imperceptibly faster. The market's clearing system is flawed in ways that
reward huge misinvestments in solving the wrong problem.

~~~
varelse
So I looked at Wikipedia:
[http://en.wikipedia.org/wiki/Stock_market#Function_and_purpo...](http://en.wikipedia.org/wiki/Stock_market#Function_and_purpose)

Didn't see anything about rewarding better decisions about allocating
society's resources. I did see a lot about raising capital for corporations
and for providing an indicator of the general mood of the economy though.

The stock market is effectively a giant casino for pros. If you're not one,
you're just plain dumb if you try to play their game by their rules.

And of course, with the sort of thinking you're putting forth, we should
obviously _OUTLAW_ that 66 GH/s bitcoin hardware. It's so unfair to everyone
else who can only afford a cluster of 8 AMD 7970s, no?

------
kjackson2012
If the report is accessed via HTTP, I wouldn't be surprised if the clocks on
the government server are off by a few ms, so all the HFTs that are pounding
the URL are trying to get access to the data. The first one who got it made
their trades before everyone else.

------
SanjayUttam
Off-topic; what was used to generate those graphs?

~~~
pmh
It looks like it's JTools:
<http://www.nanex.net/CodingNxCore/JTools/JTools.html>

However, I don't know what JTools uses internally to generate the graphs.

------
pbhjpbhj
What is the benefit of trading over ms resolutions. What problem is it
solving?

Wouldn't trade be more efficient if it were lock stepped - say one trade per
hour (per day?): you agree your trade and the exchange processes it on the
hour.

What would be lost that benefits the pseudo-capitalism of these systems by
having such a regime. How would this negatively impact production.

~~~
lmm
>What is the benefit of trading over ms resolutions. What problem is it
solving?

The problem it's solving is that people want to do it. Exchanges on which they
can't are outcompeted by exchanges on which they can.

>Wouldn't trade be more efficient if it were lock stepped - say one trade per
hour (per day?):

No.

>you agree your trade and the exchange processes it on the hour.

You "agree" your trade outside the exchange? What do you think an exchange is?

>What would be lost that benefits the pseudo-capitalism of these systems by
having such a regime. How would this negatively impact production.

Spreads would get wider, i.e. more investor cash would get creamed off by
middlemen, leaving less for the productive companies it was actually invested
in.

~~~
pbhjpbhj
> _The problem it's solving is that people want to do it._ //

As a generality I'd say that's false. Why buy in to a system that makes a tiny
proportion of the populous vastly wealthy only because those people already
are wealthy.

> _You "agree" your trade outside the exchange?_ //

No, hence the conjunction. Perhaps "you issue a bid or offer" would have been
better?

> _Spreads would get wider_ //

OK, can you give a reason why that happens and why it leaves more with
middlemen. At the moment a change in price so transient as to pass in
milliseconds gets exploited to extract value from the system. In essence
surely a greater spread means more to lose from selling quickly, that would
appear to tend to stabilise.

Presumably the middle-men here are primarily market-makers.

To this layman it appears that traders currently extract value by reacting
fastest to the rapid variations. That the value extracted far exceeds the
original notion of guaranteeing trades in order to provide liquidity. Slow the
variations and there are less opportunities to extract capital. Again this
seems like it would tend to money being invested for longer term growth.

Thanks.

~~~
lmm
>As a generality I'd say that's false. Why buy in to a system

Because it gives you a better price. There are two sides to any trade; people
go to the HFTs because their prices are better than anyone else's. If they
weren't providing value, no-one else would trade with them.

> that makes a tiny proportion of the populous vastly wealthy only because
> those people already are wealthy.

HFT has greatly democratized market-making; in the old days stockbroking was
an old-boy's network, virtually impossible for new participants to enter.
Nowadays, three or four blokes with computers and one investor can start a new
trading firm, and many of the biggest HFT players started that way.

>OK, can you give a reason why that happens and why it leaves more with
middlemen

Because there's a higher risk. If I offer to buy microsoft for $50 but that
offer has to stay out there for an hour, and news of a lawsuit comes out 20
minutes later, I'm going to lose lots of money. So the middlemen need to be
able to bear that risk, they need much bigger capital reserves, and they can't
afford to offer the penny spreads we see nowadays (because they need to make a
greater profit to offer the same return on their bigger capital reserve).

> At the moment a change in price so transient as to pass in milliseconds gets
> exploited to extract value from the system.

Where's the value being "extracted" from? Certainly not from a fundamentals
trader, who's getting the best possible price with the smallest possible
spread (sadly, regulations require shares to be sold in increments of $0.01
and no smaller, so there's always $0.005/share to be made on every trade,
which by modern standards is absolutely huge - and is why the HFT guys are
willing to spend so much on low latency to maximize their chances of getting
that $0.005/share. But the fundamentals trader always pays exactly
$0.005/share; the HFTs are just fighting among themselves for who gets it).

~~~
pbhjpbhj
> _Where's the value being "extracted" from?_ //

> _there's always $0.005/share to be made on every trade_ //

I [clearly] don't know enough to know if you're exactly right, but lower the
latency and increase the trades and there you have it. Aren't bids and offers
listed in pips (like $1.4032).

As I see it production, processing, administration, etc. are the only value
inputs. When a 400ms glitch extracts something of the order 1e5 USD the value
that money represents comes from those inputs. Yes liquidity is an
administrative input but the way the system is set up trades appear to extract
far greater amounts of money than their value to society; of course that money
comes from other investors, but money is not value, the value the money
ineffectively represents is brought to the system by those said inputs.

The problem appears to be that those in a position to rectify the aberration
are too busy getting rich off it to care.

Thanks for your input(!) and education.

As an aside, do you [or anyone] know of something along the lines of a (FOSS)
toy stock market program, something that allows one to model a market futz
with parameters on trades (like changing minimum stock increments, or fixing
time periods) and see the effects graphically. Like an ecological modelling
program I suppose.

~~~
lmm
>lower the latency and increase the trades and there you have it

It is true that lowering the latency and more to the point narrowing the
spreads increases trade volume, which I kind of glossed over, but again if you
lower your margins and sell more of your product you're not extracting value
but creating it. Fundamentals traders are (hopefully) buying and selling
shares for good reason; helping them do it quicker and more cheaply is a good
thing.

>Aren't bids and offers listed in pips (like $1.4032)

AIUI there's an exception for low-value shares, but most are required to be
sold in increments of $0.01. (Of course that only applies to stocks traded on
public exchanges, which is by no means all or even most HFT activity)

>As I see it production, processing, administration, etc. are the only value
inputs. When a 400ms glitch extracts something of the order 1e5 USD the value
that money represents comes from those inputs. Yes liquidity is an
administrative input but the way the system is set up trades appear to extract
far greater amounts of money than their value to society; of course that money
comes from other investors, but money is not value, the value the money
ineffectively represents is brought to the system by those said inputs.

You're right that there's a kind of "tragedy of the commons" going on; because
there's that massive $0.005/share to be made and free competition on latency
to be the company to get it, the competing companies naturally push harder and
harder until they're all spending $0.004999/share on FPGA programming and the
smartest employees they can find to get that $0.005. But it is at least kind
of circumscribed; it's that fixed (ish) quantity of money getting wasted,
nothing more.

>The problem appears to be that those in a position to rectify the aberration
are too busy getting rich off it to care.

Maybe. I've seen elsewhere in these comments that large institutions are now
trading directly with HFT players like GETCO and Knight, because they can
offer better prices (narrower spreads - less than $0.01) there than they can
publicly. These guys are now doing their own trade crossing, effectively
acting as a private exchange - and competition between these private exchanges
will make the spreads narrower still, and reduce the rents the market makers
get. Of course, there are all the downsides of a private exchange - without a
public order book it's a shark pool in the same way as the bond market.

------
fnordfnordfnord
For those who are skeptical of Nanex, or want some background on some of the
problems with HFT, here is a Dec 2012 analysis by Credit Suisse. HFT
Measurement Detection and Response

<http://www.scribd.com/doc/116761218/CS-HFT-DETECTION>

------
fideloper
I really hate that I can't find articles when I need to. However, I have read
that investment companies build datacenters as close to internet hubs as
possible, to take advantage of the extra milliseconds gained.

If I recall correctly, this is notable in Manhattan.

~~~
sseveran
Actually northern New Jersey.

------
dfc
I always get a kick out of reading these NANEX reports. Reading a new report
usually means I spend 30 or 40 minutes filling in the gaps (giant) in my
knowledge. Are there any other organizations that put out similar quality
reports?

------
Raz0rblade
BTW 400ms rules out that someone had a closer internet connection. As in 400ms
light travels 119917 kilometers.

Someone sure new something earlier. But those high frequency traders don't go
by names, and so never can be put to normal justice.

------
geuis
Would putting a minimum time barrier on HFT systems help? Basically say, you
can't make any trades faster than 100ms. You still get the benefit of fast HFT
systems, but not the insane microsecond stuff going on now.

------
camkego
Those are some heavy graphs, but I wonder what was the spread before the
release of the news? Traders buying early on the news might be dealing with a
big spread as everybody is expecting the news to come out.

------
swah
Is there some kind of wargame where we could all write little algorithms to
trade/compete with one another for a couple minutes, and then check the
winners/learn strategies?

------
arjn
So is this evidence of insider-trading ? Why would they not prosecute or at
least investigate ?

~~~
peterwwillis
Somebody has to provide evidence (and, you know, a perpetrator) to a
prosecutor first. But there's no real "investigation team", save maybe FINRA.

If FINRA detects something they can refer an investigation to the S.E.C. or
other law enforcement body. But they don't exactly have a stellar track
record. And nobody even has to listen to them; they just gather evidence and
make recommendations. It's very much a dog with no teeth.

------
lefinita
They really should learn data visualization to make it more readable and
beautiful.

------
iso-8859-1
What would happen if that crucial number was published as a riddle instead?

~~~
mcherm
The high-frequency traders would pay $1,000,000 /year salaries to high-speed
riddle solvers who trained themselves on Jeopardy shows for high-speed buzzer
pressing.

------
undrcvr
if the market was architected to be fair, all data releases would happen at
midnight london time. it just isn't and will never be...

------
MrMan
you guys remember the uproar that the telegraph caused in the trading
community? the street has never recovered.

------
joering2
any idea what kind of profit we talking about here? Perhaps this is just not
worth for Federalies to pursue...

~~~
lostlogin
Is cost/return a factor in these things? I'd hope not. If expense stops
prosecution, the offender can learn where the cut off lies, and just offend
below the benchmark. Somewhat like the way I drive (speed camera goes at 10kmh
above limit, I drive at just under this).

------
madaxe
So, that initial downwards spike at -400ms that immediately kicks back up to
the halfway point?

Those can only possibly be pre-programmed strategies. From the big boys. 'Cos
you can't stack 'em exchange side last I checked, if they're price dependent.
Which means they have a gloriously low-latency-close-to-the-exchange-link.

Which means this news was leaked well before the event.

~~~
sirclueless
Sure you can "stack" them. If you place a sell order larger than any of the
buy orders on the exchange, then what you will see is a "stack" of executed
orders beginning with the highest-priced buy order and working down to cheaper
ones.

In any case, the bounce back looks to happen over ~100 ms or so, which is
plenty of time for anyone's strategy to catch up. I don't see how you can
conclude there is some big player who knew "well before the event."

------
martinced
What if someone got the natural gas report was earlier than 400 ms but used a
computer whose clock set 400ms too early?

It then would be the regular "insider" knowledge trying to pretend business is
as usual. But he got noticed because instead of starting abusing the system at
10:30:000 he unknowingly started abusing it at 10:29:400.

Petty amounts of money anyway and nobody is forcing to gamble in the stock
market ; )

