
This Is What One Half Second of High Speed Trading Looks Like - sytelus
http://blogs.smithsonianmag.com/smartnews/2013/05/this-is-what-one-half-second-of-high-speed-trading-looks-like/
======
snowwrestler
One time I printed out all the code--at that time ASP, HTML, CSS, and
Javascript--that got executed for one page load of the homepage of my
employer's primary website. I taped the pieces of paper end-to-end and hung
them on my wall.

It made a great conversation piece with my coworkers. Non-technical folks
(most of them) were astounded that so much text was being interpretted and
executed every single time they loaded the homepage (in less than a second).

What was the point? It helped set a certain emotional tone to our
conversations. Folks found it a lot harder to demand their project be done
"right now, it's easy" when directly confronted with the complexity behind a
single page load.

Likewise, stories that highlight the immense speed and complexity of high
frequency trading help set an emotional tone of alienation and fear. It helps
trigger the same reflex Frankenstein and The Terminator played off of: "maybe
we're unleashing technological forces that no one can control."

But of course if you are well and truly versed in a technology, that feeling
goes away. I knew that our crappy brochure website was not very complicated as
websites go.

~~~
mverwijs
As an ops-guy (interestingly enough, at an HFT company), I used graphviz to
draw out a diagram of all the network interfaces and routes that were being
used when trading on a single exchange for one single instrument.

It was a huge, big, complex, ugly thing. We laughed. Good times.

~~~
aangjie
That sounds cool.... I would love to see it though understand the security
concerns..

------
minimax
If there was a similar video showing all of the electronic sensors of a modern
airliner flowing into the central flight control computer would it would
engender the same sort of trepidation about air travel? Should we be afraid of
something just because it's "complicated" to someone unfamiliar with the state
of the art?

There are a lot of valid criticisms you can make about modern financial
markets, but claiming that we are using computers to do things better than we
did 10 years ago is a bad thing isn't something I'd expect to see here on HN.

~~~
m0th87
I don't see any comments here critiquing HFT simply because it's complicated.
Rather, they seem related to whether it's actually a net-positive for society.

~~~
crdoconnor
Every description of an HFT algo seems to fall into one of three categories:

* Acting on public information, milliseconds before anybody else - which helps everybody else about as much as insider trading does.

* Acting on trading information gleaned from "bid stuffing" (making and canceling orders really quickly) - which helps everybody else about as much as front running does.

* Detecting the presence of large trades about to go through or going through and capitalizing on the price movements they cause - i.e. imposing a tax others making large trades in the stock market.

In return, they flood the market with liquidity when it is least needed and
withdraw it suddenly when it is (e.g. when prices are diving).

There's literally nothing to like about HFT.

~~~
rgarcia
_Acting on public information, milliseconds before anybody else - which helps
everybody else about as much as insider trading does._

"They can do it faster, therefore it must be illegal."

 _Acting on trading information gleaned from "bid stuffing" (making and
canceling orders really quickly) - which helps everybody else about as much as
front running does._

The only people that notice this behavior are...other people capable of doing
this behavior. Why should it matter to you, the long-term investor, the
trading tactics computers employ at the millisecond scale?

 _Detecting the presence of large trades about to go through or going through
and capitalizing on the price movements they cause - i.e. imposing a tax
others making large trades in the stock market_

This happens in any market at any timescale. Everyone capitalizes on trends.

~~~
CJefferson
You are replying to the wrong points.The question is not are these things
illegal, or does everyone do them.

question is, do they benefit the market as a whole? Traders do not have a
right to perform HFT, and if it destabilise markets, we can take steps to
reduce it (of course, there is then the issue of how to take steps without
causing different problems)

~~~
dchichkov
How else would you connect the exchanges? Or you don't want major exchanges to
be connected, and say, if you dump a lot of stock at one exchange you don't
want that information to propagate to other exchanges?

Benefits of having HFT traders, is that at the cost of having them around you
can have fair and level ground. So if you come to one exchange to do a
transaction, you don't have to think much about exchange intrinsics, like
liquidity available there, fee structure, etc. Because these differences are
being arbitraged away by HFT traders.

And again, what is the alternative? Exchanges that are connected by human
traders making phone calls?

~~~
nhaehnle
The real question is: Why are there multiple exchanges for the same papers to
begin with?

In the case of stocks, for example, there has to be one ultimately
authoritative copy of who owns which stocks anyway. So why not just do all the
trading in the place where the authoritative copy is stored?

That would certainly seem to be more efficient from the perspective of
minimizing the overall social cost.

~~~
aitkin
The reason there are multiple exchanges is the same reason you can buy the
same pair of shoes at 10 different places: it is competition.

Also, what is the social cost you are referring to? There are a lot of people
here saying that HFT doesn't provide "social value". Does it somehow provide
less social value than traders screaming at each other in the pits? Similar
types of trading always existed, except now it's done by computers. For what
it's worth, HFT provides well-paying jobs for a lot developers and ops people
and often is very supportive of the engineering community.

~~~
nhaehnle
If you try to design a system of publicly traded companies from first
principles, then the purpose of exchanges should be to provide a service _to
publicly traded companies_. That service would be maintaining the
authoritative record of who owns how many shares of the company.

Given this notion, competition is a good reason for the _existence of_
multiple exchanges, because it means companies can change which exchanges
their shares are traded over. However, it is _not_ a good justification for
the fact that trading of the same stock happens on multiple exchanges.

As for the social cost, you do realize that HFT does not come for free to
society. At a minimum, society must somehow pay for those well-paying jobs to
developers. And what do those developers give back to society? Before you
answer, please consider whether you should distinguish between _algorithmic
trading_ and HFT. I do believe that algorithmic trading is useful to society
(because it can do the job that screaming traders used to do better and
cheaper), but the subset of HFT is not useful.

The only things anyone ever seems to be able to answer for HFT two-fold: one,
that stock prices change faster, and two, that the spread is smaller. But one
point one, nobody in the real economy cares about that, and to point two, the
decrease in spread also doesn't matter to anybody in the real economy, because
what you really care about there is the fluctuation of the share price over a
larger timescale such as one full day. I have not seen any evidence that this
fluctuation is affected by HFT in any way.

~~~
aitkin
_If you try to design a system of publicly traded companies from first
principles, then the purpose of exchanges should be to provide a service to
publicly traded companies. That service would be maintaining the authoritative
record of who owns how many shares of the company._

I don't think there's any reason to think that exchanges' main purpose for
existence is to provide service to publicly traded companies. Exchanges are
just a store, and their purpose is to buy/sell stuff and make a profit. So, in
that sense, having multiple exchanges selling the same stock is the same as
having multiple bike stores selling the same bike...it's competition and there
aren't many downsides to it.

 _As for the social cost, you do realize that HFT does not come for free to
society. At a minimum, society must somehow pay for those well-paying jobs to
developers. And what do those developers give back to society? Before you
answer, please consider whether you should distinguish between algorithmic
trading and HFT. I do believe that algorithmic trading is useful to society
(because it can do the job that screaming traders used to do better and
cheaper), but the subset of HFT is not useful._

Nothing comes for free to society. Funding a never-ending number of failing
start-ups doesn't come free to society either. What is that giving back to
society? Your implication is that HFT trading is somehow _taking_ something
from society that it doesn't deserve. What is it taking? Also, in terms of
making distinctions: HFT is the superset and AT is the subset, not the other
way around. All AT shops are HFT shops, but there are actually quite a few HFT
shops that don't run algorithms and instead run based on inputs controlled by
traders.

 _The only things anyone ever seems to be able to answer for HFT two-fold:
one, that stock prices change faster, and two, that the spread is smaller._
This is the crux of the whole thing. Why does anyone have to _answer_ for HFT?
This is implying that HFT is hurting people however you haven't not given any
proof of this (besides that it doesn't _add value_ ). If anything, I think
HFT's biggest crime (as someone else has mentioned) is that sucks some really
smart people into working on problems of a rather limited scope.

------
nohuck13
Two things I take away from this

1) Cool use of infographic :)

2) The little dots represent price changes, not trades.

FWIW, some more context around this would be-

All the major exchanges stream their best prices to all the other exchanges.

This is required because US exchanges must execute trades at the National Best
Bid or Offer, or NBBO [1].

So how do you track the NBBO. In theory, by having all the exchanges send
their best bids and offers to a central Securities Information Processor, or
SIP (the 12:00 box) which then disseminates them (which you can see by the
waves of stuff it sends out)

In practice, it's slow to go through the SIP on every price movement and the
exchanges want to have the best possible information, so the exchanges mostly
stream prices directly from the other exchanges (hence the fanout effect on
every price movement)

Most of these prices movements don't result in trades, they just represent
somebody entering or leaving the market at the BBO.

This feels like a distinction that could have been made better, rather than
just saying "yeah, this stuff's complicated."

[1] <http://en.wikipedia.org/wiki/Regulation_NMS>

edit: formatting / citing

~~~
0x0
That's a nice explanation of the graphics.

I'm still having a very hard time understanding what kinds of events would
cause so many rapid price changes. So many dots in just half a second -- even
when time-stretched to 5 minutes! Are those price changes meaningful at all?

------
jhales
All comments so far are negative, but I fail to see how this video shows
anything other than incredible engineering.

The negativity seems to be a result of previous bias.

~~~
SeanDav
It is incredible engineering and it has incredible downsides, not the least of
which is the ability to destroy a market in seconds, due to a blip in
information, with cascading domino effects.

~~~
jrockway
Which market has been destroyed in seconds?

~~~
skylan_q
None. This provides profit opportunities to those who will "heal" the price.

Also, exchanges will revert back trades that happen over these flash crashes.

~~~
omegant
How long before someone (a hacker no doubt) games the system to create a
selling spiral, just waiting to take profit of the rebound. If it´s well
planned the one causing the dive is not going to be the one taking the profit
(at least that it can be proved).

~~~
jzwinck
We need not wait for that to happen. Such shenanigans have been observed and
documented already, such as in the book "Reminiscences of a Stock Operator."
It was written in 1923, and worth reading today.

------
pak
If we eventually decide that all of this ridiculously fast bid/offer blitzing
by our HFT overlords is a destabilization risk to the economy [1], couldn't we
start taxing these transactions, or the bids/offers themselves? This level of
high-frequency noise being thrown into the market can't reasonably go on
increasing forever, right... at some point we just bury ourselves in a never-
ending war of inscrutable, difficult to predict machine trading algorithms,
and nobody will be able to translate market movements into anything linkable
to reality, right [2]? Shouldn't either market operators or the government at
some point in the future start dialing back the traffic with penalties on the
sheer number of actions taken?

... or is that just an insane non-starter, since it affects too many rich
people in the finance sector and their colleagues rotating through the current
regulatory structure?

I think it's an interesting long-term problem faced by society to decide
whether we are OK with investment markets becoming completely impossible for
"normal", human investors to navigate. It's already rapidly going that way--
the capability to profit is already wildly tilted toward firms that act at HFT
volume. If a poker site became overrun by bots to the extent that it was no
longer fun for humans to play, they would fight back against the bots (as most
do to some degree). But with investments... we may be getting to the point
where the average day trader needs a 3000U server farm to compete effectively,
and then I question whether we've lost sight of why we subsidize and spend
money regulating that market at all. It's like a chess league where the
grandmasters get to use computers as aids, and the newcomers have to play
blindfolded.

[1] Nothing catastrophic has happened yet, but there have been a few flash
crashes that have necessitated trading to be shut down, and having to "roll
back" trades is never perfectly equitable and only decreases confidence in a
market.

[2] Some of those flash crashes were traced back to one overzealous seller
that threw HFT's for a loop (read
<http://en.wikipedia.org/wiki/2010_Flash_Crash> \-- it is quite interesting).
I shudder to think of a day when the market fluctuates wildly and the data is
literally too complicated for regulators to trace it back to any identifiable
cause. That day is closer than anybody in finance is likely to acknowledge.

~~~
dchichkov
> we may be getting to the point where the average day trader needs a 3000U
> server farm to compete effectively

Yes. That's actually a bit of a problem. It is perfectly all right that day
traders are being out-competed by HFT firms, by itself that's fine. Humans
replaced by computer, nothing bad about that. What is not good, is that a
_successful_ entry into arbitrage market nowadays require a lot of
infrastructure and technical skills.

That increased barrier _reduces_ a number of players on the market. That
reduces diversity. And that increases risk. That's bad.

A simple example how that increases risk. Imagine that a single HFT firm had
managed to out-compete all others. And now owns perfect infrastructure with
nanosecond level latencies, speed-of-light channels between the exchanges and
trading algos without any bugs and teams of quants maintaining them. Obviously
no day trader will be able to compete with teams of PhDs. And no new newcomer
HFT firm will be able to compete. But as a result diversity of day traders
will have decreased to exactly one. And any glitch in the operation of that
firm (sorry, a box that was trading SPY just crashed) will be creating flash-
crash artifacts without easy resolutions to them.

So yes, an increased entry barrier into day trading is bit of a problem.

~~~
kasey_junk
The cost of entry into the markets has gone down not gone up. It is orders of
magnitude cheaper to set up an electronic trading operation than it was to be
on the floor.

Further, successful entry into arbitrage markets always required specialist
infrastructure and skills. They used to be good old boy networks, brightly
colored vests, and arcane hand signals. Now it is programming and networking.

In the previous world there was not "NASA" effect at all. Now at least the
skills can be transfered to other industries.

------
mortehu
It seems that the counter at the bottom counts all updates, and ends up at
2380, so the overall rate would be something like 4760/second, or 17M/hour.
Note that the rate has slowed down considerably before .25 seconds have
passed, and this was recorded within 10 minutes after NYSE opened, so I
suspect this is a bit above the average rate. Also, these are bids and offers,
not actual trades.

------
wallflower
Reminds me of:

"The following animated GIF chronicles the rise of the HFT Algo Machines from
January 2007 through January 2012."

<http://www.nanex.net/aqck/2804.HTML>

------
nnq
Light bulb: tax all tradings with a tax percent increasing with the number of
transactions per second, in a continuous way (even for less than 1 t/s) that
would not make any difference for regular traders (and have a tweakable taxing
formula that could regulate a market's stability and sensitivity to
information by letting more or less HFT happen by setting the 100% taxation
limit - the HFTs will have all incentives to stop trading when they are taxed
100% or beyond - i.e. when they are "fined")!

~~~
jellicle
Any kind of small transaction tax on trades would eliminate the worst excesses
of HFT. It doesn't have to be complex at all.

------
Strilanc
Why is the video showing redundant details? It looks like for each event it
shows a packet going to every server.

If they arranged the machines on the horizontal axis, time on the vertical
axis, and showed 'packet sent' events it would look much simpler. Maybe simple
enough to add in details about the trades, like dollar values, instead of
pretty expanding explosions.

~~~
cbhl
I believe that part of the actual infrastructure is you send the same message
along multiple physical cables; one to each exchange. You also have to account
for the delay that the message takes to be received by each exchange. (The
speed of light isn't "instant" when you're working on the ms level.)

------
pxlpshr
All in the name of "providing liquidity"...

~~~
purplelobster
As someone who's not too familiar with economics, how is HFT supposed to be
good for the economy?

~~~
dllthomas
If you want to buy/sell at a price, and there's no one ready to sell/buy at
that price, a speculator (of any stripe) coming in and making that deal with
you means you got the price you asked for. Maybe you could have got better -
that's what the speculator is hoping - but you're no longer taking that risk.
Statistically, the money going to the trader is similar to an insurance
premium then.

That's all "in principle" - whether the details work out right is a much
deeper question.

~~~
joezydeco
Aren't these speculators constantly entering and withdrawing these orders from
the market on the order of microseconds?

It's not like the order book for Johnson and Johnson is completely empty and
there is absolutely _nobody_ out there willing to sell shares. If I understand
it right, the HFT machines are poking numbers around the current price trying
to step ahead of a regular order before it comes in.

~~~
skylan_q
Yes. That's what it does. If you, as a human, try to trade like this, the
computers will destroy you. That's why people shouldn't trade like computers
and they should use their experience and judgment to guide them.

~~~
joezydeco
So you're implying we should all be complacent and let HFT machines take a
small parasitic profit off of every stock transaction humans make?

~~~
skylan_q
Complacent? Get in on HFT if you feel that it's making it unfair for human
traders.

~~~
joezydeco
I'm speaking for the majority of people that can't afford racks of servers and
custom-laid transatlantic fiberoptic lines. I'm speaking about typical
americans that have their money in 401(k) accounts or maybe buy stocks from an
online broker to try and fund their kids' college accounts.

These people are all being scraped, very slowly but very surely, by HFT.
That's the point I'm making.

At least we all admit that the modern stock market has absolutely no
connection to the economy or to the interests of long-term investing or
capital building. It's just a very large and very legal casino.

~~~
skylan_q
_I'm speaking for the majority of people that can't afford racks of servers
and custom-laid transatlantic fiberoptic lines. I'm speaking about typical
americans that have their money in 401(k) accounts or maybe buy stocks from an
online broker to try and fund their kids' college accounts._

Barriers to entry. Cost of business, etc... It's really not a concern to
people who actually understand HFT and trade in their own way despite. They
generally really don't mind the added liquidity.

 _These people are all being scraped, very slowly but very surely, by HFT.
That's the point I'm making._

Buy in at $40. Sell at $80. Where does the computer scrape money from the
transactions? It doesn't. Like I said, you're only losing if you're trying to
trade like a computer. It's like trying to out-robot a robot on an assembly
line.

 _At least we all admit that the modern stock market has absolutely no
connection to the economy or to the interests of long-term investing or
capital building. It's just a very large and very legal casino._

I disagree to the first part, and I think the second bit is an
oversimplification.

It obviously matters, or else people wouldn't be upset by HFT or claim that
they're losing money to it. We also have to consider the fact that pretty much
all money comes through the bond and stock markets at exchanges. The exchanges
are price arbitration centers for the economy. Paying attention to them
matters because the exchanges and what happens there matters.

A casino is a place you go to lose money. It's all games and the odds are
stacked against you. Everything we do in life is a gamble. An exchange is a
place for people to make informed decisions about the prices of goods and
financial instruments. Everyone loses and wins some.

I do relate to the long-term investing/capital building. The central banks
keep printing money which funnels into the bond and stock markets. This in
turn signals an abundance of actual economic capital, which there isn't. With
the communication that there is plenty of excess capital for the long term, we
(and especially people dealing in finance) act as if it is the case and start
consuming capital. Eventually, facts catch up and we get some sort of terrible
economic downturn. As a result of the long-term economic irregularity, actors
act for the short-term because the long term is just too undependable.

------
dirkgently
>You already have welcomed your robot overlords, and they’re building our
financial system.

Enough said.

------
smoorman1024
To get the perspective of different participants in the market watch this
video from The Milken Institute featuring Bart Chilton (regulator), Terry
Duffy (exchange), Jamil Nazarali (retail execution), and Lou Salkind
(proprietary trading).

They address a lot of the common criticisms of modern markets discussed here.
It provides a good perspective of what people within the industry see as the
current state and future of trading.

[http://www.youtube.com/watch?v=icNGvgd27-A&feature=youtu...](http://www.youtube.com/watch?v=icNGvgd27-A&feature=youtu.be)

------
philhippus
The radiolab podcast is well worth a listen for a bit of layman's insight into
HFT. The conclusion at the end makes it clear what the implications are.

------
resu
And the exchanges are making a killing off of this volume. You can thank the
SEC and FINRA for this.

------
washedup
We will keep cutting up time into smaller and smaller chunks until we cannot
go any further.

------
snake_plissken
HFT is used for many things, but it makes one task almost trivial: stop
hunting.

------
robmcm
Isn't the, "one" in the title redundant?

Would you say two half seconds rather than just a second?

------
lifeformed
Just how profitable is HFT anyways? What kind of returns do HFT traders get?

~~~
kasey_junk
Depends. It is mostly a zero sum game, so all told $0. Of course, some people
make lots and some people make -lots.

~~~
HockeyPlayer
While intraday trading as a whole is zero-sum, the HFTs are taking profits
from other traders, not from each other. HFTs as a group are profitable.

~~~
adambratt
A good amount of HFTs fail. I've heard numbers ranging from 70%-90% of new
HFTs fail within their first 2 years. Very similar to startups when you think
about it.

Also, HFTs are most definitely taking profits from each other. In fact, in a
lot of cases the faster, more successful HFTs kill off the new ones to the
market. Not to mention there are a good number of predatory HFTs who exist
solely to game other HFTs

