
High-Speed Trading Firm Deleted Some Code by Accident - fahimulhaq
http://www.bloombergview.com/articles/2015-09-30/high-speed-trading-firm-deleted-some-code-by-accident
======
smoothgrammer
In finance you'll find low accountability for developers. Basically finance is
where great developers go to get paid a lot to become horrible developers.
Very few companies in finance are pushing the bar on quality code since they
culturally cannot accept the cost of that. It doesn't make sense until it is
too late and by then their code base is too entrenched.

I've seen this at countless finance companies, and from interviewing people
across the entire spectrum of finance. It is a rare corner of the industry
where you'll find quality engineering. Usually everyone in a finance company
thinks they are amazing engineers, and are just plain wrong.

My advice: work smarter, not harder.

~~~
mdpopescu
I can confirm this. I just wish I had known about it before I accepted my
current contract...

~~~
smoothgrammer
Get out while you can.

------
herbig
The best part of this article is the feature graphic, that of a castle with
the caption:

THIS IS A TOWER. PHOTOGRAPHER: DAMIEN MEYER/AFP/GETTY IMAGES

The article mentions "Tower Research".

Is the image/caption a joke, auto-generated by content, or just plain
laziness?

~~~
davmre
It's a joke, consistent with Matt Levine's writing style. See for example the
image and caption for his article on Panda Express:
[http://www.bloombergview.com/articles/2015-09-24/panda-
expre...](http://www.bloombergview.com/articles/2015-09-24/panda-express-s-
life-coach-might-have-overshared-a-bit)

------
x0054
Every time I read stories about high speed trading, I have to wonder what is
actually the point of high speed trading to the society. The point of a free
and fast stock market is to provide monetary liquidity and stabilize the
economy by shortening the feedback loop. But something tells me that our
society and economy derives zero benefit from nanosecond resolution trading
vs. trading with say 1 minute resolution. Meanwhile, the market is exposed to
wild swings which might be smoothed over or even prevented if markets worked a
little slower.

~~~
auntienomen
Faster trading allows market makers to operate with smaller capital
investments. This lowers the barrier to entry in the market making business,
creating more competition. More competition reduces the costs of trading. (And
not just in theory. Market making used to be exclusively the domain of a few
big players. It's not anymore.) Reduced cost of trading matters to almost
everyone who's involved in the markets. Passive index funds, for example, have
to trade to reinvest profits and to rebalance their composition. That's why
Vanguard -- the most evangelical of the passive fund managers -- firmly
defends HFT.

You may ask, well, why do we need market makers at all? Well, you don't have
to use them. You have other options for trading. Call up your broker and ask.
What you'll discover is that these other options have far larger transaction
costs.

A lot of the complaints about HFT I've seen on Hacker News seem to think that
no one should get paid for market making. I don't think that's possible. (And
if it is, I'd love to know how.) Transaction costs are an inescapable part of
trading; ultimately, you're faced with the choice between tilting your hand to
other traders or paying some intermediary to take on risk for you. Generally,
it costs less to choose the second option.

~~~
x0054
How does HFT help market making? Would a market that operates with a 1 second
resolution be inherently unable to provide the same levels of liquidity that
we enjoy today?

You argument is that Vanguard is a market maker, Vanguard likes HFT, thus HFT
must be good. That doesn't really follow at all. Vanguard could be arguing in
favor of HFT for all kinds of reasons, one of them could be that they have the
money to rent server space right next to the main exchanges, and pay for the
ludicrously low latency connections to the exchanges, so they benefit from HFT
by being able to be quicker than others, and they don't want to loose there
edge.

~~~
mrchicity
No. Vanguard is a large participant in the markets because it manages funds
for long-term investors. Markets have bid and offer prices where you can
instantaneously sell or buy shares, and these bids and offers are often
provided by professional market-makers. Before automation, these were humans
on an exchange floor. They could only trade a single stock at a time, hence
the name "specialist", since they specialized in that issue. Volumes were also
lower. To make enough to earn a living, that human on the exchange floor had
to charge a big spread between buy and sell prices. For a long-term investor,
this spread is an additional friction cost that drags on their returns when
rebalancing portfolios.

Now a small team of researchers can make markets on many securities at once.
They don't have to make as much money on each, so the spreads get smaller.
Additionally, instead of a monopolistic specialist, these professional traders
compete to win transactions by making better prices (higher bids and lower
offers) than others. Many market makers try to keep a neutral "book" of
positions, so if they sell some GS they may bid higher in other bank stocks to
reduce their exposure to market risk. The faster a market maker can update his
quotes or hedge when conditions change, the less spread he needs to charge.

[http://meanderful.blogspot.com.au/2013/01/hfts-dirty-
little-...](http://meanderful.blogspot.com.au/2013/01/hfts-dirty-little-
secret-where-money.html)

------
keyle
Doesn't matter how much money is involved or the quality of a team, someone
somewhere will always do an oopsie. Maybe that particular 'code' should have
had a little comment such as // used by ISO stuff do not delete

Commenting tricky / non-obvious code is well recommended.

~~~
deckar01
Some unit tests would be nice. Maybe even a special folder full of tests that
know when laws have been broken.

~~~
azinman2
Probably exists now, and not before.

Writing this software would be very scary for me -- I'm confident in my work
for the most part but I'm only human. To then let a "for-loop" go wild making
thousands of trades per second on the entire market with real money against
other high frequency traders no less.. man... you gotta be really really
really really sure of your code!

~~~
smoothgrammer
It is not about confidence. It is about ownership and being a good engineer. A
good engineer would have cared about the code base and ensured that a test
would fail, a use case would be linked to that failure, and someone would see
that they have now violated a regulation.

Then if the regulation changed, someone would grep through the use cases, and
see what behavior is occuring/ how to modify the code base to cope with the
change in regulations (this occurs a lot). Of course most finance firms don't
do this and actually violate regulations all the time. There is literally a
gold mine for the SEC/FINRA to go after, but the government is fairly
incompetent at mining through the data and holding companies accountable. The
reason being that they don't pay enough to hire the talent that would uncover
the violations.

~~~
tbrownaw
Real engineers are like true Scotsmen, because you can never have enough
process.

In this case, _of course_ tests really do prove the absence of bugs rather
than just (not) proving their presence.

I really don't like the whole "unit tests lets you not worry about breaking
things" view. That's not confidence, it's _overconfidence_. And it doesn't
even mostly work for things where "correct" is more fuzzy than "eventually
produces this exact output".

~~~
smoothgrammer
It is confidence. It is the right tool for the job.

What would you propose as an alternative?

------
jwatte
Acceptance testing fail! When changing code, and especially code that has
consequences, doing so without tests is a bad idea. Doing so to protocol code
without conformance tests is doubly bad.

~~~
smoothgrammer
Sadly, this is the norm in finance. You would be shocked at the shittastic
state of the companies that dominate in this industry. They are literally a
house of cards waiting for a strong gust of wind.

------
tbrownaw
...I'm missing something here. How do they not know how much they've asked to
trade and not heard back on yet? Wouldn't they _have_ to track that, to not
try to trade the same exact shares the same exact way a bunch of times? Is
their local system maybe just more synchronous than I'm assuming would be
possible?

------
Sander_Marechal
Can someone explain _why_ the SEC rules insist you _must_ buy from the
exchange with the lowest price first? It appears to me to be a rule just to
keep the small exchanges alive.

~~~
kylebrown
Otherwise a broker would just route mom n' pop's orders to the exchange that
pays the biggest bribe to the broker (and screw mom n' pop).

~~~
mrchicity
Yes but for proprietary traders using their own capital, it doesn't make a lot
of sense to force them to trade in a particular way since executing sub-
optimally can only impact their own performance. There are legitimate reasons
not to trade the lowest price market (maybe you believe the quote is slow,
want to trade bigger size all at once, don't want to risk signaling to the
market): [http://www.hudson-
trading.com/static/files/RegNMShudsonriver...](http://www.hudson-
trading.com/static/files/RegNMShudsonriver081304.pdf)

And retail brokers already do what you suggest. All their marketable flow gets
sold to off-exchange market makers, while their limit order flow gets routed
to exchanges that pay the highest liquidity rebates, Reg NMS just mandates
what price it can trade at:
[http://news.indiana.edu/releases/iu/2014/02/study-of-
potenti...](http://news.indiana.edu/releases/iu/2014/02/study-of-potential-
conflict-of-interest-in-routing-limit-orders%20.shtml)

------
fahimulhaq
On a related note, Knight Capital lost $440 million due to a software bug a
few years ago.

[http://www.bloomberg.com/news/articles/2012-08-02/knight-
has...](http://www.bloomberg.com/news/articles/2012-08-02/knight-has-all-
hands-on-deck-after-440-million-bug)

~~~
HolyLampshade
Blew up the whole firm actually, and was more closely related to change
management policies and systems monitoring.

The SEC order there is an interesting read for anyone involved in large scale
transactional systems:
[https://www.sec.gov/litigation/admin/2013/34-70694.pdf](https://www.sec.gov/litigation/admin/2013/34-70694.pdf)

------
fahimulhaq
Given hindsight, it's always easier to ask for more tests or code comments. I
guess the rule of thumb is that if the code is there, it most probably is
there for a reason.

When something like this happens, it's always a series of oversights.

1\. Someone forgot to add comments or test cases while writing the code (or
perhaps he wrote it but people are not running the test cases before commits).

2\. Someone else thought that the code is dead and deleted it (and skipped the
test cases if there were some covering that particular scenario).

3\. Either the person deleting the code didn't wait for the code review or
code reviewers missed that as well. (If code reviews are non-existent, it's a
disaster waiting to happen).

So in the hindsight, write comments, write test cases, run test cases and do
code reviews carefully.

------
pjc50
Move fast. Break things. Pay millions of dollars in SEC fines.

------
osullivj
A change like this making it through to production is indicative of a test
failure, either automated or human. I have had the luxury of substantial human
test teams on trading systems before 2008, and it was great. They wouldn't let
something like this through...

------
JimDash2145
Who doesn't notice a huge deletion in one changeset? This seems to be a
convenient excuse. The SEC report.

[http://www.sec.gov/litigation/admin/2015/34-76029.pdf](http://www.sec.gov/litigation/admin/2015/34-76029.pdf)

------
Tinyyy
I don't know much about stocks trading, can someone ELI5? Thanks.

~~~
smoothgrammer
Basically imagine you want to buy apples from 3 different fruit stands in
different parts of a city.

Fruit stand 1 is the cheapest, fruit stand 2 is the 2nd cheapest, and fruit
stand 3 is the third cheapest.

You want to buy all the apples out there.

The rule is that you have to go to the cheapest fruit stand first, and exhaust
their supply before moving onto the next cheapest fruit stand. If you want to
go to ALL the fruit stands at the same time you need to tell them you are an
'OK' guy and are really buying fruit everywhere.

Well, someone deleted code and ignored one of the fruit stands. This violated
a rule and cost the fruit buyer a lot of money in penalties.

~~~
bdonlan
Why does the SEC care that you bought shares at a price higher than the
cheapest? How does that hurt anyone other than yourself?

~~~
chrisbennet
If I ask my broker to buy some stock, I want it at the lowest price. My
broker, left to his own devices, may prefer to buy at a higher price because
it is more profitable _for him_.

This rule seems designed to protect the consumer when the consumer and the
stock brokers goals are not aligned.

~~~
tbrownaw
The article says they only do proprietary trading, which I thought means they
only use their own money. I guess that's rare enough that the rules apply
regardless in order to simplify things?

~~~
neomantra
The rules apply to the whole system -- notably exchanges, the exchange
"members", and broker-dealers. Latour is a broker-dealer (BD) and thus have to
play by the rules, even if they don't have customers.

If they weren't a BD, their BD (which is required to ultimately reach the
markets) would be on the hook for letting this happen. ISOs are something that
get a lot of compliance/regulatory scrutiny and most BDs don't let their
customers use them. If they are allowed, there's a lot of
reporting/transparency/surveillance to ensure they are issued properly. And as
you can see from this fine, there are a lot of rules and corner cases one must
cover and apparently even the best firms mess it up one way or another.

Latour's previous fine (read Matt Levine's continually excellent articles
about this stuff) was for "Net Capital Requirements" which are designed to
keep customer money safe, particularly in the presence of other customers who
might blow up ... of course Latour has no customers, but it is still the
rules. [Although IMO this one was a grey area where Latour probably had better
models than what the rules required, but since they didn't match the rules
properly they got fined.]

------
zeeshanm
Another reason code needs to be idiomatic.

