
Wall Street’s Big Banks Are Waging a Technological Arms Race - chollida1
https://www.bloomberg.com/news/features/2018-04-05/wall-street-s-big-banks-are-waging-an-all-out-technological-arms
======
chollida1
> Bam. Bam. Bam. Dummy trade signals that were supposed to stay within the
> company’s electronic systems broke loose and slammed into computers at the
> New York Stock Exchange’s options markets. So many orders crashed through
> that by 8:44 a.m., safeguards within Goldman Sachs sprang into action,
> severing the connection between the company and the exchanges.

Anyone whose built a trading system lives with this type of fear on a daily
basis.

Write your risk system first and your algo second.

> The cause? A coder had mistakenly programmed a router to send placeholder
> bids as live orders. If not for the good graces of the options exchanges,
> the bank would have lost $500 million, according to the U.S. Securities and
> Exchange Commission. Cancellations and price adjustments reduced that to $38
> million

And this is at the top of my things that are unfair in the markets. Knight
Capital wasn't bailed out by the exchanges for the same thing and it
bankrupted them. Goldman with its alumni every where.... was.

~~~
ransom1538
Imagine coming back from lunch, you start to unwrap your sandwich at your
desk. Then, your boss walks by and throws your sandwich across the room and
explains while you were out your bug caused 440 million dollars of erroneous
trades - and the company now no longer exists.

[https://dealbook.nytimes.com/2012/08/02/knight-capital-
says-...](https://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-
mishap-cost-it-440-million/)

~~~
alecco
> your bug

Knight had a terrible software delivery process and a lot of deadline-oriented
pressure on techies.

~~~
lagadu
After it passes review and all stages of testing it's everyone's bug.

~~~
taneq
Exactly. You don't just do code reviews and run tests to catch bugs. You also
do code reviews and run tests so that, when something _does_ slip through,
it's no one person's fault.

~~~
mooreds
... and this everyone's responsibility.

Makes total sense. It's almost as if everyone at a company should be on the
same team :)

------
thomaspaine
> The cause? A coder had mistakenly programmed a router to send placeholder
> bids as live orders. If not for the good graces of the options exchanges,
> the bank would have lost $500 million, according to the U.S. Securities and
> Exchange Commission. Cancellations and price adjustments reduced that to $38
> million

It's interesting to read the full SEC report because it's a little more
complicated than that:
[https://www.sec.gov/litigation/admin/2015/34-75331.pdf](https://www.sec.gov/litigation/admin/2015/34-75331.pdf)

It basically outlines a cascade of failure in controls, bad configuration
defaults, and poor SDLC. In particular:

>In addition, the firm’s operation and management of its electronic “circuit
breakers” did not effectively block the erroneous orders sent on August 20.
These circuit breakers existed to prevent erroneous orders by halting all
message traffic to the exchanges once that traffic had exceeded a certain
rate. However, on August 20, the firm’s control personnel repeatedly lifted
the circuit breakers blocks between 8:44 a.m. and 9:32 a.m., thereby
permitting additional erroneous orders to be sent to the exchanges. Before
lifting the circuit breaker blocks, the control personnel did not obtain
authorization from the responsible technology employees, as required under
written firm policies.

>The firm’s policies relating to the manual “lifting” of those circuit
breakers were not disseminated to or fully understood by the employees
responsible for deciding when the circuit breakers should be lifted, and,
prior to August 20, 2013, GSCO personnel had lifted circuit breaker blocks
shortly after learning of the block and while still investigating the cause of
the circuit breaker trip.

From what I remember, this circuit breaker was notorious for raising so many
false positives that control personnel just got used to lifting it without
thinking.

~~~
marcc
That’s really a terrible place to put people in. From what this says, the SEC
put part of the blame on the employees who lifted the circuit breakers without
approval, and also state that employees didn’t know that they needed approval?

 _Before lifting the circuit breaker blocks, the control personnel did not
obtain authorization from the responsible technology employees, as required
under written firm policies._

 _The firm’s policies relating to the manual “lifting” of those circuit
breakers were not disseminated..._

------
nostromo
> By the time the trades were blocked for the last time, less than an hour
> after they began, Goldman Sachs executed orders to sell more than 1.5
> million options contracts for $1. The cause? A coder had mistakenly
> programmed a router to send placeholder bids as live orders. If not for the
> good graces of the options exchanges, the bank would have lost $500 million,
> according to the U.S. Securities and Exchange Commission. Cancellations and
> price adjustments reduced that to $38 million.

It sure does seem like there are two sets of rules on Wall Street. I doubt any
small trader would be able to reduce their losses 90+% after errantly
submitting a bunch of live options orders.

~~~
dnomad
There are N^2 sets of rules. At the end of the day you have two counterparties
engaged in voluntary exchange. Everything is negotiable, anything is possible
-- you just have to ask or pay. What people don't seem to grasp about modern
finance is how mind boggingly complex and dynamic it really is. The idea that
a tier one investment bank and a small trader should be held to the same
"rules" is so wondrously silly... it's hard to describe but there ought to be
a word for it.

The only thing really annoying here is that all this wackiness is backstopped
by the Federal government. Nobody has any doubt that at the end of the day
they will step in to save GS if everything really goes sideways. And that's
probably a good thing. But it's very strange that the big banks, which are
essentially quasi-state actors at this point, get to keep so much of their
profits when the public is bearing so much of the risk.

This is the real difference between China and America: in China the government
does everything in its power to make sure its state-owned enterprises succeed
and then it takes its pound of flesh. In America the government does
everything in its power to make sure its state-owned enterprises succeed and
then it gives them a huge tax cut to make the shareholders that much richer.
It's a bold move, let's see how it plays out.

~~~
feiofh398r39rty
>The idea that a tier one investment bank and a small trader should be held to
the same "rules" is so wondrously silly... it's hard to describe but there
ought to be a word for it.

Just like the idea that the aristocracy should be accountable to the same laws
as commoners, right?

~~~
dx034
No but if GS asks a counterparty (with whom they have billions in other
contracts) to cancel contracts as a result of a system error, the counterparty
will probably agree (if they don't lose as a result) knowing that GS will next
time do the same. With a small party, this could just be an attempt to undo a
bad deal.

I think what the poster meant is that deals are just contracts and can be
amended if both parties agree. For large banks with longstanding
relationships, that is naturally much easier than for a trader that no one
knows.

------
olivermarks
I highly recommend Rana Faroohar's book 'makers and takers' to add some
perspective to this insanity... [https://www.amazon.com/Makers-Takers-Finance-
American-Busine...](https://www.amazon.com/Makers-Takers-Finance-American-
Business/dp/0553447238)

------
Zenst
One upside is such environments help nurture future technologies as they will
happily pay millions for something just a bit faster than what they already
have and more so before others have it. Hence it is avenues of business like
this that have help feed certain industries over the years, from networking to
FPGA's. Another avenue of financing technology which we all know is the
military.

As for the whole stock market being driven faster and faster in precision from
seconds to nanoseconds and picoseconds, I've always questioned the need to
drive towards a faster processing and if the stock market only updated once
every minute, it would sure help in curtailing many ineeded aspects that are
creeping into stock trading.

------
osrec
I personally feel we need to change the way exchanges operate. Too much
emphasis exists on speed of execution. We must consider alternative auction
formats where speed of execution is not the only criteria for matching an
order, as it gives rise to masses of spam generated by a thundering herd of
traders as they all try and fulfill the same opportunity.

~~~
mifreewil
This may be dated info, but IEX (exchange) purposely ran longer than necessary
fiber to slow down the speed of automated trading.

~~~
dsacco
Did they really? It's incredible how wasteful some organizations can be in
pursuit of a misguided sense of "fairness."

~~~
kasey_junk
They did it for marketing not fairness. It added 350 mikes and some of their
orders bypassed it.

Having a ‘speedbump’ was the trendy thing a bit ago in exchanges with several
different ones opening with several different versions.

~~~
ngz00
Still a thing, that and the tick size.

------
parm289
Why aren’t these firms insuring against this risk? Seems like any other risk
that can be managed - pay premiums and so that a third party would fund any
erroneous losses.

~~~
kasey_junk
Because they don't have to? They don't suffer from the risk...

Clearing frequently act like a form of insurance for smaller firms, up to and
including asking for killswitches in your trading engines and audits of your
risk procedures.

But big firms clear themselves so don't have that. Theoretically the internal
risk team is responsible for that but those teams are frequently undergunned.

~~~
parm289
What about a firm like Knight?

~~~
kasey_junk
Cleared themselves. I'd be shocked if any company in the world would act as
counterparty to a trade that hedged 'operational risk at Knight scale'.

One of the thing many of us in the industry at the time commented on, was how
little was done to bail Knight out. Other than the 'oligarchy' argument that
states that the old timers hated them (they did) the argument I subscribed to
was, they weren't systematically important. Their entire function in the
market could be taken over quickly by someone else with little disruption and
largely they were the only ones that lost money on that day (not really but to
an approximate).

~~~
AnimalMuppet
"If you just hurt yourself, nobody cares. If hurting yourself hurts others,
then other people care." That's pretty cold-blooded and brutal, but it does
make some sense...

------
goldmanguy
Except it wasn't the machines that caused the issue, it was a bug (and lack of
following the controls in place) caused by Lucas Renick, of Goldman Sachs, who
was fired on the spot and remained unemployed (though still on the Goldman
payroll, in order to avoid a lawsuit) for many months until he joined Bank of
America and later Morgan Stanley

------
lawrenceyan
Places like D.E. Shaw, Two Sigma, Jane Street, Renaissance, Citadel, etc...
were doing this long before any of these banks. To call it innovation is a bit
of a misnomer in my opinion, more of a catching up with the times really.

------
notyourday
s/$TITLE/Salaries of people with clue are going up, up and up/g

------
senatorobama
How do I get a job in this field?

~~~
paulie_a
Step one: sell your soul

~~~
poster123
High-frequency trading reduces transaction costs and therefore reduces the
cost of capital, encouraging investment.

