
Oil crash busted broker’s computers and inflicted big losses - wclax04
https://www.bloomberg.com/news/articles/2020-05-08/oil-crash-busted-a-broker-s-computers-and-inflicted-huge-losses
======
haltingproblem
_“Five days, including the weekend, with the coronavirus going on and a
complex system where we have to make many changes, was not a sufficient amount
of time,” he said. “The idea we could have bugs is not, in my mind, a
surprise.” He also acknowledged the error in the margin model Interactive
Brokers used that day.....We have called the CFTC and complained bitterly,”
Peterffy said. “It appears the exchanges are going scot-free.”_

Thomas Peterffy must think we are idiots. Anyone who trades commodity
contracts for any period of time knows that the real cost of the contract is
the _actual cost of the commodity - storage costs_. When storage costs spike
and the actually commodity spot costs go down, the future will become
negative!

One way to get a handle on storage costs is think of them being inversely
proportional to the _value density_. The higher the value density, e.g. gold
the less the storage costs. Oil is not so dense so storage costs matter.
Financial instruments like the Treasury Bonds and the S&P futures contract
have zero storage costs. Storage cost is of-course different than carry cost
(the cost of funding your long position).

On another aside, I have known folks who have worked at IB in the past, and
their systems absolutely suck dead goats. Huge masses of legacy C++ code with
poor testing. Most of these brokerage firms have legacy code base from the 90s
that is poorly understood. They also have nonexistent organizational quotient
around code validation, correctness and testing their risk models. A futures
margin model is not something one can whip up over a weekend but a good CS
undergraduate can program one over a couple months.

Sorry for the IB customers but I have _zero_ sympathy for IB or should I say
negative ;)

~~~
Bluecobra
Peterffy is talking out of his ass, they had more than 5 days. CME's current
specifications for order entry/market data has been in place for years now.
Whoever was writing their software on their backend failed to read the
documentation.

[https://www.cmegroup.com/confluence/display/EPICSANDBOX/Posi...](https://www.cmegroup.com/confluence/display/EPICSANDBOX/Positive%2C+Zero+and+Negative+Futures+Trade+and+Settlement+Prices)

~~~
rossdavidh
While I don't doubt that what you say is true, systems which, by spec, should
be able to handle a given situation, that then years after they were written
finally are called upon to handle that situation, fail more often than not.

The CEO also said they would pay $100million to clean up, which is not chump
change, even for them I bet.

I'm not saying they were blameless, but I think most systems which have never
had a given variable go negative, that years after they were written finally
have that variable go negative, will have problems.

~~~
futevolei
Nat gas prices went negative pretty famously (in the biz) around 2010 or so. I
give everyone a pass based on what you said until then. After that no free
passes. IB screwed up.

------
elliekelly
> Peterffy said there’s a problem with how exchanges design their contracts
> because the trading dries up as they near expiration. The May oil futures
> contract -- the one that went negative -- expired the day after the historic
> plunge, so most of the market had moved to trading the June contract, which
> expires May 19 and currently trades around $24 a barrel.

> “That’s how it’s possible for these contracts to go absolutely crazy and
> close at a price that has no economic justification,” Peterffy said. “The
> issue is whose responsibility is this?”

It’s pretty well known that commodity futures contracts are a game of hot
potato for most investors as the expiration date approaches. But the
Interactive Brokers CEO doesn’t offer an alternative solution. How would the
contracts be structured instead that would avoid this? I don’t see how it
would be possible.

~~~
wpietri
> It’s pretty well known that commodity futures contracts are a game of hot
> potato

Very much so. I wrote software for financial traders in the 1990s, and I heard
tell of a couple of clerks (in this context, sort of "trader intern") who
thought they were smart enough to do a little commodity metal trading on the
side. However, they didn't quite understand the details of contract
expiration, and so supposedly they ended up with 25,000 pounds of copper
delivered to somebody's parents house. Oops!

~~~
simonh
There are contracts that specify physical delivery, but they also specify the
warehouse or storage facility the commodity will be delivered to. The buyer
can either collect the commodity or pay the storage facility to hold it for
them. It's not likely it would be delivered to someone's house unexpectedly,
because someone would need to pay for the additional cost of transportation.

~~~
quickthrowman
A source to back you up, this is precisely how it works. The CME link goes
into more detail on how physical delivery works for base metals.

Other physically settled commodity contract have different delivery locations,
/CL (WTI crude) delivery takes place at a terminal and storage facility in
Cushing, OK.

[https://www.cmegroup.com/education/courses/introduction-
to-b...](https://www.cmegroup.com/education/courses/introduction-to-base-
metals/what-is-the-base-metals-delivery-process.html)

~~~
beerandt
Oil and gas also have the benefit of published prices for most pipeline
transport, although you still need a buyer/terminal that's physically
connected to the network to take delivery.

------
AndrewBissell
Incorrectly assuming values can never be negative is an all-too-common
occurrence in trading and financial software. In 2012 Swedish stock futures
trading was suspended for a time because their matching engine used an
unsigned type for order quantities and someone submitted an order with a
negative value which wrapped around to 4 billion:
[https://www.reuters.com/article/markets-sweden-
bug/swedish-s...](https://www.reuters.com/article/markets-sweden-bug/swedish-
stock-futures-market-problem-caused-by-mega-order-glitch-
idUSL5E8MTB4I20121129)

Interactive Brokers' software is usually very solid. I'm surprised they
weren't ready to handle this, the possibility of oil going negative had been
discussed for some time before it happened.

~~~
H8crilA
I particularly dislike the attitude of the CEO which shows he either doesn't
know or pretends that he doesn't know how such contracts work. Please read the
contract specs and educate yourself a little, Mr. Peterffy, they're public and
free.

> _Peterffy said there’s a problem with how exchanges design their contracts
> because the trading dries up as they near expiration. The May oil futures
> contract -- the one that went negative -- expired the day after the historic
> plunge, so most of the market had moved to trading the June contract, which
> expires May 19 and currently trades around $24 a barrel._

> _“That’s how it’s possible for these contracts to go absolutely crazy and
> close at a price that has no economic justification,” Peterffy said. “The
> issue is whose responsibility is this?”_

Nobody ever promised neither liquidity nor positivity of prices, it is the
fault of the brokerage, plain and simple. Thankfully $100M is something that
IBKR can take on their books (they have $3B of cash according to the latest
filling).

~~~
AndrewBissell
Yes this is a very CYA statement. IB has been in this business for a long
time, Peterffy knows very well that expiring futures contracts can experience
all kinds of liquidity problems and large swings in price.

~~~
azinman2
But if it’s never happened before, and you’re some random programmer making
this stuff, I’m not surprised there are all kinds of assumptions being made
when choosing data types and validating inputs...

------
mathgenius
If you look at the CME website using the wayback machine, you can clearly see
a hi and low limit for the price. There is no high limit, but the low limit is
set at "0.01". [1] This is kind of weird, because the snapshot is pulling
current market prices, and still showing an incorrect hi/lo limit. It looks
like they have since removed the hi/lo display: [2]. Someone should take a
screenshot of this.

[1]
[https://web.archive.org/web/20200117115242/https://www.cmegr...](https://web.archive.org/web/20200117115242/https://www.cmegroup.com/trading/energy/crude-
oil/light-sweet-crude.html)

[2] [https://www.cmegroup.com/trading/energy/crude-oil/light-
swee...](https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-
crude.html)

------
nromiun
> Its software couldn’t cope with that pesky minus sign, even though it was
> always technically possible -- though this was an outlandish idea before the
> pandemic -- for the crude market to go upside down.

Wow, just wow. They are handling millions (billions?) of dollars every day and
couldn't find the time to test that they can just DISPLAY a minus sign. That's
insane.

And it's not even that outlandish. People were saying it could go into the
negative weeks before it happened. This just seems like pure laziness. Just
pretend everything is business as usual.

~~~
EdwardDiego
Makes me wonder about their testing - any decent tester is going to throw a
negative number in as a matter of course.

I have a gut feeling that this bug was flagged and closed as WONTDO because
"that'll never happen"

~~~
dmurray
Just as likely it was a feature. If you really think "that'll never happen",
then the right thing to do is explicitly ignore negative prices or refuse to
send orders at negative prices, and that's exactly the kind of failsafe I'd
want to have in my trading software, since you can get spurious prices for all
kinds of reasons.

It turned out the assumption was wrong and yeah, you should remove the logic
handling that once it's evident futures may go negative, and you should have a
process capable of making that change with only a day or two notice. But being
robust the rest of the time at the expense of mishandling a once-in-fifty-
years event is not in itself a bug.

If you disagree, should a trading system also allow negative prices for
precious metals futures? Stocks? Currencies? Options? Bonds? Futures on stocks
or bonds? I can contemplate all of those trading negative in extraordinary,
contrived scenarios but I would design systems today not to trade them at
negative prices.

~~~
EdwardDiego
But the fact that they didn't display negative prices to customers indicates
that they were explicitly ignoring them in a very very bad way. I'm suspecting
an abs() function was involved.

------
codenesium
Wasn't the original purpose of futures to let farmers and others lock in
prices early so they can mitigate risk? Speculation on futures seems dumb if
you have no intention of taking delivery.

~~~
ashtonkem
Taking physical delivery and hedging are not one and the same. It’s entirely
possible to use a cash settled future to hedge against market movements; the
farmer sells at a steep loss, but their cash settled wheat futures offset a
large percentage of the loss on a cash basis.

The distinction you’re looking for here is those who are speculating on market
prices, vs. those who are hedging against market prices. If you use or sell
oil in large amounts, it makes sense to use futures to stabilize your downside
risk, even if those futures are cash settled.

That being said, I think that cash settled futures make purely speculative
trading much easier, so you’d have a good point if that’s what you were
heading towards.

~~~
codenesium
That is the point I was trying to make. Basically if you're in the business of
producing or buying and selling the commodity the futures are for you. If
you're just speculating how does that help anybody? I guess you could make the
argument that having more eyes on the market means there is more information
so the price is a better reflection of the true value.

~~~
wpietri
It's a good question. Most markets have (or at least used to; I've been out of
this a long time) a special class of speculator called a market-maker; they
get discounted fees in exchange for providing liquidity. That way a seller
doesn't have to wait around for a buyer.

Speculators in theory also provide liquidity, and in theory also contribute to
keeping the prices "correct" (meaning at levels that reflect what's known).
But it's not like any given speculator personally cares about that; they're
just looking to gamble and win. Last I heard there was reasonable evidence
that more gets spent on speculation than is delivered in benefits to the
economy. So you suspicion is not unwarranted.

But there's another class of people who neither create nor consume the
product, but have some financial interest in something related. E.g., suppose
you sell farm equipment. You know that if wheat farmers have a bad year,
you'll have a bad year, because they will put off buying your new tractors. To
even things out, you can use wheat derivatives to essentially buy insurance on
wheat prices. If wheat prices are normal, you lose a little money. But if they
fall through the floor, you make money, hopefully counterbalancing the income
from lost sales.

When this activity is significant enough, it can lead to the creation of
synthetic commodities. E.g., if you are in the snowplowing business, maybe you
want to insure against winters being abnormally snowy. You could maybe do
something with a fuel oil future. But that's kinda tenuous. Instead now you
can just trade weather futures:
[https://www.cmegroup.com/trading/weather/](https://www.cmegroup.com/trading/weather/)

~~~
ashtonkem
Minor nit: market makers typically aren’t there to speculate, they’re there to
provide liquidity and make small profits per trade doing so. Most market
makers try their best to trade down to no position overnight, since they don’t
want to be long or short in anything.

~~~
wpietri
Thanks! Maybe the market maker I worked for was atypical. We definitely had
opinions on where the market was going and made good money from that,
especially in times of high volatility.

~~~
twic
It's hard to make money by pure market-making, so it's common for market-
makers to also do some amount of speculation.

After 2008, banks were banned from speculating (ish) [1], but were allowed to
do market-making. It's common for market-making desks to do speculative trades
under cover of market-making activity. It's hard to conclusively prove that
any given trade is speculation rather than market-making (which involves
hedging), so they generally get away with it.

[1]
[https://en.wikipedia.org/wiki/Volcker_Rule](https://en.wikipedia.org/wiki/Volcker_Rule)

~~~
wpietri
Ah, yes. We were purely a proprietary trading firm. As my boss explained it,
the traders who started the company brought a pile of money with them, and it
was our job to make it a bigger pile of money. It was in some ways very pure;
for my first year there we didn't even have the company name on the door. We
already knew where the place was, and money spent on frivolities just meant
less money to trade with.

------
flint
I'm surprised IB let speculators trade in a contract going to delivery. I
worked as a risk manager in a commodity trading firm and only hedgers
qualified to take delivery were permitted to hold contracts going to delivery.

~~~
everlost
Wasn't this a cash settled contract?

~~~
whatok
No.

~~~
mathgenius
It may have been ICE's crude contract, which is cash settled:
[https://www.theice.com/products/213/WTI-Crude-
Futures](https://www.theice.com/products/213/WTI-Crude-Futures)

"The West Texas Intermediate Light Sweet Crude Oil futures contract is cash
settled against the prevailing market price for US light sweet crude."

~~~
whatok
Sure, but a majority of trading was done on CME futures. In either case, you
still ran into the issue of negative prices. USO only recently started
explicitly stating that they would also potentially invest in ICE futures
alongside CME futures.

------
socrates1998
If you don't know what is going on, then why on earth would you risk so much
money?

IB fucked up, no doubt, but these idiots are trading shit they know nothing
about.

Don't trade on margin.

~~~
SilasX
Well, yeah, it's reasonable to expect investors to appropriately researching
something before buying, and be cognizant of the risk that it could crash.

It's not reasonable for them to expect to deal with a platform that
misrepresents the state of the market and executes trades at a non-market
price, as was happening here. (It was telling them the oil futures still had a
positive price when it was negative, and making them pay on that basis.)

~~~
james-mcelwain
> Well, yeah, it's reasonable to expect investors to appropriately researching
> something before buying, and be cognizant of the risk that it could crash.

Is it? Isn't this the entire reason risk management departments exist?

------
0x8BADF00D
It's really odd that this bug occured, as IB has no issue pricing credit
spreads with negative values. Must be an issue specific to commodities futures
contracts. I wonder what data types they were using.

~~~
derriz
I’ve written code (a year or so ago) against the IB API and it seemed clear to
me that the API was a thinish skin over multiple backend systems. The feed you
get for products from different markets (even different futures markets) was
different - the population of fields in price and trade feed was wildly
inconsistent. I’m guessing each market is accessed in by a different IB
system. Negative prices are a feature of some massively traded futures -
interest rate futures for example - but can effectively never occur for index
futures or the like.

~~~
centimeter
From what I've seen, almost all financial companies developed a bunch of
systems for different security types (which all have different rules and edge
cases) independently, and only tried to tie them together as time went on.
It's a recipe for a lot of confusion and inconsistency.

There are tons of opportunities in finance to make short-sighted proclamations
like "the number of futures in this kind of contract is always 100" or "this
type of security can't go negative", and have it be true at the time, but
false 5 years later when they add a new type of contract.

~~~
0x8BADF00D
Kind of orthogonal, but I’ve often faced pushback from Product Managers when
trying to future proof a feature that I’m going to ship. The pushback is
usually “oh, there’s no way our customers will ever need that!”. Only until
many years later will it blow up in your face, with costly consequences.

~~~
centimeter
I have had similar experiences. Even if the future-proofing is very
inexpensive, they'd rather waste more work un-doing the future-proofing than
the total amount of work that went into it in the first place, because it's
"unnecessary".

------
supernova87a
For futures on physical deliverable objects (well, I guess most futures are
such, but anyway) -- would volatility and speculation be dampened/improved if
the clearinghouse forced everyone (or the seller) participating in a trade to
certify that they had rights to the specific thing being traded? Could
actually produce the contract -- like the oil producer is certified to have
<xyz> barrels allowed to be sold?

My notion is that if much of the trading (and it can be shown by futures
volumes) cannot possibly be on actual physically deliverable quantities, then
most must be "speculation" by people who cannot actually produce the asset.
Would this be a help to stabilize the market?

I know it all has to get settled in the end by the expiration date, but just
an idea.

~~~
kasey_junk
It likely will settle the markets by driving spreads through the roof and
costing buyers/sellers of the physical items.

The underlying theme here is that speculators are not providing utility to the
market, which is wrong. They are bridging time and risk.

------
LatteLazy
Fuck IB and this “trader”, idiots should lose their shirts, that’s one of the
intended outcomes of an efficient market.

That said, there is an issue here with futures contracts: you can get very
very large leverage when the price is near zero. This is the real issue with
instruments that can negative price and just like their are “circuit breakers”
in markets for big price swings, there should be breakers for entering the
“near zero” range.

~~~
mostlymeme
An efficient market that don't let traders operate? Did you even read the
article?

Futures contracts that CAN BECOME NEGATIVE don't let large leverage when price
is near zero, that's NOT TRUE. Future contracts margin is calculated with
SPAN, and if it's done correctly, it considers the scenarios where price can
go below 0.

~~~
LatteLazy
>Futures contracts that CAN BECOME NEGATIVE don't let large leverage when
price is near zero, that's NOT TRUE

It clearly is. If I can buy a contract for 1c, I can get 100,000 contracts for
1000usd. Then if the price rises of falls by 1usd, I'm up/down 100,000
dollars. Can you think of any retail product with that sort of leverage?

That's the danger of putting zero in a denominator.

~~~
CrazyStat
You're ignoring the margin requirements.

Assuming your broker is competent (which the broker in this story is not),
they won't let you buy 100,000 contracts for 1000 USD unless you can cover the
potential downside, which would be of millions (possibly tens of millions) of
dollars. So you can't actually leverage $1000 into going up for down $100,000;
you would be tying up $X,000,000 to go up for down $100,000.

Unfortunately IB was not competent and they did not calculate the margin
requirements correctly, which allowed their customers to leverage themselves
improperly.

~~~
scurvy
Sounds like IB hired the Robin Hood team.

------
danans
Hopefully, one of the things that we price in going forward is the volatility
in the price of oil vs alternatives, especially those that can be produced
domestically in a way decoupled from international events. That includes
natural gas and especially renewables. Remember that just a 6 years ago, oil
was north of $100/barrel [1], and recently it's close to zero. Wind on the
other hand has had a steadily reducing LCOE[2][3]. Solar's LCOE is also
steadily reducing [3].

Volatility has a cost. With oil, it's one that the US and other countries hae
historically tried to dampen with various industrial and political methods
(the national strategic oil reserve, military/political "influence" on foreign
oil producers, subsidies for domestic production), but seems like the current
situation is beyond those methods' ability to control.

1\. [https://www.macrotrends.net/1369/crude-oil-price-history-
cha...](https://www.macrotrends.net/1369/crude-oil-price-history-chart)

2\.
[https://www.energy.gov/sites/prod/files/2015/08/f25/LCOE.pdf](https://www.energy.gov/sites/prod/files/2015/08/f25/LCOE.pdf)

3.[https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#...](https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#/media/File:US_projected_cost_of_wind_power.png)

4\.
[https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#...](https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#/media/File:EU-
PV-LCOE-Projection.png)

------
kragen
I've written real-time trading systems trading on IB. Their client library is
a real pile of shit, so it's not surprising to hear that their internal
systems are too. Reassuring to hear that they're going to eat the US$100M loss
themselves instead of letting their customers have it. (Not that they were
going to get US$9M out of this Shah guy anyway.) They never did us any wrong
when we were their customers.

Does this pose a risk of IB going insolvent? If they do, is there a risk of
their customers being just another creditor of a bankrupt corporation, with
respect to the stocks and futures that IB holds on behalf of those customers?
Or are those instruments held in bailment, or actually by some other company,
rather than as IB assets?

------
ralph84
Would be nice to have more details on exactly what happened with the trades.
Did the trades clear? Did IB liquidate the contracts before expiration? Was
someone on the hook for taking physical delivery?

~~~
ashtonkem
I feel like any sane clearing house should’ve rejected these trades.

------
Stierlitz
> Crude was actually around negative $3.70 a barrel when Shah’s screen had it
> at 1 cent

A bit of a design flaw, I wonder did they spend as much money on the software
as Equifax did?

if (-3.70 != 1) { send an alert; }

------
muyth
Would any of the traders mentioned that owe all that money to Interactive
Broker's have to pay the total amount? Or is that part of Interactive Brokers'
loss claim.

~~~
whatok
From the story:

> Customers will be made whole, Peterffy said. “We will rebate from our own
> funds to our customers who were locked in with a long position during the
> time the price was negative any losses they suffered below zero.”

I'd imagine this is so they don't get sued and have some shoddy code pop up
during trial.

~~~
vmception
or they're raising capital and the end justifies the means

same result of avoiding litigation, but even fewer consequences

------
raviolo
Comments about storage costs are nice theory but don’t tell me those storage
costs went up 50 dollars a barrel to cause CLK0 to go from +10 to -40 within
something like 30 minutes. It was all forced liquidation by brokers like IB
and people absolutely bamboozled by negative prices puking their positions.
Nothing to do with costs for actual storage.

------
neonate
[https://archive.md/EGIGB](https://archive.md/EGIGB)

------
dilandau
More to the point is that a bunch of dumbasses were speculating on oil futures
without understanding the nature of the market, and no ability to take
delivery on the oil in any event. Nobody should be holding those when they're
so close to expiry unless they know what they're doing.

------
thanatropism
About two months ago I delivered to a private client a (monthly-basis)
prediction model for ethanol prices that included Brent -- in logarithm -- as
a predictor.

I know the monthly spot average of that statistic (fetched from FRED, the Fed
of St. Louis system) won't turn negative, but still...

------
huy-nguyen
Another entry should be added to the “things programmers believe about
commodity prices.”

------
changoplatanero
non paywall link [https://business.financialpost.com/pmn/business-pmn/oil-
cras...](https://business.financialpost.com/pmn/business-pmn/oil-crash-busted-
a-brokers-computers-and-inflicted-huge-losses)

------
tester89
I don’t really understand why the traders would end up owing money. If they
thought they were paying 0,01 $/bbl but we’re actually “paying” -37 $/bbl
wouldn’t IB owe them?

~~~
Stierlitz
@tester89 > I don’t really understand why the traders would end up owing
money. If they thought they were paying 0,01 $/bbl but we’re actually “paying”
-37 $/bbl wouldn’t IB owe them?

My thoughts precicely, is there anyone on here that could explain the
intricacies of these kind of trades?

------
unixhero
I would recommend [http://www.saxobank.com](http://www.saxobank.com) , it's a
kick ass broker.

------
SrslyJosh
I am playing my second tiniest violin for them right now.

------
ninetyfurr
Oh a guy with $77,000 in his day trading account didn't know futures can go
negative. I'm shocked.

Ironically this is the same guy who will sell you a gym contract without a
cancellation option and blame you for not doing your research when you owe him
$1100.

------
BubRoss
As usual, this nonsense headline is clickbait for the truth - some traders
couldn't trade when oil went below zero.

