
Citigroup fined $7m after legit transactions mistaken for test data for 15 years - adamnemecek
http://www.theregister.co.uk/2016/07/13/coding_error_costs_citigroup_7m
======
kazinator
Synopsis: _SEC sends clear message to tech people in finance: shut the fuck up
if you find something, silently fix it, and sweep the remaining crumbs under
the rug, or else your company will be fined millions._

~~~
AdeptusAquinas
If they did that and the SEC found out, they probably wouldn't stop at just
fining you. Which is probably what Citigroup guessed and why, after realising
the extent of the issue, they reported it fully within a month

~~~
DugFin
Indeed, Martha Stewart is a prime example of how NOT to deal with the SEC.
Turns out she wasn't guilty of insider trading, but she still went to jail for
lying to the SEC about the trades in the first place. They take that very
seriously.

~~~
stickfigure
This particular example seems to reinforce the argument for _saying nothing at
all_.

~~~
simonh
Directors of companies have legal responsibilities to provide these reports to
the government. So instead of the company getting a fine for making a mistake
even though you tried to comply, now you are personally choosing not to fulfil
your legal obligations as a director. Smart move.

~~~
Bromskloss
> Directors of companies have legal responsibilities to provide these reports
> to the government.

What if you don't tell the directors, but just fix the problem you have
discovered? Is everyone off the hook then, or does that also amount to
breaking the law (in some country)?

~~~
zhte415
Why you're in Operations or Technology in large financial companies, you're
confronted with 'could do better' issues all of the time. All of the time. The
more issues the greater the legacy of the system you're working on, as in this
case, functionality has been altered from the original vanilla system (here,
alphanumeric field type replacing a numerical feed type, the the documentation
not being updated/references/or probably in an unreferenced document
documenting the original system).

Spotting these things is a _fantastic_ way for a VP to get on the fast-track
to Director. And it is fantastic, as a Director, to have people on your team
actively looking for holes.

Why?

Because Operational Control is a #1 strategic target for all banks. $7million
is nothing. At an extremely senior level this is evidence that a culture of
transparency and compliance exists in the company, and ammo the next time the
SEC or FED express a 'concern'. At a low level it is a Director or VP
demonstrating to their boss that they understand the strategic direction, and
that under their watch nothing big is going to blow up, nothing $7billion big;
something $7million big is nothing, they know their boss knows this and will
get a thank you for it being raised.

So you tell the directors. You make a nice PPT and include it in 'initiatives'
when a senior visitor comes to visit and gets a de-brief on your department.
You make sure it's carefully and clearly explained, so they can explain it to
their boss in a nice, pro-active, continuous improvement kind of manner.

A bank which does not operate like this, in the post 2008 era of regulatory
punishment for purgery, is an organisation with a very short future.

Source: Work in Operations and Technology in large banks.

~~~
Apofis
> nothing $7billion big; something $7million big is nothing, they know their
> boss knows this and will get a thank you for it being raised.

Further, this was obviously discussed with the boss before it was raised.

------
fredfoobar42
$7m is pocket change for Citigroup.

No wonder the big banks keep shitting the bed. There's never any real
consequences for their fuckups.

~~~
aaron695
This is the sort of comment why people vote for Trump.

They made a mistake, a programming mistake no less, something many people here
have done.

Then fully admitted it when they found out.

Yet this is them being evil?

No wonder people vote against the left.

~~~
cm2187
The other thing is with a company the size of citigroup (250,000 employees),
it is statically impossible not to have:

\- incompetent and/or careless employees and managers

\- dishonnest employees

\- computer bugs, glitches, clerical errors

If you take down a large corporation every time you find any of these, there
will be no corporation left within a year. Just small companies that were
statistically lucky to have neither of those that particular year.

Name me a program, any program (other than Hello World) where no bug has never
been found!

~~~
eru
They had this bug for more than a decade. They should have systems in place to
look for these things.

People make mistakes, but these mistakes should be caught before they get into
production. And the ones that still make it into production should be hunted.

~~~
cm2187
Should we discuss about all the 15 years old bugs that are found in Windows,
Linux and MacOS which are well into production (and many of them critical bugs
that affect the core of the product)? Has a software company ever been fined
or held liable for bugs in its products? In fact too often, bug fixes are paid
updates.

Financial companies are held to extraordinary standards, and in my opinion
it's a game they cannot win.

~~~
tremon
This is not a minor bug, this is a bug that caused data to be misrepresented.
If you want to compare it to OS bugs, then you need to look at silent data
corruption: how many data corruption bugs have gone undetected in operating
systems for 15 years?

~~~
lmm
The notorious (and still disputed) HFS+ bit-rot issues?

------
jonknee
They didn't lose anything, they were FINED $7m for submitting incorrect
reports. With the number of mergers that they have gone through it's not
surprising they have trouble with company wide reporting.

~~~
josu
This is the gist of it:

"But in 1998, the company started using alphanumeric branch codes as it
expanded its business. Among them were the codes 10B, 10C and so on, which the
system treated as being within the excluded range, and so their transactions
were removed from any reports sent to the SEC."

~~~
jonknee
The original title stated said "Citi lost $7m after legit transactions
mistaken for test data". That was what the Register used too, but it's very
misleading.

------
twunde
I think the most impressive part is that apparently the same system and code
had been running for 15 years as the number of transactions reported on had
increased exponentially

~~~
mherkender
I think it's the bad kind of impressive, one where ATMs still run COBOL.

~~~
PeCaN
The bad kind of impressive would be getting running ATMs on NodeJS.

 _ducks_

~~~
AtheistOfFail
May I introduce you to the "Lamassu Bitcoin ATM"?

------
ComodoHacker
Part of responsibility here is on SEC too IMO. When designing that "blueprint"
format and data exchange protocols, they didn't implement mechanisms to verify
correctness and completeness of data received.

------
jswny
Important to note that they only left these transactions out of report data
sent to the SEC. It's not like they were not honoring the transactions or that
some people were missing money.

~~~
bboreham
But if those trades included insider trading or market manipulation then they
were hidden from the regulators' investigations.

Barings was brought down by trades in a special "error account" 88888 that was
excluded from (different) reporting.

------
trequartista
I have done bluesheet reporting code for another large bank and it is one of
the most tedious things you can ever do in software. It's a report of all the
trades that the bank does and the SEC can come back and ask for historical
data at any point of time. So there's a huge database with feeds coming in
from multiple trading systems. Usually these feeds have to be enriched with
the right account numbers and so on. This needs lookups to other reference
data systems, which frequently are changing because of changing regulations,
growing businesses etc. I am not surprised this bug remained undetected for
more than 15 years. The guys who coded this initially are probably long gone
and nobody did knowledge transition of the fact that some account ranges are
not test accounts, even though they look like test accounts.

------
sf56
Totally unrelated, but I am not surprised. I once logged into my Citi credit
card account and was granted access to another user's account. Certain places
were off limits but I was able to view a lot of details. Pretty scary! I never
heard back after reporting the issue.

~~~
Sami_Lehtinen
Did you collect proper snapshot? Did you report it to the right official
authorities?

~~~
sf56
I did grab a few screens and contacted the only available "support" link I
could find easily on their site, but after a few weeks with no response I
eventually deleted the files.

------
argelius
This makes me feel more comfortable with the reality that some broken code I
wrote a long time ago might still be running somewhere... :)

------
azurezyq
Seems to be something like "089" < x < "100", should never use "range" to
handle string ids.

~~~
jmtulloss
There's probably a comment above it

    
    
      // XXX: too general, revisit when we have more than 100 branches

~~~
bbcbasic
Should have written TODO

~~~
smcl
I've seen the XXX suffix in the CPython source code - what does it mean
compared to TODO?

~~~
brianshaler
I would assume searchability. A non-case-sensitive non-regex search of a
codebase could yet false positives for things like "autoDot"

------
critium
I suspect the bugs like this come about because of a patch not because of the
original development. Devs and dev teams tend to get sloppy after the first
push and budgets tend to shrink dramatically.

One time I found a bug that was running for a few years and the result of it
was the company was under reporting by millions of dollars per quarter (the
running total was near to $100mm, and im sure it crossed it after my contract
ended).

This was VERY well tested software in the beginning (one of the best test
suites i've seen actually) and audited up to high heaven. The problem started
when the patches rolled in and those, are not tested anywhere near as much.

------
comp1927
//Try this in your console.

function should_not_exclude_10B_10C(x) { if ("089" < x < "100") {
console.log('excluding ' \+ x + ' in report.') } else { console.log(x + ' is
normal.') } }

should_not_exclude_10B_10C("10B")

should_not_exclude_10B_10C("10C")

~~~
rfurmani
Actually, no: ascii 10B>100\. The bug above is that you can't do chained
inequalities like that in Javascript. Try: should_not_exclude_10B_10C("200")

~~~
comp1927
B > 0 assumption is broken in their code. Read Page 4, footnote no.4:
[https://www.sec.gov/litigation/admin/2016/34-78291.pdf](https://www.sec.gov/litigation/admin/2016/34-78291.pdf)
This was not caused by Ascii confusion...

------
gravypod
I've never understood how these fines are meant to benefit anyone. If no one
is affected, then why is there a fine? Who is this money going to for damages
to be repaid?

Also, why are you not allowed to use real data for testing purposes?

~~~
bagacrap
The SEC was not impressed and said in a statement announcing the fine that the
"failure to discover the coding error and to produce the missing data for many
years potentially impacted numerous Commission investigations."

------
BallinBige
what were these transactions for?

~~~
andyjdavis
There is mention of trading data which suggests to me that it might be to do
with traders employed by Citigroup ie staff members whose job it is to use a
block of the bank's money or outside investor's money to buy and sell shares,
commodities and pretty much anything where they believe they can buy and sell
to turn a profit.

------
jsprogrammer
Yawn. This company died eight years ago.

