
Citi’s $900M Misfire Happened During Software Switch - whatok
https://www.bloomberg.com/news/articles/2020-08-25/citi-s-900-million-misfire-happened-in-midst-of-software-switch
======
jakub_g
The title might suggest that the rollout of new software was the issue, but
the article states the very contrary: it was the old software that was the
culprit:

> An internal review at the bank found humans manually operating the _old_
> software were ultimately at fault

~~~
PaulAJ
> An internal review at the bank found humans manually operating the old
> software were ultimately at fault

Which is of course an entirely bogus cop-out. If a mistake can be made in a
manual operation then sooner or later it will be. Lower down the article says
that manual checks that were supposed to catch this error failed to do so.
Ineffective checks are a management responsibility, and that responsibility
goes all the way up to the CEO.

~~~
sharken
At least Citibank is trying to upgrade their ancient systems, but it sure
looks as if previous or current CEOs failed to exercise due diligence.

Doesn't look like good risk management at all.

~~~
tumetab1
Actually, banks risk management is easy.

Most in-bank or between banks transfers are reversible and usually a non-
issue. That why the risk management probably says something like this:

    
    
      Risk: Incorrect transfer of funds to customer in another bank
      Mitigation: Manual review of all funds transfer above 5 million dollars
      Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank
      Status: Risk accepted
    

Edit: Clarified "Mitigation: Besides litigation issues bank transfers are
reversible" into "Mitigation: Besides litigation issues, lost funds are easily
recovered by asking the receiving bank"

~~~
pieno
Most bank transfers are actually not reversible, except for some limited
retail client (including small companies) operations where specific terms &
conditions allow the bank to reverse payments to the extent possible, which
they really prefer avoid using as it looks really bad for a bank whose most
important asset is the confidence of its clients and counterparties. Reversal
may also no longer be possible if the money has already gone out in a system
that does not allow reversal, or if the client is bankrupt in the meantime
(depending on local banking and bankruptcy laws and circumstances).

For any other payment system for larger sums / corporate and institutional
parties, settlement finality is a huge thing that is the subject of all sorts
of specific legislation, as it would be a real issue for the health of the
financial system if a settled payment can simply be reversed, as it would have
a lot of unintended consequences further down the line. So banks actually do
have strict risk management policies to avoid wrong payments, but there are so
many complex transactions for which ultimately a human (actually at least 2
due to 4-eyes principles) must confirm whether conditions for payment are
satisfied and whether payment details are correct, and humans are always prone
to making mistakes once in a while.

~~~
tumetab1
I didn't make myself clear, I wanted to mean that transfer are reversible
(with the cooperation of the other bank) not that the source bank can
unilaterally do it.

The law sides with the banking making the mistake as discussed on
[https://news.ycombinator.com/item?id=24222045](https://news.ycombinator.com/item?id=24222045)

With bank cooperation, which usually happens, settlement are non-issues. When
an operation can be reversed by one of parties the settlement agreement
usually mentions that the settlement is only final when the reversion period
is over.

------
LordAtlas
This is the key part:

"But the employee didn’t select the correct system options -- instead allowing
the loan to be repaid in full with interest. Colleagues who are supposed to
catch such errors didn’t."

Saved you a click.

~~~
HenryBemis
From experience in investigating mishaps like that:

1) no maker-checker control,

2) no imposed limits (with forced maker-checkers - more than one
checker)($900m with one click???? what the actual ....),

3) lack of training,

4a) pressure to do this NOW NOW NOW NOW (sorry for the caps),

4b) overworked/tired (matching point 6 below), if that person is "stuck" at
home with two screaming kids aged 2-6 for the past five months, I feel for
them.

5) toxic environnment that did not allow the employee to spend 2 extra mins to
think twice before clicking,

6) in these COVID times not having someone next to him/her and/or was too
afraid to ping someone to ask "hey dude, just to make sure, am I using
MenuOption1 or MenuOption2 for this almost $1b thingie?" (again, inadequate
training & toxic env.)(easier to tap someone in the bag and ask them to look
at your screen that get on a Lync call, share screen.

Absolute controls in place would be limits & maker-checker.

And this is the point, when I browse the "jobs" HN, I NEVER see any on
audit/controls/GRC.. as if DevOps are the gods of everythinig and auditors are
useless and not needed.. _sigh_

I know there are other (better?) websites when it comes to looking for
Audit/Sec work, but I feel that things like that should be taken care of in
the development cycle, not the post-mortem of a mishap.

~~~
cosmie
> And this is the point, when I browse the "jobs" HN, I NEVER see any on
> audit/controls/GRC.. as if DevOps are the gods of everythinig and auditors
> are useless and not needed.. sigh

The roles that get posted to HN are almost exclusively development related or
development adjacent (such as PM roles).

If you're not looking for those roles, it tends to not be very helpful
directly. But if can be useful to look through, identify companies that appear
to be doing interesting things, and then looking up their full job board to
see _all_ of the roles they're hiring for.

------
kmarc
I'm put on a banking project (as external) which already flushed down the
toilet around ~$500M. Based on my experiences of the meetings and meetings
about meetings, I totally understand how the incompetence lead to this
clusterfuck.

My question to my boss was rather: "but _where_ do these banks get this huge
amount of money from? I guess it's not from the $5 account fees." He answered
that although he is in the banking business for decades, he still doesn't
know.

These 100s of Millions of losses are not necessarily threatening core
business. I find it amusing.

~~~
howeyc
> My question to my boss was rather: "but _where_ do these banks get this huge
> amount of money from? I guess it's not from the $5 account fees."

They create it.

[https://www.bankofengland.co.uk/quarterly-
bulletin/2014/q1/m...](https://www.bankofengland.co.uk/quarterly-
bulletin/2014/q1/money-creation-in-the-modern-economy)

~~~
retube
This is a very misunderstood article. The money they "create", i.e loaned or
paid out, has to be funded by a deposit or similar borrowing. Making sure they
can fund all their commitments is what liquidity managers and treasury
departments do, it's why regulators subject banks to annual ILAAPs (Internal
Liquidity Adequacy Assessment Process), it's why banks have liquidity risk and
modelling teams to manage any "gap" risk banks are running in this respect.

If banks could simply create money then they'd never go bust. The only
exception is the Central Bank, which can create new money that is it uses to
buy assets of the same value, supporting prices and improving liquidity in the
financial system.

~~~
syndacks
[https://en.m.wikipedia.org/wiki/Fractional-
reserve_banking](https://en.m.wikipedia.org/wiki/Fractional-reserve_banking)

Banks do create money, by loaining other people's. I think you're saying the
same thing?

~~~
stu2b50
That's what he's saying. They do create money effectively by loaning out
deposits, but that's far from literally creating money.

If they have a deficit on their sheets they can't just create money for
themselves like a central bank could.

~~~
mason55
> _They do create money effectively by loaning out deposits_

That's what we're taught in school but it's really backwards. They make loans
that they think will be profitable and then figure out how to get the reserves
needed to cover the balance sheet (either through issuing equity, drumming up
more deposits, or borrowing in the overnight repo market).

[https://www.investopedia.com/articles/investing/022416/why-b...](https://www.investopedia.com/articles/investing/022416/why-
banks-dont-need-your-money-make-loans.asp)

A good phrase to search for to learn more is "loans create deposits"

~~~
stu2b50
Sure, their assets are somewhat fungible in both time and space so long as
they meet liquidity regulations.

Regardless, the point is that they can't poof money for themselves like the
Fed, or another central bank, can. They can increase the economy's supply of
money effectively, but that is different from having direct power over
monetary supply.

------
neonate
[https://archive.is/Wwicp](https://archive.is/Wwicp)

------
brooklyndude
I worked at Citi for a very short time way back when. We were doing some
things I thought were a bit “sketchy”, and was wondering if we were breaking
the law.

The response from my boss: we’re only breaking the law if we get caught, so
theoretically we’re not actually breaking the law since no one has “caught
us.”

Guess there was some logic there. Of course this was a very long time ago. And
sure they follow those pesky banking rules now , never, ever “breaking the
law.”

------
massaman
From the blameful-postmortem:

Q. Why did we click `Send $900M?` A. Not sure. Tom felt all clicky-clicky, so
he clicked it.

Q. Why did we hire Tom? A. Also not sure. [Action Item: Fire Tom]

------
massaman
Somewhere in your company hiring queue is a resume with:

Strengths: Architecting risk management systems

Weaknesses: Sometimes I click on things to see what happens

~~~
sukilot
Those are both strengths. If Citi had that person they'd have learned from a
lot of $1M messes before having a $900M mess.

------
totaldude87
reminds me of that missile alerting system gif..

[https://giphy.com/gifs/emibob-ads-missile-warning-system-
xUL...](https://giphy.com/gifs/emibob-ads-missile-warning-system-
xULW8wq0KtPBfXPRpC)

------
fooyc
The title is misleading, the switch doesn’t appear to have anything to do with
the misfire.

~~~
wwright
Yeah, but how many lay people care about causation in an RCA? ;)

~~~
leetrout
Root cause analysis

~~~
wwright
Just because they're about the cause doesn't mean the people writing or
reading them care about the cause

~~~
leetrout
Just adding words for the initialism for those that don’t know what RCA is.

------
noahmbarr
I can only imagine a guy like larry ellison reacting to this article.

------
jordache
stupid article. Reveals nothing of what contributed to the human error.

They had been using the legacy software for many many years, w/o significant
issues around human errors.

------
scott31
Expected, someone should have caught it in code review. Switch statements are
generally harder to follow than if/else chain and fallthrough etc make it even
more complicated.

~~~
jordache
how did you summarize this to be a coding issue?

~~~
yesplorer
by not reading the article itself.

------
londons_explore
I assume financial software has the concept of a set of atomic transactions -
ie. "debit bob $X and credit mary $X".

Given that, presumably all buttons an operator clicks should generate a set of
atomic transactions between customers and the bank.

An automated system can then check that the total loss to the bank after these
transactions have been executed isn't too big.

I can't really imagine how any bit of software didn't have those checks in
place...

~~~
yellowstuff
I don't fully understand what happened, but it sounds like essentially someone
was supposed to update a payment from "repay full amount" to "pay interest
only", and didn't do it. The system correctly executed the instructions it was
given. The only way this would show up as a "loss" to the bank is if some part
of the system expected to pay only the interest, and it sounds like that
wasn't the case.

It also sounds like there was another human who was supposed to catch the
error and didn't. My guess is that the volume of transactions is high enough
and errors of this type are rare enough that most people wouldn't be able to
catch it manually.

