The title might suggest that the rollout of new software was the issue, but the article states the very contrary: it was the old software that was the culprit:
> An internal review at the bank found humans manually operating the old software were ultimately at fault
> An internal review at the bank found humans manually operating the old software were ultimately at fault
Which is of course an entirely bogus cop-out. If a mistake can be made in a manual operation then sooner or later it will be. Lower down the article says that manual checks that were supposed to catch this error failed to do so. Ineffective checks are a management responsibility, and that responsibility goes all the way up to the CEO.
I mean...if this were any other business, someone clicking the wrong button and then someone else not catching this error would maybe mean someone's account was accidentally closed, or someone receiving a free pizza, or whatever.
The IT systems within banks are more or less the same as IT systems anywhere. Just as advanced, just as crappy. The difference is that if there is a human error with banking software, you're not sending free pizza, you accidentally pay ouy $900,000,000.
Most in-bank or between banks transfers are reversible and usually a non-issue. That why the risk management probably says something like this:
Risk: Incorrect transfer of funds to customer in another bank
Mitigation: Manual review of all funds transfer above 5 million dollars
Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank
Status: Risk accepted
Edit: Clarified "Mitigation: Besides litigation issues bank transfers are reversible" into "Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank"
Nothing about bank risk management is easy. You haven’t accounted for a whole host of risks with this simple analysis, including the most important one: customer retention. The high-value customers sending high-value interbank transfers won’t be impressed that our systems let their wire go out the door incorrectly. Even if I can totally reverse the transaction (and it’s not nearly as easy or guaranteed as you’re assuming) I still have to tell the client about it in most cases. The client will, correctly, think “what if they can’t get it back next time this happens? I’ll find a bank that doesn’t have these kinds of issues.” And aside from crimes, losing a high-value client is perhaps the worst offense you can commit in banking.
Regardless of the dollar amount or outcome I also have to tell the audit committee, the board, the auditors, and all of my regulators. And exactly none of those groups would let me put your write-up along with the conclusion “risk accepted” in front of them.
Most bank transfers are actually not reversible, except for some limited retail client (including small companies) operations where specific terms & conditions allow the bank to reverse payments to the extent possible, which they really prefer avoid using as it looks really bad for a bank whose most important asset is the confidence of its clients and counterparties. Reversal may also no longer be possible if the money has already gone out in a system that does not allow reversal, or if the client is bankrupt in the meantime (depending on local banking and bankruptcy laws and circumstances).
For any other payment system for larger sums / corporate and institutional parties, settlement finality is a huge thing that is the subject of all sorts of specific legislation, as it would be a real issue for the health of the financial system if a settled payment can simply be reversed, as it would have a lot of unintended consequences further down the line. So banks actually do have strict risk management policies to avoid wrong payments, but there are so many complex transactions for which ultimately a human (actually at least 2 due to 4-eyes principles) must confirm whether conditions for payment are satisfied and whether payment details are correct, and humans are always prone to making mistakes once in a while.
I didn't make myself clear, I wanted to mean that transfer are reversible (with the cooperation of the other bank) not that the source bank can unilaterally do it.
With bank cooperation, which usually happens, settlement are non-issues. When an operation can be reversed by one of parties the settlement agreement usually mentions that the settlement is only final when the reversion period is over.
Risk: An employee can transfer all of a customers funds to an oversees bank account
Mitigation: Multiple employee must approve transfer of funds
Mitigation: No individual employee can deploy modifications to the computer system actually doing the transferring.
Mitigation: No team can both write code and access (the important) production systems
Mitigation: Must stand on head while deploying code, because people standing on their head are more honest
Status: Risk accepted, we'll have 10 more meetings to review this next month.
The idea that bank transfers are reversible is false. Some are, some aren't, the adversaries are interested in the ones that aren't. The idea that manual review is a trivial fix is very false. Even if there was a trivial fix, the idea that you could get this past the numerous gatekeepers with a simple and easy process is probably false in most banks.
"But the employee didn’t select the correct system options -- instead allowing the loan to be repaid in full with interest. Colleagues who are supposed to catch such errors didn’t."
From experience in investigating mishaps like that:
1) no maker-checker control,
2) no imposed limits (with forced maker-checkers - more than one checker)($900m with one click???? what the actual ....),
3) lack of training,
4a) pressure to do this NOW NOW NOW NOW (sorry for the caps),
4b) overworked/tired (matching point 6 below), if that person is "stuck" at home with two screaming kids aged 2-6 for the past five months, I feel for them.
5) toxic environnment that did not allow the employee to spend 2 extra mins to think twice before clicking,
6) in these COVID times not having someone next to him/her and/or was too afraid to ping someone to ask "hey dude, just to make sure, am I using MenuOption1 or MenuOption2 for this almost $1b thingie?" (again, inadequate training & toxic env.)(easier to tap someone in the bag and ask them to look at your screen that get on a Lync call, share screen.
Absolute controls in place would be limits & maker-checker.
And this is the point, when I browse the "jobs" HN, I NEVER see any on audit/controls/GRC.. as if DevOps are the gods of everythinig and auditors are useless and not needed.. sigh
I know there are other (better?) websites when it comes to looking for Audit/Sec work, but I feel that things like that should be taken care of in the development cycle, not the post-mortem of a mishap.
> And this is the point, when I browse the "jobs" HN, I NEVER see any on audit/controls/GRC.. as if DevOps are the gods of everythinig and auditors are useless and not needed.. sigh
The roles that get posted to HN are almost exclusively development related or development adjacent (such as PM roles).
If you're not looking for those roles, it tends to not be very helpful directly. But if can be useful to look through, identify companies that appear to be doing interesting things, and then looking up their full job board to see _all_ of the roles they're hiring for.
> ...Colleagues who are supposed to catch such errors didn’t.
This might well be a case of to be the hard problem of shared responsibility becoming someone else's responsibility.
There is no perfect solution - even the implicit death penalty does not prevent avoidable airplane crashes (AF 447, PIA 8303...) - but it seems that things could have been done better here, as indicated by the preceding sentences:
"After Revlon repurchased part of the debt, a Citigroup employee was supposed to manually adjust the share of the loan the remaining lenders still owned ahead of interest payments scheduled to be sent out this month."
It is asking for trouble to have a process that allows you to start a task having no immediate, irreversible consequences, but which presents few or no barriers to accidentally executing one that does. Were the "colleagues who are supposed to catch such errors" notified of the actual transaction that was about to be performed, or only of the one that was intended?
I'm put on a banking project (as external) which already flushed down the toilet around ~$500M. Based on my experiences of the meetings and meetings about meetings, I totally understand how the incompetence lead to this clusterfuck.
My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." He answered that although he is in the banking business for decades, he still doesn't know.
These 100s of Millions of losses are not necessarily threatening core business. I find it amusing.
This is a very misunderstood article. The money they "create", i.e loaned or paid out, has to be funded by a deposit or similar borrowing. Making sure they can fund all their commitments is what liquidity managers and treasury departments do, it's why regulators subject banks to annual ILAAPs (Internal Liquidity Adequacy Assessment Process), it's why banks have liquidity risk and modelling teams to manage any "gap" risk banks are running in this respect.
If banks could simply create money then they'd never go bust. The only exception is the Central Bank, which can create new money that is it uses to buy assets of the same value, supporting prices and improving liquidity in the financial system.
This is also the root of the overnight repo market that everyone was up in arms about awhile ago.
The bank originates a new loan that they think will be profitable. Then they need to come up with the assets to offset that loan on their balance sheet (or at least a small percentage of it).
From a macro perspective you'd expect some banks to have more assets than they need and some to have less (because if I loan Joe money then eventually it will end up back in a bank somewhere, or at least a percentage of it). Each night the banks that need assets borrow from the banks that have assets and pay a small fee.
> They do create money effectively by loaning out deposits
That's what we're taught in school but it's really backwards. They make loans that they think will be profitable and then figure out how to get the reserves needed to cover the balance sheet (either through issuing equity, drumming up more deposits, or borrowing in the overnight repo market).
Sure, their assets are somewhat fungible in both time and space so long as they meet liquidity regulations.
Regardless, the point is that they can't poof money for themselves like the Fed, or another central bank, can. They can increase the economy's supply of money effectively, but that is different from having direct power over monetary supply.
Only central banks create money. The others redirect money away from insurance vaults.
Insurance requires a load of money in a vault for when disaster strikes. When it does, they curtail lending. The impact of losses is not felt immediately. It is felt much later.
The banking industry owns more money and is scheduled to own a whole lot more money in the future. And by "more money" I mean more market share of all the money in the world.
The biggest threat to their treasures is when government prints more money. Because it limits what they are able to spend lest they cause spiraling inflation.
The banking industry is the MOST powerful industry in the world. And they don't know it or they keep the realization of that power in a tight fist.
Edit: in the context of this loss, they didn't lose anything. They just bought the debt from the creditor. They are still owed money from the debtor.
I worked at Citi for a very short time way back when. We were doing some things I thought were a bit “sketchy”, and was wondering if we were breaking the law.
The response from my boss: we’re only breaking the law if we get caught, so theoretically we’re not actually breaking the law since no one has “caught us.”
Guess there was some logic there. Of course this was a very long time ago. And sure they follow those pesky banking rules now
, never, ever “breaking the law.”
Expected, someone should have caught it in code review. Switch statements are generally harder to follow than if/else chain and fallthrough etc make it even more complicated.
I don't fully understand what happened, but it sounds like essentially someone was supposed to update a payment from "repay full amount" to "pay interest only", and didn't do it. The system correctly executed the instructions it was given. The only way this would show up as a "loss" to the bank is if some part of the system expected to pay only the interest, and it sounds like that wasn't the case.
It also sounds like there was another human who was supposed to catch the error and didn't. My guess is that the volume of transactions is high enough and errors of this type are rare enough that most people wouldn't be able to catch it manually.
> An internal review at the bank found humans manually operating the old software were ultimately at fault