
TSB Train Wreck: Massive Bank IT Failure Going into Fifth Day - pm24601
https://www.nakedcapitalism.com/2018/04/tsb-train-wreck-massive-bank-it-failure-going-into-fifth-day-customers-locked-out-of-accounts-getting-into-other-peoples-accounts-getting-bogus-data.html
======
merricksb
This is from 3 days ago, same day as another thorough discussion about the
topic on HN:

[https://news.ycombinator.com/item?id=16910947](https://news.ycombinator.com/item?id=16910947)

~~~
tim333
This kind of #3 in the HN TSB screw up series

\- 'TSB gave me access to someone's £35,000' (bbc.co.uk)

\- TSB Bank: Botched upgrade has left customers unable to access their
accounts

\- this

\- Warning signs for TSB's IT meltdown were clear a year ago, according to
insider

------
xedarius
I am a TSB customers and the service has been insane.

Here's some of the highlights

\- Login screen regularly throws a message from the backend saying
'java.lang.NullPointerException' (so now I know what the backend written in)

\- My balance was totalled incorrectly, which made me appear to be very
wealthy.

\- Payments had gone missing from my statement (how do I know they were
missing, one of them was my pay)

\- When I did login I tried to transfer some funds and was met with 'Payment
successful' followed by a java.lang.IndexOutOfBoundsException. Had the payment
worked? Who knows

\- They have a fundamental scaling issue.

I've never seen anything like it. I do not get the impression this will be
fixed quickly.

~~~
3pt14159
Why don’t you just bank elsewhere?

~~~
swarnie_
British banks are developing a bit a of a habit for massive IT failures.
Moving is fine but someone else will fail next month.

~~~
twic
Although, as the article mentions, TSB are owned by a Spanish bank, Sabadell,
and it was the move to Sabadell's platform, developed in Spain, that caused
the trouble!

------
sbarre
I feel bad for the rank-and-file developers (the ones without decision-making
power) who are stuck trying to fix all this in what are probably insanely bad
working conditions right now.

Yeah yeah "get a different job" or "you made the mess, you fix it" but massive
failures like this aren't orchestrated by the front-line developers, they
happen because people many steps removed from the work make bad executive
decisions.

A deadline was probably set long ago, and no one adjusted it when unexpected
problems and complexities arose.

~~~
joelhaasnoot
Sure, this definitely is true, but when your website or app is throwing an
IndexOutOfBoundsException left and a InvalidBeanException right, your code
quality also has issues. Security auditors (good or bad as they may be) would
likely have your head for _any_ leaked exception, for all they know the
application may also be leaking other info.

~~~
edf13
My guess is the code just wasn't ready for production... someone higher up was
screaming they didn't care and a deadline is a deadline so push it out.

~~~
EnderMB
Alongside this, it's all down to risk.

I have limited experience with financial systems (zero directly working with
banks), but I've noticed a trend where institutions are willing to take huge
gambles with the integrity and security of their systems if the risk isn't
that great.

For them, risk was losing money, and while you'd think that having severe
downtime or awful security practices would result in losing money, I didn't
meet a single manager that believed this. They were happy to cut corners
wherever because it got the job done faster, and under budget, and that's what
they were judged on.

A mate of mine worked for a large UK-based bank, and pretty much confirmed
this for me when we asked about what it was like to build web services for a
huge bank. From the outside, their devs were the cream of the crop. Oxbridge
educated, some ex Google or Microsoft guys, and years of experience in
building huge systems, but a management culture that saw them as code monkeys,
and wanted work delivered that was an inch off of negligent if it meant it was
done quick.

The risk to the managers was their job, but from an upper management
perspective all that mattered was whether it would directly affect the bottom
line in an obvious (non-tech) way. Bugs, security breaches, or poor
availability didn't affect this, and were worthwhile risks.

~~~
ownagefool
I've worked with the UK banks. From experience they're typically large
outsourcing firms that contract their work out to the large consultancies that
are more interested in appearance and winning bun fights than delivering
quality.

I just left one of these orgs about a month ago where the program CTO of the
bank (title inflation) wanted to hire me direct because they believe they need
to have a core competency that isn't aligned with external political agendas,
but largely the other senior managers just see having their own techs as a
risk because they can't outsource the blame.

That and directly answering questions or questioning process upsets a lot of
people because they have fiedoms. Like someone asks you for all your IP
address you have on cloud and what server they're associated with. They have
no ability to handle the dynamic nature of these changing but their job is to
write them down so you need to fall inline sorta deal...

------
planetjones
British Banks have been playing "fast and loose" with their IT systems for a
long time now. TSB won't be the last time we see such a catastrophe. It is not
the first in recent memory either.

"Fast and Loose" may mean rushing changes through with unrealistic timelines,
outsourcing the software development or/and infrastructure management to the
cheapest location possible, sacking onshore experts to save money, etc. The UK
Government is fully complicit, as their support for RBS (owned by the
Government themselves) during their ruthless cost-cutting showed.

Banks are of systemic importance. People suffer if they can't make payments on
time or can't access money that's needed for something essential. But the
systemic importance is not reflected in the legislation: we have a situation
where Banks outsource their IT to Indian service companies and do their upmost
to move the "heavy lifting" of software development to the cheapest location
they can find, while the social networks and search engines are recruiting the
top talent.

~~~
jackweirdy
Indeed. Banks are technology companies. British high street banks either don't
understand or refuse to believe that they are technology companies.

~~~
gaius
Like the Civil Service, the senior ranks are selected by where their parents
sent them to school, and how well they learned to conjugate Latin verbs while
they were there. Technical skill is not only viewed with contempt it is
actually a bar to promotion.

Investment banking is actually a lot saner - Thatcher’s reforms in the 80s
went a long way towards destroying the Old Boys Network in that industry
niche.

------
21
This is one of the reasons I have 4 bank accounts in 2 different countries,
and also cash at home.

I can't understand people with one account, one card, and 10 pounds in their
pockets.

See also recent Swedish fears that their cashless society is very vulnerable
to a digital attack.

~~~
Silhouette
I haven't gone quite as far as that, but I agree with the principle.

As much as reasonably possible, I keep facilities like personal banking,
personal credit cards (always at least two), any mortgage or other secured
loan, and any business financial services with different organisations that
aren't part of the same group. There's often small print in bank terms and
conditions about being allowed to grab money from anything of yours they have
access to if there is a problem with anything else, and frankly I don't trust
them not to abuse that, particularly if they've already made a mistake that
caused a serious problem in the first place. Also, the FSCS compensation
limits tend to be per person per firm, so dividing assets can sometimes
increase the amount covered if anything really bad happens to the financial
firms.

And yes, I do also have some emergency cash hidden in safe places. I don't get
crazy paranoid about it, but it's a substantial amount, certainly enough to
fill my tank and buy essential provisions for a while. I would be concerned
about becoming a truly cashless society as well, for this among other reasons.

------
weavie
"Do not request a bean from a BeanFactory in a destroy method implementation!"

User friendly error message of the year!

[[https://twitter.com/thejackthomson_/status/98856435451268710...](https://twitter.com/thejackthomson_/status/988564354512687104/photo/1)]

~~~
IncRnd
Jack! Did you really sell the cow for beans??

------
onion2k
This article is 3 days old. Customers are _still_ having issues, so it's 8
days so far.

------
krona
As is usually the case, the owner outsourced to IBM last year. What happened
next was all but inevitable.

~~~
88
The CEO brought IBM in this week to clean up the mess.

It doesn’t appear they were involved in the original migration.

~~~
krona
The big support contacts started rolling in years ago. I know, because I was
there.

------
kennydude
This kind of thing gives me confidence in "challenger banks" like Monzo who
are building their systems in-house and I hope with a lot more testing than
this.

As far as I know Monzo is using microservices (a bank is probably big enough
for that to be the right choice) to try and limit the damage of anything
insane like this. The way they handled Prepaid to Current Accounts was
absolutely seamless [https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-
payment...](https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payments-
work/)

~~~
tachion
This has nothing to do with microservices and everything to do with lack of
skill, proper engineering and chaos in IT departments of almost all banks. You
can write and deploy (don't forget about the fact developers aren't skilled in
ops at all and that's a common cause for these things) good software in
microservices and monolith architectures, you just need to know what are you
doing.

Monzo/Mondo had issues on their own, and in fact they have very frequent
issues with their payment processors.

~~~
philcrump
They _had_ very frequent issues with their payment processors.

The current account (now the only account as the prepaid beta programme has
ended) runs on their internally developed payment processor, and has so far
had minimal issues.

More information here: [https://monzo.com/blog/2018/04/04/ending-
prepaid/](https://monzo.com/blog/2018/04/04/ending-prepaid/)

------
jasonsync
_bank IT is a systemic risk waiting to happen_

The same thing happened with Tangerine.ca when they launched their new website
late last year. Tangerine is owned by one of Canada's "big 5" banks,
Scotiabank.

For at least a week after the launch, users were experiencing login errors,
timeouts, incorrect balances and features not working. Backend code exceptions
would bubble up to the surface. It was ugly.

But even with the backend issues ironed out weeks later, users continue to
lament about the horrible new UI, which over 6 months later, is still a UX
nightmare on desktop web browsers.

Reading the 400+ Facebook comments on their "launch day post (they since
removed the post from the timeline) is quite an eye opener.

Apparently they did minimal testing and disregarded all user feedback during
the beta phase.

The comments are still accessible from the post photo here:
[https://www.facebook.com/TangerineBank/photos/pb.46798899664...](https://www.facebook.com/TangerineBank/photos/pb.467988996640009.-2207520000.1507258210./1225878164184418/?type=3)

------
Spooky23
This is one reason that I refuse to give up on paper statements mailed to my
home.

~~~
kozhevnikov
I download all PDFs and back them up as any other important data. I fail to
see how paper statements solve anything besides collecting dust.

~~~
codeulike
We know that PDFs are editable, but everyone seems to act like they're not, at
some point thats going to turn into a mini-crisis.

~~~
kozhevnikov
Everything is editable in a hex editor, but PDFs can be certificate signed
allowing for document authenticity and integrity verification.

~~~
pbhjpbhj
And of course the banks sign their PDFs ...

~~~
kozhevnikov
That's not PDF's fault, just like you can't blame Git for developers not
signing their commits.

------
jamiethompson
I'm a Lloyds customer, and I'm not suggesting that this is in any way related
but my current account today showed a planned 0% overdraft limit of £2 shy of
£10bn.

------
richardhod
Ironically, before they merged in the 90s, Lloyds had a terrible computer
system, and TSB was market-leading in major bank IT customer systems, at least
from a customer POV. Then they didn't learn and continue with TSB's IT systems
and management, but became more lloyds instead. Sadly, this new TSB is likely
only in name anything like the old one.

------
cesarb
I've been wondering: if this issue isn't solved soon, how long before we see a
run on that bank?

~~~
21
Jokes on you, you can't do a run on a bank where you can't login or take any
money out.

More seriously, I would definitely move my money somewhere else if I had an
account there. Not because I would worry that it will disappear (government
insurance & all that), but just because this looks like the kind of problem
which will take a long time to fix, and which could strike back.

~~~
PeterisP
You definitely can do a run on a bank where you can't log in - you come to the
office (or mail a registered letter) with written instructions to transfer all
your money elsewhere and close the accounts, and legally they have a quite
limited time to comply, I believe one or two business days at most, until
various EU consumer protection laws for financial products kick in.

Furthermore, there's a lot of process for switching banks that can be done by
the customer through the bank they're switching _to_. While your savings may
be stuck in TSC for some days, you don't need to go to TSC at all if you'd
choose to switch your wages and recurring bill payments to somewhere else, you
just go to the new bank.

~~~
TazeTSchnitzel
Current Account Switch Service, right?

------
dsabanin
Is this a Mr. Robot episode?

~~~
codeulike
I think it might be the ending of Fight Club

------
djhworld
The CEO says he's given a deadline of Saturday for IBM to fix the problem,
that seems a pretty tight deadline.

Are they using IBM technologies for their stack?

~~~
onion2k
It'd be a bit unfair to say that to IBM if they're not.

------
sorokod
The mess is big enough for the post-mortem to be made public. Should be
interesting.

~~~
gadders
I would imagine the UK regulators would launch an enquiry and publish a report
like they did when NatWest managed to break all their overnight batches:
[http://www.fca.org.uk/your-fca/documents/final-
notices/2014/...](http://www.fca.org.uk/your-fca/documents/final-
notices/2014/rbs-natwest-ulster)

------
kown223
I'm surprised no one mentioned the solution, buy bitcoin..

~~~
akerro
HEY LET ME JUST LOGIN TO MY TSB TO MAKE THE TRAnsfer...

