
TSB Bank: Botched upgrade has left customers unable to access their accounts - edward
https://www.theguardian.com/business/2018/apr/24/tsb-customers-online-banking-chaos-accounts
======
trollied
The amusing thing is, they forgot to cancel the automated press release that's
boasting about a successful go-live:

[http://prensa.bancsabadell.com/en/News/2018/04/banco-
sabadel...](http://prensa.bancsabadell.com/en/News/2018/04/banco-sabadell-
successfully-completes-tsb-technology-migration)

~~~
isaack
Taken down now

Backup 1: [http://archive.is/WnwiB](http://archive.is/WnwiB)

Backup 2:
[http://webcache.googleusercontent.com/search?q=cache%3Ahttp%...](http://webcache.googleusercontent.com/search?q=cache%3Ahttp%3A%2F%2Fprensa.bancsabadell.com%2Fen%2FNews%2F2018%2F04%2Fbanco-
sabadell-successfully-completes-tsb-technology-
migration&rlz=1CALEAJ_enUS730US730&oq=cache%3Ahttp%3A%2F%2Fprensa.bancsabadell.com%2Fen%2FNews%2F2018%2F04%2Fbanco-
sabadell-successfully-completes-tsb-technology-
migration&aqs=chrome..69i57j69i58.965j0j4&sourceid=chrome&ie=UTF-8)

~~~
sorokod
_The teams that have participated in migration have proven to have the skills
and knowledge required, which they have significantly developed and honed
during the course of this project. Banco Sabadell’s proposal to be an agent of
change has become evident, with an advanced technological proposition, not
only in the domestic market but also on an international scale. With this
project Banco Sabadell has created a competitive and value-added platform for
the TSB franchise._

Funny and poignant

------
cup-of-tea
The IT systems of these companies are held together loosely with gaffa tape.
This kind of thing could happen at any time. This is what happens when you
deliver your project due to pressure from non technical people even though you
know it's not properly finished. Why do people not listen to the professionals
when it comes to IT?

In my current job I have repeatedly told non technical superiors that reverse
engineering an outdated API is not a proper solution and could fall apart at
any time. They do not care. It doesn't even register. Do civil engineers have
this problem if they realise there might be a structural weakness?

~~~
c12
As far as I know civil engineers can go to prison if the construction they
took part in collapses - no such provision is afforded to software engineers.

~~~
FLUX-YOU
Besides simply quitting, what can a civil engineer do to blow the whistle on a
company that is cutting corners for a bridge?

~~~
scottfr
In the US civil engineers must go through a multi-year certification process
that involves a significant amount of on the job work in addition to
standardized examination before they are labelled a Professional Engineer
(PE).

As a PE, they can face significant legal repercussions for errors in work they
approve.

The benefit to the PE is that many Civil Engineering projects are legally
required to have a PE design/approve them. The PE literally has a physical
stamp they use to stamp blueprints and designs when they approve it. Work
won't proceed without that stamp.

So, if a PE refused to approve a bridge design, presumably it would be very
difficult to find another PE who would approve it (especially given the record
of the first PE rejecting the plans) and the work would not proceed.

~~~
ryandrake
If software engineers could potentially face legal repercussions or jail time
for serious errors, unsound technical judgment, or bad ethical decisions, it
would enable a lot more push-back on these poorly-planned projects and corner
cutting. I've always thought it was pretty sad that lawyers are held to higher
professional and ethical standards than software engineers.

~~~
EpicEng
> I've always thought it was pretty sad that lawyers are held to higher
> professional and ethical standards than software engineers.

I think that's a bit silly. The vast majority of software is trivial stuff and
no real harm is caused when something goes sideways. Few of us work on safety
critical systems. When a lawyer screws up it often does a lot of harm and they
are also inherently bound by our legal systems. Corruption and
negligence/incompetence in this area lessens the public's faith in those
systems.

Lawyers and civil engineer inherently take on far greater responsibility than
your average Joe Schmoe programmer writing yet another CRUD app.

~~~
skolemtotem
That's true, but what about the software engineers who _do_ build world-
changing stuff? I think that we should definitely hold them to a higher
standard than we currently do.

~~~
EpicEng
>That's true, but what about the software engineers who do build world-
changing stuff?

Yeah I can get behind that. I work in biotech/cancer diagnostics/prognostics
and we could definitely afford a higher standard, especially around the core
stuff (e.g. the CV/algorithms people, but also the typical IT end of things.)

------
isp
Cross-post from
[https://news.ycombinator.com/item?id=16911321](https://news.ycombinator.com/item?id=16911321)
:

This appears to be an IT failure comparable to the 2012 and 2015 IT failures
at the Royal Bank of Scotland. These involved multi-day / multi-week
full/partial outages, and multi-million pound fines.

FT:
[https://www.ft.com/content/9178ef60-46d9-11e8-8ee8-cae73aab7...](https://www.ft.com/content/9178ef60-46d9-11e8-8ee8-cae73aab7...).

Background: TSB is a UK bank. They had a planned 50-hour maintenance window
over the weekend (Fri 16:00 - Sun 18:00), to move away from Lloyds's systems
(following the "un-merger" in 2013-2014). This went wrong, in what is likely
to become a textbook example of an IT failure.

From a letter from Nicky Morgan, chair of the UK Parliament's Treasury
committee, to TSB CEO Paul Pester:

> The reports of unauthorised transactions, access to other customers’
> accounts, and failures of in-branch services have all the hallmarks of an IT
> meltdown. This is yet another addition to the litany of failures of banking
> IT systems. Potentially millions of customers could be affected by
> uncertainty and disruption.

Some interesting screenshots of people seeing debugging information in the
front-end:
[https://twitter.com/bcleeve/status/988333991676796928](https://twitter.com/bcleeve/status/988333991676796928)
[https://twitter.com/apphancer/status/988698182170357760](https://twitter.com/apphancer/status/988698182170357760)
[https://twitter.com/thejackthomson_/status/98856435451268710...](https://twitter.com/thejackthomson_/status/988564354512687104)

Ironically, TSB's parent company issued a press release (in error?) claiming
that the migration has been a "success"!
[http://press.bancsabadell.com/en/News/2018/04/banco-
sabadell...](http://press.bancsabadell.com/en/News/2018/04/banco-sabadell-
successfully-completes-tsb-technology-migration)

------
errozero
This article from 3 years ago predicted this situation:

Sabadell warned about ‘high risks’ in IT integration of TSB
[https://www.ft.com/content/c5157c1e-20ab-11e5-aa5a-398b2169c...](https://www.ft.com/content/c5157c1e-20ab-11e5-aa5a-398b2169cf79)

------
jamiethompson
It's got so bad that when my partner went into our local branch they were
handing out cash and writing it down in a little book because none of their in
branch systems were working either.

~~~
dazc
I'm actually old enough to remember when that was normal. You could also phone
your branch and the security was basically someone who recognised your voice.

------
errozero
I logged in to my TSB account yesterday and found angular placeholders and
bits of un-rendered html all over the place. The previous version was all
server side rendered and worked perfectly.

~~~
ams6110
Move fast and break things....

------
dx034
Why didn't they roll back the update as soon as that was discovered? Would've
caused another downtime of a few hours but better than the current chaos.

~~~
isp
I have no inside information for TSB, but I have worked on other large
migrations before at $DAYJOB.

Based on the original planned 50-hour outage, I suspect that TSB chose a "big
bang" migration: all-or-nothing, no rollback possible. These are technically
easier (so cheaper) to develop, but far far riskier than taking a phased
approach.

I have successfully argued against such approaches in the past, due to the
high risk of catastrophic failure if it goes wrong (i.e., exactly what has
happened to TSB).

I regard TSB's failure as an in-the-making textbook example of "how not to do
it".

~~~
mcroft
Often, these kinds of migrations are the only option due to previous cost
cutting/money saving decisions - likely due to pressure from the business and
nothing to do with any technical reason.

I remember one migration which involved lots of internal services which were
all tightly coupled, meaning updates across the whole backend, not just the
part that needed it and no way to do a phased rollout. It was made worse by
the fact that the update was to the platform and that depended on a DB
upgrade, but that upgrade was incompatible with the old version which meant
_everything_ had to be done at once.

They had no disaster recovery plan. They got it done, but I'm sure a lot of
people involved went grey early thanks to that nightmare.

Of course the business had no idea of the utter mess that caused all of this
and continued to make harmful decisions in the name of saving money and
further underfunding the IT department.

~~~
ams6110
Even with tightly coupled systems, ultimately it's all bits on the wire. You
can always develop shims or intermediaries to perform a phased migration with
rollback options. Of course that takes time and costs money, which is why some
people take the damn the torpedoes approach.

------
maxehmookau
Ironically I tried to close my TSB account on Saturday morning because I'd
just switched to Monzo Bank and was told they couldn't close my account due to
the issues they were having.

Still no access to my old account this morning...

~~~
jdefelice
Have you switched to Monzo as your full-time bank? I use it for everyday
spending but I not sure I want to put my wages into it yet.

~~~
sofaofthedamned
I tried Monzo and they're awesome, but ended up ultimately moving to Starling.
Reason being they have a full FSCS license, so your balance is guaranteed by
the government up to around £85k.

Apart from that they're both much the same, and both leagues better than a
normal bank. It's amazing buying stuff or getting cash out and the
notification comes through on the phone before the ATM tells you the cash is
coming!

Another good thing both have - you can switch off cards instantly. I thought
i'd lost my Starling card when out last week, instead of cancelling it I just
temporarily halted it until it was found.

~~~
maxehmookau
Monzo now has a full FSCS licence.

~~~
sofaofthedamned
Good to know, thank you.

What I do love about Monzo is their approach to development - they blog
everything, even the outages. Good to see the process involved in making a new
bank from scratch. I'm keeping the account as a scratch account but have
already moved the direct debits from my old Halifax account to Starling.

~~~
dankohn1
Monzo's head of engineering will be keynoting at KubeCon + CloudNativeCon in
Copenhagen next week, and talking about past outages:
[https://kccnceu18.sched.com/event/Dsan/keynote-anatomy-
of-a-...](https://kccnceu18.sched.com/event/Dsan/keynote-anatomy-of-a-
production-kubernetes-outage-oliver-beattie-head-of-engineering-monzo-bank)

~~~
maxehmookau
That sounds great. Gutted I won't be there, hope it's recorded!

~~~
dankohn1
Yes, we'll have it on the CNCF YouTube channel a few days later.

[https://www.youtube.com/c/cloudnativefdn](https://www.youtube.com/c/cloudnativefdn)

~~~
maxehmookau
Amazing, thanks!

------
swarnie_
The banks owners LLoyds PLC have an earnings call 26/04/18\. Will be
interesting to see if this incident is brought up.

EDIT: Sorry correction, looks like they sold the stake in 2015.

