

RBS IT cockup: This sort of thing can destroy a bank, normally - arethuza
http://www.theregister.co.uk/2012/06/25/banking_fail_rbs_natwest/

======
tomgallard
Some more information about the tech behind the failings here:
[http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_wen...](http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_went_wrong/)

TLDR: Looks like a failure in their CA-7 batch processing systems, which
compounded so they weren't able to back out. Lots of the experts in RBS's
implementation of CA-7 have been made redundant, and been replaced with
offshore-expertise

I'm not going to question the proficiency of the offshore talent, however, I'm
sure most on HN would agree that there's great value with legacy systems in
not getting rid of the people who have been working on them for the last 20
years, and know all the idiosyncrasies and foibles.

Lots of the UK banks still use highly complex decades old main-frame based
systems which interact with hundreds of other systems. It feels to me like the
systems have got so complex, that its hard to see a way of replacing or
rewriting and simplifying them.

~~~
timrobinson
Specifically, it looks like a manual error during a deployment:

[http://www.guardian.co.uk/technology/2012/jun/25/how-
natwest...](http://www.guardian.co.uk/technology/2012/jun/25/how-natwest-it-
meltdown)

"It seems whoever made the update to CA-7 managed to delete or corrupt the
files which hold the schedule for the overnight jobs, so they did not run, or
ran incorrectly"

~~~
Zenst
The aspect that they didn't export the job queue prior to update is something
that cross's the border of neglegence with both feet.

Still can't understand why a bank that had people who made bad deals and lost
money causing the bank to partialy fail are kept inplace and paid bonus's and
the IT people who did there job well are replaced by cheap labour external to
the country and this is at a time when there going on about TAX evasion, this
too me makes no sence and is why I don't run banks :|.

~~~
Isofarro
IT is seen as an overhead, and as something that doesn't contribute directly
to profitability. When compared to dealers, dealers bring in the profit, IT
slurps it away.

~~~
Zenst
Very true sadly. Don't see many outsourced HR and accounts sections though.

Sad part is that when it goes wrong it does effect the balance sheet. Even
sadder is how they internaly cost IT and do it wrong. Remove the IT and see
how many people/time is needed to do the same job and that is the true
cost/potentual impact of IT. Sadly though that is never done and only comes to
light when things fail and then they blame IT and not the effects of seagul
managment, budget cuts etc.

~~~
gadders
Oh man. Please, please, please outsource HR. Preferably to Mars. Biggest waste
of space in the whole company.

------
andyking
I have been affected by this over the past week or so. I've been on holiday,
and came back to a house with no fresh food, and no working bank account to
pay for any, so I've been eating processed shite out of the freezer like a
student for four days, and it's making me feel a bit poorly, not to mention
the freezer is starting to look a little bare. I haven't had breakfast for
days, and I have about two meals left if I stretch it. Surely this isn't
acceptable in 2012?

Perhaps a little less seriously than the effect on my health from a lack of
fresh food, I was paid my monthly salary yesterday, and that hasn't appeared
in my account. I'm assuming that it'll appear in the next few days, but if it
doesn't, I can't purchase the car I'd arranged to buy on Friday. It's all a
massive balls-up and affecting my life in so many random ways.

Can any British readers recommend any alternative banks? I'm based in the
centre of Sheffield, so I have the choice of pretty much all of them.

~~~
pja
The Co-Op? (Although rumour has it that they're going to completely replace
their computer systems either this year or early next year so that might be a
case of jumping from one basket case to another!)

RBS appears to have fouled up the scheduling of their overnight batch run in a
way that meant large tranches of transactions simply weren't going through the
system. Fixing the problem has taken days (which _may_ be down to poorly
managed outsourcing) and it's then taking even more time to catch up with the
backlog of unregistered transactions, because they can only fit so many extra
into the overnight run.

If you've got a pay slip or something, then get yourself to an RBS branch and
they'll give you cash to tide you over in the meantime. Don't starve yourself!

~~~
andyking
I can't use the Co-op. They won't let me have an account because I apparently
have a bad credit rating. (The real reason, I expect, is because I have _no_
credit rating as I've never applied for a credit card or loan, or been late on
a bill.) They're probably overdue a computer system overhaul as they've
recently taken over several other banks and building societies, notably the
Britannia where I had a few friends working.

My pay now seems to have appeared, and my second account doesn't have £0.00 in
it any more, so I've got my fingers crossed that I'll be able to go to Tesco
tonight. I might go to the branch and get cash though, just in case I get the
dreaded 'card declined'.

~~~
pja
It's worth having a credit card just for the cushion it gives you in exactly
these kind of circumstances tbh.

~~~
jmaskell
It's also worth using a credit card for the added protection and other
benefits (see <http://www.moneysavingexpert.com/cards/credit-card-tips>).

Just make sure you pay off the balance in full each month so you never get
stung by interest.

------
arethuza
For anyone interested, the BBC documentary on the near failure of RBS in 2008
is pretty interesting/horrifying:

<http://www.youtube.com/watch?v=k1cRXipswfA>

Alistair Darling's autobiography describes some pretty scary moments -
apparently when the Chairman of RBS phoned him (in a bit of a panic) they only
had enough cash to last, at most, 2 hours.

[NB Slightly amusing to be posting this to HN when I am maybe 200m from the
old RBS HQ]

------
hopeless
This has also been affecting customers of Ulster Bank in Ireland. The biggest
problem appears to be catching up on the backlog of transactions:

"The bank processed more than 110,000 transactions with a value of €500
million over the weekend" -
[http://www.irishtimes.com/newspaper/breaking/2012/0625/break...](http://www.irishtimes.com/newspaper/breaking/2012/0625/breaking6.html)

Does anyone else think that 110,000 transactions shouldn't take a whole
weekend? Of course, we don't know exactly what a transaction involves but I
would have expected this to take only a few hours at most. But, of course, I'm
assuming modern hardware. It seems as if they just don't have the spare
capacity to catch-up after an outage like this.

I wonder if this will make the conservative bank IT departments even more
conservative or will they go in the opposite direction and realise the
benefits of modern technology (performance, scalability and a pool of
knowledgeable talent)?

~~~
dasil003
If it involves talking to other banks around the world then it seems
reasonable.

~~~
yardie
What? It's not like they are sending the transactions by post anymore. This is
a colossal screw up and 2 weeks is unacceptable. I wouldn't stand for 2 weeks
of downtime on my shitty little hosted website, 2 weeks from a bank would send
me to the moon.

~~~
dasil003
I'm saying 110,000 transactions could take all weekend if they're dealing with
latency from a gazillion other banking system.

------
highlander
Attention entrepreneurs: there is a real opportunity, at least in the UK, for
a bank built the right way from the ground up. I'm talking about contemporary
systems, real product ownership in the technology group where IT isn't just an
outsourced cost, automated testing and continuous integration in the systems.
Most of the efforts at 'new banks' in the UK have been built on the
foundations of ancient, creaking mainframe systems. There is a lot of
important logic in those legacy systems for sure, but I think progress could
be made in building fresh solutions by starting at the relatively simple end
of things e.g. customer setup, identity verification and basic savings
accounts. Could built out from there to offer more complex products. The
upsides of a fresh approach: lower maintenance costs, higher reliability, more
operations can be carried out in realtime etc.

------
TomGullen
I despise Natwest.

First problems with them at University, they gave away free 5 year railcards
to all students who opened a student bank. No one I knew got one, you only got
it if you went through a laborious complaint process. Once I realised I did
not have my card after several months I complained, and they refused to issue
me one because I waited too long even though it was their fault for not
dispatching it in the first place like they promised.

Second problems was with their disgusting ethics when I fell upon hard times.

Third time was when I went travelling and they lied to me straight up about
the fees I would be charged using my card abroad, which caused problems. (They
did settle with me afterwards and give me some money back, but it was another
laborious mission).

Throughout, general appalling service speckled with useless calls from
customer service research departments asking me how long I had to queue for in
my recent visit and other pointless questions. Flacid attempts at 'caring for
customers'.

I'm at a stage now where I can move from them, and will do soon. I'm always
happy to spend time telling people about my experiences with them and my
recommendation to completely avoid them if possible.

I know banks are not exactly beacons of ethics and good business in society.
My expectations for banks is low because of this yet Natwest consistently have
managed to go to new depths.

------
jacques_chester
Bankers do sometimes forget that these days, their entire business is an
elaborate front end for a general ledger program.

I predict that banks which see IT as a key source of competitive advantage
will prosper in the coming decades. Banks which see it as a cost centre will
eventually be slugged with debacles like these.

Of course, such self-serving aggrandisement as this is not without its own
problems. Banking software is enormously complex and is meant to be utterly
reliable. Two things that rarely mix well.

Add in the storied difficulties of running enormous software projects and you
begin to understand why most banks just don't want to hear about how the
transactions occur, just that they occur.

------
yread
sorry for OT, but this

 _The Register uses cookies. Some may have been set already. Read about
managing our cookies.

Please click the button to accept our cookies. If you continue to use the
site, we'll assume you're happy to accept the cookies anyway._

Is this really the way to handle the new cookie regulations?

------
maxer
One of the real impacts of this was holiday makers abroad as reported by the
bbc

<http://www.bbc.co.uk/news/uk-northern-ireland-18590028>

