
Building a Modern Bank Backend - antoine1fr
https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/
======
raesene6
I'd read this article and something about it bothered me, took a while to put
my finger on it.

To me creating a back end banking system using a load of software that's very
new and still very much developing is a risky move. Once they're in production
bank back ends tend to be around for a LONG time (in my experience in UK FS
there were a load of old back-end systems that were maintained out of
necessity as they were too expensive to re-write and the product they
supported was still in use)

To me banking back-end systems should be boring from a tech perspective. Use
well known components where the limitations and downsides are well known, and
also if it were me I'd use components that I could get a support contract for.
You don't want to be relying on open source support at 3am on a Sunday when
you hit some bug or other and the FCA is breathing down your neck to get your
transactions processed in a timely manner.

~~~
dsr_
At 3 AM on a Sunday, you discover a critical bug in a library provided by a
third party.

Commercial support: send email. You get an automated message back with a
ticket number. Around 6:30 AM you get a call from Delhi that satisfies the 4
hour response SLA, and the support analyst says that they will look into it.
At 11 AM you get a request for a dump and certain logs, some of which you had
already sent them...

Open source support: after getting zero google results, you look through the
source code while running a debugger. It takes you two hours to find the place
where it makes an assumption that a signed variable will never be negative, an
hour to test your fix, and five minutes to send a patch to the mailing list.
It is coming up on 6:30 AM, and you conference in the two people who need to
approve emergency changes in order to get that approval. (This is a bank. You
can't unilaterally make a production code change.) Everything is deployed
before 8 AM.

~~~
generalk
Or, alternatively:

3 AM on a Sunday, you discover a critical bug in a library provided by a third
party.

Commercial support: You call them. It takes an entire day of awkward back and
forth emails and phone calls, but you eventually get an engineer to send you a
a patched library for testing. It works.

Later, when auditors ask you about vulnerabilities, you can ensure them that
you track $VENDORS supported long-term releases, and your contract has a
security and support SLA.

Open source support: you find nothing on Google or Stack Overflow. You haven't
looked at this source since your initial code review (you're a bank, you
review all third-party code in use.) It's 3am, you're tired, and none of this
is making sense. You've posted in the projects IRC channel but nobody's
listening, and there's no good responses to your posts to the mailing list.
You have to pull other team members out of bed, and together you get a
reasonable test and patch implemented, which for some reason the maintainers
aren't accepting quickly (if ever) and now you're running a local fork of your
library. Hope your internal security auditing practices are solid, because
there's no guarantee you didn't accidentally create a vulnerability during
your pre-morning hackathon.

\---

I don't love commercial software typically, but the support and auditing
benefits are _frequently_ the overriding factor when choosing for a regulated
industry.

~~~
raesene6
some good points there, but LOL about all 3rd party code being audited.

~~~
tluyben2
We had to: not one line of code not audited, 3rd party or otherwise. Frontend
and backend.

~~~
raesene6
Wow, can I ask was it a big stack? And when you say audited, do you mean
manual code review, static analysis or something else? How far down the stack
did it go?

I'd be genuinely interested to hear, as it's something I see bandied around a
lot but extremely rarely followed through on as code reviewing a modern stack
is a huge proposition, and manual code review is very expensive/time
consuming.

------
jwr
> we’ve built our backend as a collection of distributed microservices

I hope I will never have to rely on a bank that uses this architecture. It
seems they are vastly underestimating the complexity of distributed systems
and the range of their failure modes. It's all cool if you're building a web
site, less so if it's about your money and being unable to pay at a gas
station.

~~~
simonhorlick
Having worked in the industry, most top tier investment banks run on
microservices, some for over a decade. Just because microservices is the new
buzzword doesn't mean it's suddenly a bad, or risky idea. I don't know about
the backend state of modern retail banking, but I'd be surprised if this
wasn't also the case.

~~~
collyw
The one I used to work for was all heavyweight relational databases.
Admittedly this was nearly ten years ago.

~~~
threeseed
Most enterprises/banks still have their Teradata or Oracle EDW at the heart of
the company.

It's just that rather than have a single monolithic app in front of it they
have broken it up into a suite of smaller, independent services. And many of
those services will have their own database which contains a transformed,
optimised slice of the total EDW data.

------
dx034
>Large internet companies like Amazon, Netflix, and Twitter have shown that
single monolithic codebases do not scale to large numbers of users, and
crucially large numbers of developers.

How is that true? These companies have shown that you can scale well if you
don't have monolithic approaches, but that doesn't mean that you cannot scale
if you have one. Just because someone is successful by using approach B
doesn't make approach A unsuitable.

Quite the opposite. Some banks handle more than a billion transactions per day
on monolithic infrastructure (think of big investment banks with a sizeable
amount of trades per day). It's very costly, but it does work reliably.

EDIT: One thing I don't get: Most banks use a central database somewhere
because you need to have accounting values that you can rely on (e.g. how much
money you have on your account). How do they do that? They must use some
synchronization with locking that works across all services which rely on
them, but how will that scale well when they get dozens of transactions per
second? How do they guarantee the integrity of their data?

~~~
obeattie
I'm the post's author. The key point I was trying to make was not that
monolithic systems cannot handle the throughput, but that they become more and
more difficult to extend and maintain as they grow. None of the companies
mentioned started out with a microservices architecture – they moved to them
(at great expense) when they found development difficult to scale.

~~~
dx034
Thanks for the answer. I'm not against your approach, just curious if it will
work. How do you handle synchronization for large numbers of users? Many bank
related transactions require some kind of locking (getting account balances,
authorizing transfer, ...), doesn't that lead to a high probability of
deadlocks at some point?

------
DanielBMarkham
Intuitively I like this -- hits all the good buzzwords and has a ton of good
things going for it. I was especially impressed that they're possibly heading
towards a Simian Army situation. To me this is the only logical way to deploy
infrastructure in a modern setting. I'd sure like to look under the hood,
though.

Most interesting was _"...Deploying Kubernetes in a highly available
configuration on AWS is not for the faint of heart and requires you to get
familiar with its internals, but we are very pleased with the results..."_

Wonder how much pain and suffering there was getting this to where they wanted
it.

The only part I was concerned about was 150 microservices so early in the
game, moving to message queuing while still very small, and the RPC/locking
stuff. Smells like over-engineering. But hey, I'm not there. There's a ton of
stuff that goes into making decisions like that. Thanks for the story!

~~~
collyw
"hits all the good buzzwords"

You see that as a good thing?

~~~
DanielBMarkham
Not by itself. By itself it would be counter-indicative. But assuming good
intent, I felt the buzzword-compliance indicated an understanding of the
deeper issues involved.

It's a difficult situation. I don't want to spend all day reading an article
defending CoreOS. But if you throw that buzzword out, I'm assuming you've had
some good discussions about your OS selection and could provide that depth if
needed. Same for the others. They're not right or wrong choices, but I'm
assuming they're there not because the team wanted to be trendy, but because
there was real architectural need.

If we're discussing architecture, by necessity we're going to have to skim
over a lot of stuff. I was okay that given the team's desire to prevent a
single point of failure that the pieces listed worked towards that goal.

It's very easy to put too much stuff into a system. It's also very easy just
to do trendy stuff without thinking through the consequences. I'm giving them
the benefit of the doubt until shown otherwise -- and at the end of my comment
I was clear about what some of my concerns were.

------
mping
Geez, all the haters here.

I would argue that as long as you don't actually lose any customer's money,
its OK to screw up once in a while. My e-bank site its down once a week for
deploys or so, my bank's mobile app didn't to certificate pinning, and they
are still operating.

Actually I believe CAP isn't actually that relevant, because most money is
transferred and there are double ledgers for that, which for sure isn't fully
consistent.

As for the paid software and support, its a crap reason. The support guy isn't
omnipotent, at best he's seen the issue you're having and will help you
faster. If you have paid software and you have downtime because of them, good
luck trying to get money from them.

And there's nothing wrong with microservices, as long as you don't need to
call a bunch of them to update my balance. I don't care if I try to change my
address and the service is unavailable, or if I don't receive a weekly
notification. Its much better than having e-bank site downtime because they
want to deploy a new version of the webapp.

As long as you maintain account balance integrity, I'm sure you guys will do
just fine. All the best

~~~
raesene6
it's nice to hear that you're a flexible and accomodating customer who's not
adverse to companies having some IT issues.

Unfortunately banks in the UK are regulated by the FCA and they are more than
a little picky about that kind of thing. As an example RBS were fined £56
million for an outage a couple of years ago
([https://en.wikipedia.org/wiki/2012_RBS_Group_computer_system...](https://en.wikipedia.org/wiki/2012_RBS_Group_computer_system_problems))

~~~
mping
That didnt' seem that it was a minor outage. Anyway, these things apparently
happen even without microservices or kubernetes. Time will tell.

------
lordnacho
Sounds very impressive. It'll definitely scale if they can somehow build it
with 3 coders.

I would have thought you could start with a bog standard CRUD setup. How many
customers are they serving at the moment? How many queries is one customer
going to make each day?

What part of a bank are they actually trying to make? Just current accounts?
Or multi asset market making? The requirements are quite different.

Also, on the business side, why does the UK market need another bank? It's a
totally commodity service. Why would I want to use this bank? Do they offer
something better?

~~~
dbbk
Just current accounts. And yes, the UK does need a new kind of bank. Monzo's
USP is that their technology is modern and built from the ground up. For the
end user, this means you get push notifications of card transactions
instantaneously. You can freeze and unfreeze your card instantaneously. The
Monzo app uses your phone's location to determine when you travel to a new
country, and doesn't block your card for suspicious activity as a result. It
gives you realtime currency conversion for your transactions. The app allows
you to budget your money, and categorise where you spend it. And more.

-edit-

Oh and I forgot to add, they offer a complete API, so you or third party
developers can build on top of your financial data. No other bank offers this
today.

~~~
saiya-jin
> Oh and I forgot to add, they offer a complete API, so you or third party
> developers can build on top of your financial data. No other bank offers
> this today.

this part ain't completely right - all current banking backends on the market
offer instant messaging integration, otherwise they wouldn't be bought. it's
not straightforward API calls directly in the backend (which some would prefer
to not expose directly anyway). a bit of glue code is required but this is how
in the moment these systems evolve to conform to ever changing regulations for
example. otherwise banks would have to shut down

~~~
dbbk
Publicly accessible API then.

~~~
saiya-jin
this is not an advantage then, you don't need to be a seasoned architect to
see all kinds of potential issues with publicly exposing these APIs, even with
proper security. I mean this is the core of every bank, not some application X
storing some small subsets of crucial data.

Best and easiest security to achieve and maintain is no public access at all
(something like physically separated networks vs authenticated ones)

~~~
jhuckestein
All banks in the EU will have to offer APIs in 2017 because of PSD2:
[http://www.out-law.com/en/articles/2015/january/key-
features...](http://www.out-law.com/en/articles/2015/january/key-features-of-
psd2-and-what-they-mean-for-the-payments-industry/)

The alternative to open APIs is that third parties ask customers for online
banking passwords. This practice is significantly less secure than, say,
OAuth2 or similar and is generally against a bank's TOS, which makes the
customer liable.

------
cm3
Genuine questions:

Is it commonplace for a bank to be a British Ltd, and are they part of a
separate banking emergency fund to accommodate for that because they have a
banking license?

Also where do I have to look to find a real world address for Focus FS Ltd
(company behind Mondo)? Looked around the internet after finding nothing on
monzo.com. I know startups do not list contact info these days, but whois info
is also opaque. I'm asking this because they advertize themselves as a modern
bank, but I cannot find a real world address after 5 minutes of searching.

~~~
grahamel
You can find Focus' details at companies house,
[https://beta.companieshouse.gov.uk/company/09446231/filing-h...](https://beta.companieshouse.gov.uk/company/09446231/filing-
history)

Bit more about Mondo/Monzo on wiki
[https://en.wikipedia.org/wiki/Monzo_(bank)](https://en.wikipedia.org/wiki/Monzo_\(bank\))
(they recently change the bank name)

Initially they only offered prepaid cards while they waited for their banking
licence. They're not covered by the Financial Services Compensation Scheme
(FSCS) at this point but will be when they start to offer current accounts.

They've recently got their (restricted) banking licence so they'll be part of
the FSCS fairly soon. Part of their restrictions is that they can only hold up
to £50,000 ($65,000) in customer deposits, the FSCS covers up the 75k in
savings so customers money is protected. ([http://uk.businessinsider.com/uk-
digital-bank-mondo-lays-out...](http://uk.businessinsider.com/uk-digital-bank-
mondo-lays-out-plan-to-get-full-banking-license-2016-8))

------
dharma1
Oh, they're called monzo now? I guess changing one letter was the cheapest in
terms of logo redesign :)

I think what they are doing is bold (especially in terms of security, I think
their head of security role is still open?) but ultimately the right thing to
do for a truly disruptive banking startup.

I don't think it's possible/easy to bring the features people want from modern
banking software front end (free text search through transactions etc), with a
legacy banking backend.

~~~
ryanhuff
Why not? I've seen it done.

------
gaius
_Large internet companies like Amazon, Netflix, and Twitter have shown that
single monolithic codebases do not scale_

Except Google _does_ have a "single monolithic codebase", so actually, the
jury's still out on that one. Personally I wouldn't bet the farm either way at
this point.

~~~
rco8786
Heh. No. They keep the code in a single repository but it by no means a
monolith.

Kubernetes is FROM Google. Along with some of the most advanced distributed
systems implementations in the world.

------
yid
> Large internet companies like Amazon, Netflix, and Twitter have shown that
> single monolithic codebases do not scale to large numbers of users,

And yet, both Facebook and Google have large, monolithic codebases.

------
fixxer
There are tons of opportunities to improve banking and credit, but these
overarching statements about new technology yay and micro services good and
batch processing verboten... Ugh, total jargony bullshit.

Building the best X is about using the best tools to do what X needs. Not
everything craves massive scale and a TON runs better on a well provisioned
RDB!

Want to revolutionize banking? Fine, but don't forget the fucking CAP theorem
when you're storing my cash.

~~~
jhuckestein
I work at Monzo.

I agree that using microservices and kubernetes is _not_ what allows us to
provide a good user experience. We could provide the same user experience if
we had built everything on rails and postgres. In fact, that would have been
much easier.

The reason we are investing so heavily in a rock solid platform is that we
want to still be able to offer the best possible user experience in 10 years
time. This is what we mean when we say we want the platform to be
"extensible".

Many large banks IT systems are not extensible, in the sense that it is very
expensive to make changes. Take, for example, the ability to freeze a card in
the app at the tap of a button. A friend at RBS told me that they considered
this feature many times, but it took a long time to work out which 20 IT
systems would need changing and some of them had been under change freeze for
a few years. So eventually the idea was discarded as too expensive.

The freeze card feature would be easy to implement for a startup, regardless
of their stack. The key is to still be able to implement such a feature easily
in 10 years time :)

~~~
raesene6
Interesting, although I'd suggest that the problems for RBS in implementing
such a system are more a reflection of the complexity and age of their systems
than than technology used.

I'm interested that you say you were looking for "rock solid" when you're
using what is a pretty new stack. CoreOS and Kubernetes are very new products,
innovative, exciting yes, seems early days to describe them as "rock solid".

Out of curiousity did you consider using a more conservative stack (e.g.
ASP.NET ?)

~~~
mstade
> Interesting, although I'd suggest that the problems for RBS in implementing
> such a system are more a reflection of the complexity and age of their
> systems than than technology used.

And I'd suggest you're wrong. Neither complexity or age of a system is usually
the issue. In fact, I'd say technology is almost never the real issue, but
rather organizational red tape. Those owning the various systems often operate
as more or less autonomous entities, largely with their own change policies
and process, and I'd be willing to bet this will be a bigger hurdle than
anything technical. Also, there will be a fair bit of politics involved.

Perhaps a small outfit as monzo can move fast and be nimble now, but if you
reach the point where you have multiple system owners you will inevitably run
into these problems, and they have almost nothing to do with technology. FUD,
often fueled by compliance requirements (whether actual legal compliance or
perceived such) also plays a large part.

For me personally, I sometimes wish the red tape would just go away, and at
other times I'm glad it's there to save us from some of the madness I've seen
people try to get deployed.

Source: I have worked with and in top tier financial institutions for about
half a decade at this point, including RBS.

~~~
raesene6
Interesting perspective, I too have worked with large financial institutions
(for 15 years if that matters) including RBS :)

Organisational red-tape is obviously part of the problem but that's part of
what I meant by complexity. Change in large FS companies comes slowly because
of their size and complexity both organisationally and technologically.

Once you get a complex network setup and a load of systems that aren't
designed to work together (which tends to happen over time as technologies
shift) implementing new layers across the top of them takes time.

One of the problems with changing existing systems is that impact to
production is something which has to be avoided, so changes to those systems
have to be planned carefully. Now it's arguable that some organisations take
that too far, but it's not fair (I think) to suggest that all of the change
controls in place in those companies are "FUD".

~~~
mstade
I don't think I suggested that all change controls amount to FUD, certainly I
didn't mean to anyway. What I did say was that it plays a large part, which is
evident in how little many of the decision makers actually know about what a
change is, what it means, and what repercussions it might have, so it's easier
to err on the side of caution. The answer, of course, is to do more research,
but that's more expensive and harder than to just say "no."

My experience is that whatever technical issues may arise really aren't that
big of a deal, provided you do the proper research to realize what a change to
the system actually means. Significant effort, perhaps, but not particularly
difficult from a purely technical standpoint. But it usually means
coordinating with different parts of the organization, which can be
excruciating, and more often than not has little to do with technological
aspects and more with territorial politics and process. That's not necessarily
a bad thing, like I said I've seen it stop some really horrid stuff make it
into production, but it can also introduce so much friction that you give up
before even having started.

I don't rightly know where the balance lies. For a small outfit like monzo it
makes sense to be nimble, but for a huge organizations like RBS, UBS, JPM,
HSBC etc., there are so many things at play that being conservative isn't
necessarily a bad thing. Plenty more degrees in this hell than people
generally believe there is, that's for sure.

------
ddorian43
We knew from the start that we would be the next ((facebook+google-apple) ^
spacex), that's why we started with distributed microservices, (you can't
ignore docker these days on hn frontpage, you HAVE to use it), golang +
node.js + kafka combo. We're not in beta yet and we have 150+ services, isn't
that insane right!

We need a protocol for rpc, http+json was the slowest one --> clear winner.

Everytime we need a new function, we can make it a service and make it
distributable on our polyglot (because why not!) docker infrastructure behind
http load balancers + json data-type and make it infinitely scalable!

~~~
obeattie
Author here. We are in beta and have about 40,000 cards issued to customers in
the UK, with a waiting list of about 200,000. I think we have good reasons for
making the architectural decisions we have made, some of which are laid out in
the post, but perhaps I did not explain them in enough detail.

For example, we chose HTTP as our RPC protocol because it is widely supported
in many languages, and has mature support in Finagle. HTTP/2 fixes most of the
performance issues with HTTP, and we have options (eg. to use mux) if we still
find there are performance problems – although at the moment, we are quite
happy with performance.

~~~
ddorian43
How many of those 150+ services depend on other services ?

~~~
obeattie
Almost all of them interact with other services to do something or other.

------
asimuvPR
Why not java?

