
HSBC moves from 65 relational databases into one global MongoDB database - truth_seeker
https://diginomica.com/hsbc-moves-65-relational-databases-one-global-mongodb-database
======
koheripbal
This title is misleading, and the article is missing a key piece of
information.

This is ONE system at HSBC - out of literally HUNDREDS (maybe thousands) of
applications. It's not as if HSBC is moving ALL of its applications to
MongoDB. HSBC doesn't have a single tech stack. They have thousands of IT
employees all using different tech, in different parts of the world, for
different departments. This might be as inconsequential as the system that
catalogues security camera feed URLs - or maybe the one that monitors remote
employee company mobile data usage - who the F knows, because the article
gives us zero information.

Account Management, Credit Cards, Mortgages, Security, Asset Management, HR,
Legal, Compliance, Regulatory, Risk Management, Trading, Operations, Building
management, Payroll, ATM comms, Clearance/settlement, Website, a million
different reporting engines, etc, etc, etc... and then each region usually has
its own application for each function - literally hundreds of tech stacks in
every tech you can imagine from DB2 Mainframe to Oracle to MongoDB. All banks
are like this.

This article is just vague PR and is just referring to one single group
consolidating their regional instances. It does not deserve HN attention.

~~~
tcgv
The article is very shallow on technical details indeed. It seems they are
trying to promote MongoDB as the solution to their failed attempt of
standardizing their application/data model:

> Historically, (...) HSBC did have an application core program environment,
> which had most of an application's core functionality. But it couldn't have
> a single programme environment running for all the countries, due to the
> differences in data models and databases.

> Another benefit is to use the same database for global data analytics and
> reporting. We don't need to translate into another data model or another
> database to run the analytics and reporting from that particular data.

So this "application core program" failed to provide an unified solution to
their data warehousing needs, and now they believe they can solve this problem
by unifying 65 databases into one MongoDB instance, "taking advantage" of its
schema-less model.

It looks more like a global storage service than anything else to me.

~~~
DenisM
So, problem: schema does not accommodate enough use cases.

Solution 1: refactor schema

Solution 2: abandon schema

The truth is that you can’t really abandon schema, it’s just spread through
the code base now, and any violation of the schema will blow up not when you
write malformed data but when you read it later.

~~~
thedudeabides5
Truth.

Anyone know any good tools to manage the evolution in schemas to address this
problem?

~~~
ric2b
Flyway is a good tool if you don't want to be tied to a specific language.

The community version (open-source/free) does have a few important limitations
that may or may not matter to you.

~~~
Trisell
+1 for flyway. Used it at two companies. Ability to fully reimplement schema
from any version to the most recent version is amazing. Also the ability to
review a schemas structure at a given git commit is super useful.

------
sohamsankaran
Jepsen's latest analysis of MongoDB --
[https://jepsen.io/analyses/mongodb-4.2.6](https://jepsen.io/analyses/mongodb-4.2.6)
\-- finds that it doesn't even preserve snapshot isolation when set at the
highest consistency level. This seems like a pretty terrible decision.

I've never wanted to short a company more in my life.

~~~
downerending
I got a credit card with them a few years back and immediately regretted it.
Their tech (website, security, etc) _seems_ okay at first glance, but almost
immediately it turned into agony. Even just logging in is a chore, and often
doesn't work. Their two-factor implementation is a mess.

In short order, I cut the card up. Still get sad emails (which they provide no
way to stop) begging me to use it.

Also, interviewed at MongoDB once. Couldn't shake the feeling of a cult, where
everyone was careful not to speak of the product's serious limitations.

~~~
drevil-v2
Haha I feel this. Did you also have the credit card sized token generator with
a keypad? How convoluted was that login process? Agony is the correct term!!

~~~
jen20
I had forgotten about this but the dumb 2FA generator is the reason I switched
away from HSBC for Citibank (UK) many years ago!

------
sitkack
This is like one step above csv files on an NFS share.

Could be a great way to wash some crimes off the books, "oh our DB failed and
our backups were stored in the database as well, oopsie".

In less than 5 years there will be an EU directive on how financial
institutions need to have immutable db architectures with full provenance.

~~~
echopom
> In less than 5 years there will be an EU directive on how financial
> institutions need to have immutable db architectures with full provenance.

I'm a former Entreprise Architect from Banking Sector in Europe.

To be clear , this will never happen.

There has never been any directive that has had a "concrete" impact on IT
Architecture of Financial Institutions.

Yet there has been dozens of regulations that urged banks to "simplify" their
IT Systems.

90% of IT Staff are Baby Boomers with no background in Systems Design or
Architecture , thus when new regulation comes in it follow this scenario 100%
of the time :

\- Find a vendor that sell a software that promise compliance with new
regulation

\- Find an integrator that promise integrations within the deadline

\- Integrate the vendors software in Banks legacy stack of 3000+ monolithic
apps

\- Send report to regulator saying they have "redesigned" their architecture
and made "investment" in order to take in account and that new regulation

Best examples of this is "PSD2" which has been the biggest fiasco of the
industry , has of the last year only 18% of banks complied with the
regulation.

France and UK said they would not "fine" anyone, because banks "aren't ready"
and made "considerable" investment in it.

Regardless of what HN thinks , you won't solve Financial Institutions Multi
Decade Legacy with a single Directive that would suddenly force them to use
"ImMuTABLe Db ARChITectuRES"

It Directives have never worked and will never work.

The only way to enforce anything would be to remove their IT systems
completely and have them use APIs provided by the regulators , and have the
regulator become the sole provider of "Financial System".

They wont let that happen.

~~~
YawningAngel
Some UK banks (at least Monzo, Bo, Mettle and Starling) have fairly sound data
models for their ledgers and implement stuff like PSD2 fairly effectively.
Presumably over time legacy banks will either catch up or incur the
regulator's ire

~~~
GordonS
Are you aware of any of these banks having published anything insightful about
their data models and relevant tech?

(I'm not trying to claim you don't somehow know this, I'm genuinely interested
to read about this stuff!)

------
bob1029
I have to say that on the surface, this sounds horrific.

\- MongoDB: Yes

\- Micro Services: Yep

\- Some attempt at a grand unifying model: Of course

That's pretty much a 10/10 shitshow as far as I am concerned. Mix in banking
regs & regional differences for afterburner on that money furnace.

But, in reality, I doubt that one of those 65 relational databases involves
the core customer/account/transaction processing facilities. These are
probably (hopefully) still running on an IBM system with proper ACID & uptime
guarantees. Anything a live transaction flow (especially credit/debit
processing) is going to touch could never be trusted on something like MongoDB
in this current reality.

~~~
sohamsankaran
My fear is that those IBM systems running Db2 or something were counted among
the 65. While it seems obvious to us that they're better off without switching
those, I know of several companies in financial services that should know
better that trust MongoDB for their core transaction flow.

~~~
guillaumei
The 65 relational databases are for the same application or similar deployed
in 65 countries...At least I read it in that way.

------
_Understated_
I've worked contracts in UK banks for a number of years now. Nothing on the
mainframes but I know some of the guys that do work on those and, despite the
constant stream of flavour-of-the-month tech that we talk about constantly in
places like HN, the mainframes work well and will not be replaced any time
soon.

The dev cycle for releases and whatnot is incredibly long due to testing and,
lets be honest, fear, in case something breaks but when it comes to battle-
tested technology, mainframes and DB2 are right up there.

I do know of some code that's over 40 years old and still runs at the core of
one of the big banks...

MongoDB, despite its capabilities, is... an odd choice imo! Not trying to
second-guess but if I was in that meeting I would have certainly raised an
eyebrow.

Good luck to them.

~~~
cm2187
The problem with 40y old code, is where I work we had to pull engineers from
retirement to make a regulatory change. In 10y, I am not sure we will have
that option.

~~~
aidos
I’d wager it would cost a whole lot less to train devs up on the existing
system than to rewrite. Without understanding the existing code, you’re going
to have a tough time replacing it.

~~~
Nextgrid
I’ve read that in some cases the initial requirements and source code was
lost, so making changes to the existing software becomes a reverse engineering
problem in addition to a “language proficiency” problem.

------
cranekam
I have no actual information about this migration but I wouldn’t be surprised
to learn that the databases that are being consolidated are for some minor
role or are separate instances with duplicate data for analytics or whatever.
It sounds wildly unlikely that they are going to smush together all their
customer accounts in one MongoDB instance. As well as being a huge technic
challenge I’m sure regulators would have something to say about this.

~~~
sjwright
This is very likely. Most banks have stupid-huge quantities of products and
services, each with extremely elaborate requirements and data storage needs.
“65 relational databases” might well only represent a few percentage points of
HSBC’s data footprint.

------
andrewksl
Sounds like a win-win for them. They get to market themselves as staying up to
date with tech (although the HSBC personal banking app I use indicates
otherwise). And when they "lose track of" the money they move for the cartel
and terrorists, they can blame MongoDB and spare some execs the jail time.

~~~
Charlie_26
I'm glad someone mentioned cartels and terrorists

------
bluestreak
I was there in Aug 2019 and I did hear them talking about mongo, There was a
programme at the time to "refactor" their estate, a part of that they wanted
to replace bunch of ad-hoc databases with clunky- or no data governance with
one (or fewer) central places, which will be under Data IT control. The
rationale was to reduce risk of data loss, leak or general business continuity
issue in what was 6-12 month project.

The cost of architecting relational model that would be cover for 65 mini
databases in a bank will be astronomical. It is far easier to setup no-SQL
entry point that would auto-conform for whatever requirements upstream
applications might have.

It is a technical win for Mongo, but I don't believe this is an attempt by
HSBC to get up to speed with database trends. HSBC are removing internal audit
strikes, nothing more than that.

~~~
simonswords82
We are working with HSBC and have not heard about Mongo being a thing despite
working on data with them. What location and department were you in if you
don't mind saying?

~~~
bluestreak
This was in London HSBC HQ, Equities

------
arcturus17
There’s been a lot of talk about the Jepsen-Mongo affair on HN lately, but the
problem I have with Mongo and NoSQL in general is much more basic - it’s
modeling relationships, which always end up appearing in every data model I’ve
ever designed.

I’m aware you can keep references from one doc to another but it always ends
up being a messy affair even at smallish scales, and it feels as if the
cognitive burden of managing these relationships ends up falling on the dev,
instead of being managed by the DB.

For those of you using MongoDB in large scale large production apps, is this
not a problem? Is your underlying business domain really non-relational, or do
you manage to comfortably run highly relational models on a doc-based DB? How?

~~~
thawkins
Mongodb now supports effectivly "Joins" in its aggregation framework, so it
can do some relational style data representation. The latest versions also
support transactions.

~~~
BozeWolf
Yeah but they do not recommend using that. I think the one nosql afficianado
in my team spoke with mongo support and they recommended reassessing our
document structure for it. (So a migration after all... )

Also in their blogpost [1], about $lookup they say:

We’re still concerned that $lookup can be misused to treat MongoDB like a
relational database. But instead of limiting its availability, we’re going to
help developers know when its use is appropriate, and when it’s an anti-
pattern. In the coming months, we will go beyond the existing documentation to
provide clear, strong guidance in this area.

[1] [https://www.mongodb.com/blog/post/revisiting-
usdlookup](https://www.mongodb.com/blog/post/revisiting-usdlookup)

------
petepete
Everything about this sounds mad. I'd have put referential integrity high on a
bank's list of wants.

Clearly they have some expertise, I'd love a more in depth look at how they
arrived at the decision.

~~~
ponker
These are definitely not source of truth customer transactions. Likely
auxiliary databases to power various experiences. Like, a customer walks into
the bank and the agent needs to know what services to upsell. Or the customer
logs into the website and is shown their current portfolio value.

~~~
cosmodisk
That kind of stuff could have been implemented with any large CRM
system(Salesforce,Dynamics,SAP) without any bigger issues.

~~~
threeseed
a) Salesforce is usually out of the question due it not being on-premise.

b) Dynamics and SAP are both ridiculously expensive especially since you would
be need to also buy additional software e.g. Windows to run it on. Plus of
course it's much harder to find specialist engineering talent who know these
products.

c) All three are far too slow both on the ingestion side e.g. millions of
mutations a second and on the read side e.g. feeding into a real-time ML model
for an upsell prediction. MongoDB is specifically designed to cater to both of
these technical requirements since you can mutate sub-documents and retrieve
the full document extremely quickly.

~~~
cosmodisk
a) I'm not suggesting this for banking transactions- that'd be crazy. However
customer service, interactions whilst at a branch, handling all sorts of
loyalty programs and etc. To my knowledge, Barclays in the UK use Salesforce
for some of these. b) Don't know SAP licensing model but I can't see a major
bank not being able to afford $100/month per user for Dynamics/Salesforce. c)
They do some offering for these kind of things,but I can't comment on how itd
work on such a scale.

------
0xy
Very rarely do headlines make me physically cringe but here we are.

~~~
threeseed
So you have no idea about the business and non-functional requirements or the
architectural or engineering considerations that went into this decision. But
yet you believe it's such a bad decision that you physically cringe.

Classic HN.

~~~
choeger
Did you ever actually read mongo db code? I had to, and only the drivers, but
when I learned that the C++ driver would just nope out of town in some cases
and literally exit() the application, I made the decision to get rid of that
stuff ASAP. There are other instances, pymongo cannot be forked, for instance.
Good luck using it inside celery. The connection string parsing is another
matter. All in all their driver's code looks good at the surface, everything
documented etc. But the comments are more or less superfluous and the design
is a mess.

------
realusername
I'm not in the banking industry but this sounds like the type of industry
where you really need strong DB schemas in place and not something mongo-like
...

~~~
jusssi
But the database doesn't need to be the component enforcing those schemas,
right? They could validate the schema of a document on storage and retrieval.
And keeping the schema version within a document would allow for multiple
schema versions to exist within a single collection, which then would allow
lazy migrations.

I'm not saying I'd choose Mongo specifically for this job (if for anything),
but the lack of DB enforced schema might not be the biggest problem here.

~~~
arcturus17
This is the problem I’m alluding to in another comment about NoSQL in this
thread. Why would you want to enforce the schema in code when the DB could do
it for you? It becomes such a mess and such a burden for the developer!

~~~
sjwright
That’s not always true. Often it’s not true. There’s a reason why we don’t
have file systems that guarantee all .jpg files are valid JPEG files.

~~~
WJW
Now I want a filesystem that guarantees all .jpg files are valid JPEGs.

~~~
sjwright
Until you need to do an emergency transfer of some recovered data volume and
there’s a few dozen slightly corrupted JPEGs scattered throughout a few
million files.

Also, I don’t think I’d want my filesystem deciding what qualifies as a valid
JPEG.

But yes, I can see the theoretical appeal.

------
pier25
After the latest Jepsen analysis of Mongo I'm surprised anyone would use it
for anything _that_ serious.

~~~
truth_seeker
I can't speak for Jespen. But I am using it from last several years and it
never ditched me even at the transaction data flow rate of 20K per second.

Database alone isn't the deciding factor, how you used it in your application
architecture matters the most. No database is perfect but some are more
flexible than others.

~~~
pier25
My comment was about failed transactions.

Are you using ACID transactions?

~~~
truth_seeker
Yep. MongoDB 4.2

In fact we replaced Kafka with Change Streams( little over 7k messages per
second) and ES with text search (2K queries per second) in MongoDB on 16 CPU
core nodes in our cluster.

I don't believe in any benchmark unless it matches the business use case and
development practices I use at work. I recommend the same to others.

------
AJRF
Building in plausible deniability so they can resume their HSBC money
laundering services with impunity?

Geniuses!

Edit: In case you missed the story
[https://www.bbc.co.uk/news/business-18880269](https://www.bbc.co.uk/news/business-18880269)

------
jauer
"is looking to" makes this sound aspirational. The absence of operational
lessons learned that would come out of a deployment make me wonder if this
actually happened.

~~~
hliyan
Thank you. The headline reads "HSBC moves..." which in normal news parlance
means that they've recently completed the move or the move is in progress.

------
executive
Sounds like the opening scene of a horror movie.

------
implying
A fortune 100 financial institution I once worked for had an initiative to
migrate their extremely large, fairly well normalized DB2 database to schema-
less dynamoDB.

All new tables had to go into dynamo, and if your service begins consuming an
old DB2 table, it had the job of migrating that data to dynamo.

The end result of this was applications that half pointed to DB2, half dynamo,
confusing new data with old, and tons of bugs relating to losing entries into
dynamo tables under outdated keys, not being able to clearly match up customer
data, and myriad problems that were never a consideration using a relational
database. Needless to say, I pulled all of my assets out of this institution
as soon as I had the chance.

~~~
threeseed
I've spent decades in enterprise space and unfortunately developers often
don't have a clue how actually things work.

Companies like IBM specialise in royally screwing companies through their
licensing deals and with their legal enforcement teams. They are nothing like
dealing with a normal startup vendor or with someone like AWS. They play hard
ball.

So sure you can complain that they should've just stayed with DB2. But usually
that means paying tens of millions which then increase each year since they
know you won't migrate.

That's why companies take drastic decisions and move to the cloud even if it's
technically not the best option.

~~~
tluyben2
But drastic can be going to Postgres/Mysql instead of db2/oracle; Dynamo (by
the sounds of the case here) is more on the insane scale than drastic.

------
9nGQluzmnq3M
The article makes this sound way bigger than it is. HSBC has _thousands_ of
applications and probably tens of thousands of databases. Migrating one app to
use a different DB model means very little for the big picture.

------
tannhaeuser
Mongo works well for storing chunks of JSON from transactional browser apps
(which I guess is what HSBC is targetting), doesn't require you to have custom
endpoints, has a good replication/HA story (on paper at least, if your devs
know what they're doing and don't use defaults plus multiple nodes with CORS,
etc.), good mindshare among webdevs, and is even the de-facto API for JSON
stores with AWS' "DocumentDb" and Azure's Cosmos DB being drop-in
replacements. But using it for banking/backoffice apps would be pure madness.

------
chrisdbanks
The whole premise of this seems wrong. I've worked on large financial systems
that have happily supported different rules for different countries using a
relational database. The database was never the issue, it was how to handle
testing and feature enablement. This seems like the wrong cure to the problem,
and a cure that will end up causing more problems than it solves. i.e. a hack.
I'd love to hear more about the rational. I just don't understand it.

------
kristaps
Each and every one of these MongoDB migration rationales contains some
variation of "schema design is hard, let's go shopping."

Granted, there are plenty domains where you can get away with having a blurry
schema with the occasional bug creating a new localized data anomaly, but
banking, where such things lead to customer trust erosion - I just don't get
it.

------
jpswade
Oh my. This seems like a really bad idea. I'm calling it now, this will go
wrong.

That "Micro Service Instance" is not a microservice, it's an Enterprise
Service Bus (ESB), aka the Egregious Spaghetti Box.

That's aside from the fact that relational data belongs in a relational
database.

I really can't imagine it being a single instance, that just does not make any
sense.

------
NicoJuicy
When do they go live and how can you short them?

Ps. I think they never heard of database per service when looking at their
microservice graph. Lol

~~~
robjan
HSBC's own website lists dozens of exchange traded warrants with various
strike prices and gearing. Need a brokerage with access to the HK market
though.

[https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005...](https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005/type/put)

I'm not sure about their choice of Mongo but the initiative seems like it
should be helpful for customers like myself who have accounts in a few
countries. We can access them all from one app but the experience is
inconsistent at the moment. I'm cautiously optimistic.

~~~
NicoJuicy
Seems like the use case for graphql, not mongodb

------
prostoalex
Any time I have to use an HSBC online product (they throw in an occasional
promotion for their credit cards and savings accounts) it feels like a trip
back to the 90s - popups and extra windows on different domain names that are
slowly "loading..." information for trivial operations like scheduling a
payment on a credit card.

------
rmoriz
I think this isn't primarily about a database migration/consolidation and
specifically not about MongoDB. IMHO it's about cutting cost and complexity by
building a single world-wide software platform. Of course this also means that
local specialities and local teams will be questioned and probably be made
redundant.

------
will_raw
I recently came to this article[1] where Guardian switched to PG from MongoDB

[1] [https://www.theguardian.com/info/2018/nov/30/bye-bye-
mongo-h...](https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-
postgres)

------
FHermisch
Microservices and NoSQL - the wonder-weapons against every IT problem, aren‘t
they?!

I really have to pivot my business - new name will be „The Microservice-NoSql
Consulting Company“ and we will sell tailored versions of the „Microservice-
NoSql Strategy“! (Changing names and colors of boxes for 2000$ per day)

------
speedgoose
The whole thing sounds like a terrible idea. Why would you want to put all the
data from at least 65 different applications in the same table? In MongoDB? To
save cost?

I simply don't get it. It doesn't make sense to me, but perhaps they know
something we don't.

~~~
threeseed
360 customer view as an example. If you are feeding a real-time decisioning,
support, advertising system then you need all of the customer features
accessible as fast as humanly possible. Ideally a consistent O(1) time.

Or it could be a real-time fraud system where you need every bit of customer
data in one payload to feed into an ML model.

You can't do any of that if you are having to do multiple joins across
federated databases.

~~~
speedgoose
I would collect data and push events to a new system, not migrate 65
relational databases to a single nosql database.

------
jarym
I don’t see how one global mongo is going to help solve any of their issues.
Maybe they just want to bury their issues under the carpet a little longer.

I also hope they’ve configured their mongo correctly since out of the box
Config is not appropriate for many use cases

~~~
malikNF
I want to doubt a billion dollar company like HSBC will have engineers who
will install things just out of the box and leave it without properly
configuring it. But, they are proposing to use Mongo just almost a month[1]
after a report came highlighting several issues with the product, ESPECIALLY
for use-cases like a bank that requires a very high level of data consistency,
so your point might actually be valid.

Would really be interested to know how something like Mongo ends up getting
seriously considered for something like this, I mean the whole decision
process that goes in to selecting something like this.

[1][https://news.ycombinator.com/item?id=23285249](https://news.ycombinator.com/item?id=23285249)

~~~
jarym
I did a contract at HSBC in the past. Some teams are good but there are people
who will install things out of the box and leave it.. even in a billion dollar
company like HSBC.

------
xmorse
Hacker News has become just “mongodb bad, sql good”

------
29athrowaway
This sounds problematic:

1) mongoDB does not offer the guarantees required by a bank.

2) Application developers using mongoDB rarely protect themselves against
NoSQL injections. Most of them are unsuspecting that they even exist.

e.g.: What is your user id? {$ne: null}... whoops, now every user record is
returned.

3) They better make sure they use the Decimal128 type for their currency
fields, and make sure that the value is not being casted to float on the
client. e.g.: on a JS client, every number is a float.

Unless they've taken the necessary precautions this is a disaster waiting to
happen. Hopefully other banks do not do the same.

~~~
mushi
> what is your user id

I don’t think you actually tried it. It returns user who’s id is “{$ne: null}”
(string)

~~~
29athrowaway
It depends. Sometimes you can trick an application to parse that as an object.

------
niffydroid
I would suspect they'd keep the core money transactions in the existing DBs,
but use mongo for less important things like sending junk mail, customer
support sessions.

------
tluyben2
I have seen many rewrites for financial institutions in my life; one of the
biggest I had a good insight look at during the entire project was a Tandem
Nonstop Cobol rewrite to Java; the famous 10s of millions of lines to nice
clean Java code. Many of them never work; this one was, if I remember
correctly, upwards of 50m Euros and was thrown away because it did not work in
the end. I think the Tandems are still running.

------
mariopt
I can see how they would use in some parts: analytics, internal reports, etc.

It is interesting that you can use Mongo and keep the data in a geographical
location and still be able to run a global query:
[https://docs.mongodb.com/manual/tutorial/sharding-
segmenting...](https://docs.mongodb.com/manual/tutorial/sharding-segmenting-
data-by-location/)

------
BiteCode_dev
It they just put commercial details follow up data + some stats in there, it's
ok.

It is very infrequently updated, integrity is not at too much risk. In case
it's lost, business is not impacted too hard as long as you can find contact
details eslewhere.

But the replication story is good, and if they got tons of different schemas,
it will make their life easier.

So yes, for transactions it would be bad, but maybe for this particular use
case, it's ok.

------
ezconnect
I think they’re just using it for document storage not customer and financial
data. Document storage and retrieval is a big cost for large institution.

~~~
niffydroid
We found mongo expensive for documents. We had simple documents basically
saying X sent email Y to Z. The mongo db was cloud hosted and was over 500gb
and costing nearly 3k. Moved them into S3 with a simple SQL dB to look up
reference IDs. Certainly reduced the cost!

------
seanhunter
To put this into context - I met with a senior tech person there (one layer
below the group CIO) a few years back and he said they had something like
40000 distinct IT systems globally. So replacing 65 of them with one MongoDB
instance is something like 1/8th of one percent of their systems.

------
akho
65 databases unified globally, for HSBC’s global presence, is ~2 databases per
country. This is, probably, a minor trial effort for one of their services,
not the core system (which is un-unifiable due to banking privacy laws in
place) or any kind of cross-function unification.

------
trollied
This article is misleading. They have migrated some of their databases to
mongo, not all of them

------
anonu
I need to LOL a little bit at all this, the headline, the article.

First off, how would such an article even come about? Feels like more of a
recruiting/marketing piece.

The old story at hsbc was that the ATM software hadn't been updated in twenty
years, cause stability....

------
neximo64
If someone was involved could you share details on the rationale behind this?
I used to like mongo but has it changed.

I remember specific flaws/hacks with bitcoin websites coming down to mongo
race conditions/inconsistency issues.

~~~
brabel
The article explains quite in depth the rationale behind this. Basically, they
think it's too difficult to manage their current relational databases
differently in each country, but they need that due to the different
regulations and particulars on each country. With one global DB, they hope
they can have one single application for all countries, with country-specific
things hidden in only a few sub-documents.

_ It's now a one service environment, one database and one execution path for
all the countries. This is made possible because of MongoDB's document model
and the ability to map all the different table requirements for each country
into a single collection, using sub-documents. Everything is simplified into
one collection using country specific identifiers._

How they will address race conditions, eventual consistency and so on is not
addressed at all though. I guess they trust Mongo's new transactions will take
care of that (at which point, it would be nice to know whether that actually
worked and what the performance will be when compared to the old relational
system).

------
murukesh_s
Relax HN guys.. this is the reason for this:

"Local requirements for each country will be built into the application, but
there's no need to maintain separate data models or separate databases
anymore. We could easily design the global data model and database using the
MongoDB JSON schema model. That brings data from all operating countries into
one database and the application can run on just one database. Which is a lot
of reduction in resource and maintenance cost."

Is there any Database that can do this? other than MongoDB? I am hearing about
data corruption in MogoDB, I believe that is due to bad, out of the box
settings, which I believe they would have mitigated when using in cooperation
with MongoDB company (which I am sure they are).

~~~
BozeWolf
Whats the exact advantage, one can wonder. Each country still needs its own
application. With its own models. With its own migrations, or at least ways of
dealing with missing or mixed data types.

What if... those models map to sql database schema’s? wouldn’t that be
magical? Not since 2008.

Whats left is maintaining multiple databases. That sucks indeed.

Sql databases support bson as well these days.

What they probably want (or did) is make a base framework, suitable for all
countries. And have each country develop its own stuff on it. No need to use
mongo for that though.

One use case i can imagine is that forms and input just change and that old
information never will be compatible again with the newer forms. Mongodb
serves as a giant more or less queryable data bin. Then one can ask the poor
dba to lookup something for a client with “client_id”: xyz. Then return the
raw bson/json output. Might be sufficient.

------
asplake
Never mind the technology, a single data model would be a feat in itself.

~~~
ilaksh
I think the single data model in this case is probably something like
requiring a couple of fields and everything else is whatever you need. Heh.

------
ycombonator
HSBC IT is a hollowed out outsourced shop. Just like Avis Budget, Disney,
Walmart and Walgreens. They will all move to MongoDB data stores sooner than
later.

------
buboard
If you 've ever used HSBC web banking u ll know it's one of the worst things
mankind ever created. I m not sure if this is good news for MongoDB

------
nwmcsween
Is it common that a sufficiently large doc db is just an un-normalized rdb?
I've seen swagger uis of doc dbs that make me wonder wtf/why.

------
jsjohnst
What’s the over/under odds on betting about how long it’ll be until it’s
accidentally exposed to the internet and hacked?

------
addicted
There’s a lot of complaining here but MongoDB has been used by startups for
like a decade?

I don’t think I’ve seen a single HN article complaining about actual data
loss. That would be something that would get upvoted immediately.

So what gives? Especially since earlier and older versions of Mongo apparently
had far less data stability.

I’ve probably read far more complaints about Postgres in HN articles
(difficulty setting up, poor defaults, etc). And Postgres may not even be as
popular as Mongo. So what gives?

------
edf13
I’m in the UK and glad I don’t bank with HSBC

------
ionwake
Wasn’t there a problem where mongo would literally lose records under high
loads ?

There was an article on it about it a few years ago.

Did they fix that?

------
coding123
This is probably some aspect of HSBCs computing needs, likely less than 2% of
the data the store.

------
kchoudhu
Yeah, this is about par for HSBC IT.

------
dark_light
Hiding your complexity behind a (poorly designed) layer doesn't make it go
away.

------
sidcool
How do they get ACID guarantees and schema enforcement?

~~~
threeseed
MongoDB supports transactions and you enforce schemas in code instead of the
database.

Pretty common these days to just rely on Git and not DDLs.

~~~
thawkins
You can also tie json-schema descriptions to collections and have it throw
validation fail exceptions on writes that fail validation.

------
kgc
And I am moving all my money out of HSBC...

------
Keyframe
Modern take on shredding papers

------
surajs
Yeah this can't go well...

------
historyremade
Behind Bad Indian Coder ...

------
predictmktegirl
And it’s gone.

(Sorry)

~~~
egeozcan
While I'm generally against getting to meme-y in HN, this is spot-on, adding
to the already-long list of ways banks can make your money disappear and/or
make unreachable.

Reference:
[https://www.youtube.com/watch?v=_nVk25ZvTkU](https://www.youtube.com/watch?v=_nVk25ZvTkU)

------
jbverschoor
shocking

------
blaufast
That was a mistake

------
jonnypotty
Lol. Just lol.

------
svrtknst
Uh, how does this work with eg GDPR or data protection laws that require all
customer data to remain within the country of operation?

------
subins2000
isn't this a bad idea

------
cm2187
And of course they will expose it to the web without authentication!

------
ngcc_hk
Old guys ... what dB they use in London and Hong Kong. Are they still use
mainframe?

------
aogaili
Lots of negative comments about MongoDB for folks who don't use it, but
perhaps give NoSQL a chance and keep an open mind? there is no need to
structure the data into tables and schema does not need to be enforced at the
DB level.

Google has been running its entire infrastructure on a NoSQL database for
years.

Perhaps folks need to keep some open mind and see if that works instead of
dismissing a new trend?

~~~
arcturus17
“Google has been running their entire infrastructure...”

I’m skeptical about this - Google’s _entire_ infrastructure is so large that
surely they must be using a wide variety of db technologies?

I took a databases course that did mention that part of the Google _search_
infrastructure runs on a tailor-made NoSQL technology, but it wasn’t really a
doc-based approach, it was more akin to a relational DB but with flexible
columns (can’t remember the details sorry).

At any rate I don’t feel that this is any vindication for NoSQL, as Google
builds much of its underlying tech from scratch and its hardly comparable to
the out-of-the-box solutions that the rest of us, even massive corporations
like HSBC, have to work with.

~~~
aogaili
[https://en.wikipedia.org/wiki/Bigtable](https://en.wikipedia.org/wiki/Bigtable)

"Bigtable development began in 2004[3] and is now used by a number of Google
applications, such as web indexing,[4] MapReduce, which is often used for
generating and modifying data stored in Bigtable,[5] Google Maps,[6] Google
Book Search, "My Search History", Google Earth, Blogger.com, Google Code
hosting, YouTube,[7] and Gmail.[8]"

"Bigtable is one of the prototypical examples of a wide column store. It maps
two arbitrary string values (row key and column key) and timestamp (hence
three-dimensional mapping) into an associated arbitrary byte array. It is not
a relational database and can be better defined as a sparse, distributed
multi-dimensional sorted map."

