
Sharing SQLite databases across containers is surprisingly brilliant - jcberleur
https://medium.com/@rbranson/sharing-sqlite-databases-across-containers-is-surprisingly-brilliant-bacb8d753054
======
fgonzag
I'm amazed at the fact that more people don't push mostly static or rarely
updated data to local sqlite instances or flat files.

In my line of work, the goverment distributes a daily file with all those tax
ids that can emit tax deductible invoices. Since it has every tax id along
with additional information, the file is quite large (30Gb of XML split into 4
files because of the format). This file has to be downloaded and used daily by
all our front end servers.

Every Engineer immediately proposed throwing the data on the Galera cluster
that we're currently running. They didn't see the problem of slowing down the
main database every day with a huge update cycle (you could diff it previously
but it's still a large update). We'd also be querying a network storage for
basically static data, which I thought idiotic.

I proposed having a simple daily task in a job server that downloads the data,
parses it and writes it to a sqlite database. It then pushes this file to all
of our frontend servers. The frontend servers simply reference that file. It's
a lot faster, especially since the whole thing can fit in memory when its
compressed binary data (drops down to about 12G), and you simplify your reads.

I completely agree with the statement made by the author too:

"Aside from the quirk of sharing SQLite across containers, it delivers
extremely well on one core idea: the thematic shift of complexity from the
data plane read path to the control plane write path."

