
Scaling SQLite to 4M QPS on a Single Server (EC2 vs. Bare Metal) - based2
https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-qps-on-a-single-server/
======
justinclift
Title should probably say "(2018)".

Also, it's already been submitted a bunch of times:

[https://news.ycombinator.com/item?id=23070888](https://news.ycombinator.com/item?id=23070888)

[https://news.ycombinator.com/item?id=22856746](https://news.ycombinator.com/item?id=22856746)

[https://news.ycombinator.com/item?id=20778477](https://news.ycombinator.com/item?id=20778477)

[https://news.ycombinator.com/item?id=16118776](https://news.ycombinator.com/item?id=16118776)

------
watsocd
It's not just the server cost that matters when you compare the cloud to a
bare-metal server.

What about:

    
    
      Networking equipment
      Backup power - UPS and generators
      Internet backbone - Speed as well as reliability
      Cooling and other infrastructure costs
    

Then there is capital lock in risk. Once you buy the above hardware, you are
probably stuck with it for a few years.

If you have stable load now for 25 big bare-metal servers, it may be cheaper
to go bare metal. Otherwise, there is a lot more involved in the decision than
going to dell.com and pricing a server.

~~~
nicolaslem
Bare metal != On premise

There are plenty of hosting providers that will provide everything you mention
plus the actual servers for a monthly fee without requiring you to buy
anything.

~~~
CharlesW
> _Bare metal != On premise_

Trying to be helpful and not pedantic: "On premises" or "on prem" are good
choices here, while "premise" means something else (e.g. "The _premise_ of
moving off- _premises_ is that we can pay a little more in order not to worry
about stuff that isn't our company's primary business.")

~~~
ksec
This is not the first time I have seen this but it is actually quite commonly
used / mistake. I wonder if 20 years from now the word premise would have a
different meaning.

~~~
CharlesW
> _This is not the first time I have seen this but it is actually quite
> commonly used / mistake._

Very common, for sure. And although I think it's worth professionals being
aware of the difference, I tend not to bother with less technical folks.

> _I wonder if 20 years from now the word premise would have a different
> meaning._

Given that "literally" is now often used to mean "figuratively", it's
absolutely possible!

------
brandmeyer
In the list of features which are somewhat custom to their use case, they
listed "Disable POSIX advisory locks".

Turns out this is available stock right now as the "unix-none" VFS
implementation. We're using it in an almost-but-not-quite-POSIX RTOS that
happens to be missing advisory locks.

See also [https://sqlite.org/vfs.html](https://sqlite.org/vfs.html)

------
nijave
They're comparing bare metal to a virtual machine here, aren't they? Of course
I'd expect some pretty noticeable performance differences there. Not sure they
were offered at the time of the post but AWS also has bare metal that seems
like it'd but a much more fair comparison

Cloud VM vs on-prem bare metal doesn't seem like a very fair comparison

~~~
jjeaff
Why not? As long as you are comparing your total cost all in on both services.
But did they even say on-prem? I was assuming colocation.

------
chrisweekly
Impressive! Thanks for sharing: it's well-written and detailed, with
meaningful data and clear dataviz for comparison.

------
gigatexal
Making random() deterministic defeats the purpose of random no?

~~~
gopalv
The randomization they are working around is most likely

> WHERE indexedColumn > RANDOM() LIMIT 10

That can be bound on RANDOM() performance or we can rely on a seeded "non-
random" PRNG to produce a random selection vector without running random() for
every row.

If that has a syscall into /dev/urandom for every row, then that sucks for
sure & is not representative of the actual intention.

In Apache Hive, we have a similar argument about unix_timestamp(), which was
made deterministic, but stateful (MapReduce failure tolerance demands that if
a task fails while running a query like > unix_timestamp() + n, the next
attempt will produce an identical output, even-though some time has passed
between the attempts).

So for that, we do the unix_timestamp() replacement per-query, instead of per-
row.

I assume that's what was done here.

------
xwdv
I know people are itching to jump to the conclusion that bare metal is
ultimately way cheaper than cloud, but really, it isn’t.

At the end of the day, we want operating expenses, not capital expenses.

And if you purchase a ton of bare metal and suddenly your business needs
change it will not be easy or quick to move that much metal or install new
metal in place.

Stick to the cloud.

~~~
christophilus
Eh. Most businesses don’t really need big scale. I build exclusively in the
cloud, but the app I maintain could easily run on a single $2k server, with
the possible exception of image resizing.

~~~
BubRoss
You would have to be resizing a tremendous amount of images to overwhelm a $2k
server.

~~~
paulryanrogers
Or a very inefficient implementation.

~~~
ies7
For comparation,

2 years ago we build thumbor-like apps in python using vibora and opencv. A
single aws m8 large($80/month?) can serve hundred millions images resizing in
a month.

At that time we build it with only 2 people, copy pasting code from internet
in a few days and vibora’s github readme also state that its in alpha release.
With $2K server, you need to cheat to make those inefficient implementation :)

Side note: Several weeks later I found out about thumbor and replace our
‘emergency work’ with dockerized thumbor

I also tinkered with thumbot in aws lambda. But the cost raised to $500/month.
I think we did something wrong with aws rekognition.

