
Re-examining our approach to memory mapping - bluestreak
https://questdb.io/blog/2020/08/19/memory-mapping-deep-dive
======
hinkley
Many of the problems with staging environments are preserved if you remove
staging environments:

How do I test with realistic data but _never_ risk my customer's data in the
process? Either way I'm going to make a copy of real user data. In one case
it's much harder to have shared references to live data than in the other.

There are lots of failure modes for rehearsals that can take out the cluster.
If you're doing it live, how do you know you don't have a request loop or just
introduced an n+1 fanout in a previously well-behaved service? One that can
take out said service and parts of its dependency graph? What if I write a
query that crashes processes?

On the other hand, it's 'common sense' for staging to use similar hardware to
production so that discrepancies don't catch you by surprise. But common sense
isn't common, and people can rationalize the hell out of not spending money on
precautions, and so they never are equivalent if they are partitioned. Which
also makes thing like perf analysis very low fidelity.

More likely that nobody is right, everything is a tradeoff (sucks), and which
pain points can your organization collectively stomach?

------
bluestreak
Hi HN,

Author here. QuestDB is a fast SQL open source database for time series. About
a month ago we launched on HackerNews [1].

Today, I am excited to announce QuestDB 5.0.3 [2]. This new release includes
our changes in memory mapping strategy, giving us better performance, as well
as some major changes to the Postgres wire protocol support.

In this blog post, we explain our journey to improve QuestDB's performance,
especially regarding memory management. Relying as much as possible on the
kernel and avoiding extra layers turned out to be very successful.

Thanks

Vlad

[1]
[https://news.ycombinator.com/item?id=23975807](https://news.ycombinator.com/item?id=23975807)

[2] [https://questdb.io/getstarted/](https://questdb.io/getstarted/)

