
MemSQL is now free to use for databases with up to 128GB of RAM usage - bsg75
https://www.memsql.com/blog/announcing-memsql-free-tier
======
emmanueloga_
Another instance of the Wikipedia page for a product [1] being more useful
than the main site to describe it:

* MemSQL is a distributed, in-memory, SQL database management system.

* It is a relational database management system (RDBMS).

* It compiles Structured Query Language (SQL) into machine code, via termed code generation.

* On April 23, 2013, MemSQL launched its first generally available version of the database to the public.

* MemSQL is wire-compatible with MySQL.

* MemSQL can store database tables either as rowstores or columnstores (The OLAP vs OLTP part I guess).

* A MemSQL database is a distributed database implemented with aggregators and leaf nodes.

* MemSQL durability is slightly different for its in-memory rowstore and an on-disk columnstore.

* A MemSQL cluster can be configured in "High Availability" mode.

* MemSQL gives users the ability to install Apache Spark as part of the MemSQL cluster, and use Spark as an ETL tool.

The main value proposition seems to be the distributed nature, which probably
makes it easier to setup out of the box than, say, trying to setup a cluster
MySQL or PostgreSQL databases which are not "natively distributed". Also,
probably most useful when the data is "big enough" vs resources available on
any single server or when reliability is very important.

1:
[https://en.wikipedia.org/wiki/MemSQL](https://en.wikipedia.org/wiki/MemSQL)

------
no1youknowz
I've been using MemSQL in development since March 2017. I've seen it evolve
into one of the fastest databases for columnar storage and analytical
workloads. Having in-memory rowstore as well removed the need to have
Aerospike in my infrastructure and simplified the whole stack.

This announcement, that it's now free to 128GB pretty much saved me from
having to do a kickstarter to raise funds for my little SaaS project.

If anyone from MemSQL is reading, really thanks for doing this. I think it's a
very shrewd move. Should definitely see an upswing of potential customers
adding it to their stack and when they grow, become enterprise customers.

~~~
starik36
It says that he 128GB version comes with High Availability features. Does that
mean if I have 2 nodes, each one is limited to 64GB? Or each one can have 128?

~~~
rickn
I am the Director of Product Management for MemSQL.

The 128 GB limit applies to the whole cluster. So if you have two nodes in the
cluster they would each have to be 64GB or less. If you have four nodes they
would all have to be 32 GB or less. To have a highly available system we
recommend 4 nodes (a master aggregator, a child aggregator and two leaf
nodes). You can read more about the cluster architecture here:
[https://docs.memsql.com/concepts/v6.7/distributed-
architectu...](https://docs.memsql.com/concepts/v6.7/distributed-
architecture/)

~~~
openbasic
How does MemSQL reach consensus between nodes?

------
furicane
I tried to come up with a smart and polite comment but I can't. The target
audience for memsql aren't developers or engineers. It's the management that
has no idea about IT. I don't like closed-source solutions. I don't want to
book a demo. I want to be able to read the source. I want to install it, use
it, benchmark it, be sure that the results are 100% accurate. Sadly, I see
this post as marketing ploy and I can't find any nice words for this product.

~~~
loopp
Disclaimer: I am a product manager at MemSQL, so I may be biased :)

There are a variety of ways to try out MemSQL yourself such as installing on
Linux, Windows, Mac, AWS, etc, and maybe I am biased since I was an engineer
before I became a PM, but we optimize our product currently exactly for
technical people such as IT, devops, and of course, engineers. For a list of
installation guides, check this link out:
[https://docs.memsql.com/guides/latest/install-
memsql/](https://docs.memsql.com/guides/latest/install-memsql/)

Take a look at our docs (docs.memsql.com) and you will see that we all
actually are just a bunch of engineers and people with a technical background.
Are there certain technical topics you feel are unclear here? I'm also happy
to chat privately.

If you still feel this product isn't right for you, that is fine -- MemSQL's
focus on query speed may not be for everyone. However, with this release of
having a free product for people to try out, we definitely optimized exactly
for people that want to try the product out :). I'm actually surprised you
mention our product isn't for engineers/technicalPeople, because from our
field of view, we actually sometimes see MemSQL as _too_ technical, hence why
we focused on usability in this release, ha!

Hope that answers some doubts you may have -- thanks for the comment.

~~~
dman
Some feedback - I cant appear to find any way to install on Windows on the
following part of your website -
[https://docs.memsql.com/guides/latest/install-
memsql/](https://docs.memsql.com/guides/latest/install-memsql/)

~~~
moocowtruck
good catch i dont see it either..weird

------
quangio
What is the benefit of using MemSQL over some other free in memory databases
like Apache Ignite ? I see that they have better documentation and support
(edit: + competition on Codeforces which winners rarely receive their
T-Shirts). What about other things?

~~~
nikita
MemSQL CEO here. There are a few:

\- MemSQL is transactional and writes transactions on disk

\- MemSQL has an excellent implementation of SQL with mature query
optimization and query execution. And it get better every release. This is
from 6.5
[https://www.memsql.com/blog/6.5-performance/](https://www.memsql.com/blog/6.5-performance/)

\- MemSQL has in-memory and on-disk data storage so you can use MemSQL to
store petabytes

\- MemSQL has columnstores and vectorized query processing:
[https://news.ycombinator.com/item?id=16617098](https://news.ycombinator.com/item?id=16617098)

\- MemSQL supports geospatial, fulltext search, and json

\- MemSQL allows you to stream data from kafka in one command:
[https://docs.memsql.com/sql-reference/v6.5/create-
pipeline/](https://docs.memsql.com/sql-reference/v6.5/create-pipeline/)

~~~
noir_lord
Given the availability of open source solutions why would I in 2018 build a
critical part of my application on a closed platform?

Genuinely not intended as snark, I'm just curious why memsql is so compelling
that I would consider it.

~~~
dangoor
Isn't that what the previous answer was trying to say? Paraphrasing "MemSQL
has these features which we think Apache Ignite and others do not."

All things being equal, I agree that open source solutions are the best.
Things are just not always equal.

~~~
geezerjay
> Isn't that what the previous answer was trying to say?

Truth be told, the list if features is not very compelling. I mean, JSON
support is not a reason to pick a commercial dbms over a FLOSS one.

------
qeternity
Seems with this they've also deprecated the developer version which was
unlimited capacity but lacked enterprise features. Also, no mention of pricing
except to "contact sales".

~~~
no1youknowz
I asked back in 2017 and it was $25k.

This post 7 months ago [0] mentions the same.

It's probably still at this price, but if you are really interested. Yeah,
contact sales.

[0]:
[https://news.ycombinator.com/item?id=16617827](https://news.ycombinator.com/item?id=16617827)

------
shin_lao
I don't understand the high frequency use case they describe. High frequency
trading is something very different from "12,000 transactions a minute". What
is exactly the use case? Pre-deal checks? Post-deal checks? Book replay? Or is
it just a simulation? It's not very clear.

~~~
PeterisP
HFT can easily be 12000 transactions a minute with a requirement for each of
these transactions to be _very_ fast.

It's not (necessarily) about high throughput, but about low latency.

~~~
hendzen
No real HFT system is doing a __* database transaction __* in the critical
trading path. HFT systems are not built like web applications. They are
typically built as a tight event loop, reading market data packets directly
from the network card, doing a tiny bit of computation and then writing to a
userspace TCP stack for order entry.

I guess you could be using MemSQL for post-order or trade analysis but then it
would probably overkill since a lot of that can be done considerably slower.

~~~
shin_lao
Additionally 128 GB is just about 20 minutes worth of data.

~~~
jandrese
HFT firms have plenty of cash to buy the unlimited version.

~~~
hendzen
HFT ain’t what it used to be.

------
dominotw
[https://www.youtube.com/watch?v=_vloWsdPCDs](https://www.youtube.com/watch?v=_vloWsdPCDs)

CMU had guest lecture by memsql founder about the architecture.

Not sure how much of it changed in past 2 yrs tho.

------
pritambarhate
One thing I would like to see is the commitment to keep it free till X years
at least. OK, MemSQL has "decided" to give it for free up to 128GB of RAM
usage now. But they can "decide" 2 years later that they want to charge for
any MemSQL usage. Then what options the "small" businesses, who have adopted
MemSQL, have?

When it comes to commercial products, it is generally good to check if you can
afford their paid offering, then only make the tool core part of your
infrastructure. For databases, it's better to stick with fully open source
popular options from a long-term perspective.

------
cft
Why cannot I use free open source MySQL heap or memory storage engine? It also
provides clustering/replication

~~~
tintor
You can. :)

------
moolcool
This might be a stupid question, but what is the advantage of this solution
over an in-memory sqlite database?

~~~
manigandham
Distributed. Scalable. OLTP + OLAP queries. High availability. Very fast
performance for reads and writes.

They are entirely different systems. Sqlite is meant for self-contained
applications that need some relational data persistence with a single file for
storage, not for accessing as a central database with many clients storing TBs
and scaling across nodes.

~~~
riku_iki
> Distributed

is MemSQL shared nothing or shared everything, or can be mix of both?

~~~
manigandham
Shared nothing. It has leaf nodes that store data and do local processing, and
aggregator nodes that run queries spread to the leaf nodes and return the
results.

------
polskibus
What happens if I set hard limit on MemSQL process to use 128GB RAM and it
decides it needs more? Will it die demanding more memory or will it behave
like traditional rdbms and manage buffers to work within limits?

~~~
ruw1090
The best thing to do in this situation would be to set the maximum memory on
the MemSQL server via the `maximum_memory` system variable. The server will
then internally manage it's memory usage against this upper bound and fail
only specific operations which can't be performed without additional memory.

~~~
polskibus
What are the specific operations? Will another batch of INSERTs / UPDATEs
work? Does it matter if I use the analytical part of MemSQL ?

~~~
ruw1090
I mean that an operation that needs more memory than is available will fail,
but the server itself will remain operational. An simple example would be
loading more data into an in-memory table (row store) than there is available
memory. If you're using the columnstore then your storage won't be limited to
memory.

~~~
polskibus
That sucks, it's similar to Hekaton though. I was hoping there was some kind
of a fallback mode when the DB has to juggle memory buffers at a performance
cost.

------
lbriner
The bit I can't see on the FAQ is about how it is as fast as RAM but protects
against data loss using disk.

I think most engines guarantee Durability by assuming that once on disk, it
won't go anywhere but if it's in RAM, it is susceptible to power outage? If it
gets written to disk, it's not as fast as RAM?

~~~
janetacarr
It might use a Write-Ahead log like postgres to guarantee data integrity
between memory and disk.

~~~
nikita
That’s right and it writes it on two separate nodes if u enable high
availability

------
truth_seeker
Does the free version supports Horizontal sharding and async/sync DISK writes
for data mirroring from RAM ?

~~~
no1youknowz
From the announcement:

> You can do almost anything with MemSQL, using the free tier, that you can do
> if you have an Enterprise license, including capabilities and production
> use. The differences are that you can only configure the free tier of MemSQL
> to use up to 128GB of RAM usage, and support is only community support; for
> paid MemSQL support, you need an Enterprise license.

------
elvinyung
How is MemSQL for OLTP nowadays? Has anyone had any success
consistently/scalably using it for true HTAP-ish use cases?

~~~
manigandham
It's always been good at OLTP and great at OLAP. We used it for years as a
single HTAP solution during the 5.x versions for heavy adtech applications.

~~~
elvinyung
Thanks! Just curious, do you use the same tables for both OLTP and OLAP, or do
you have some kind of pseudo-ETL process to transform between OLTP and OLAP
schemas?

~~~
manigandham
The schemas are the same. Tables behave the same way logically, and its just
performance and physical semantics that are different.

We would have recent data in rowstore and move older data into columnstore.
You can easily join/union between both for queries. Also some constantly
changing data (like budget counters) would always remain in rowstore with many
lookups and updates per-second.

------
setheron
is memSQL a replacement for a full RDBMS in your stack ? Or just for analytics
-- siphoning off the data from your master.

~~~
CuriouslyC
Personally, I would not use memsql as a first line RDBMS. It lacks too many
useful features, and it doesn't return consistent results for all queries
(I've found inconsistencies when using group by with having, for instance).

~~~
gacba
Been using MemSQL for a enterprise-level financial services client since
mid-2017. We have this in production and are running it in a multi-TB cluster.
We've not seen ANY of these issues here and are heavy, crazy query users.
Their support has been nothing but on-the-spot and very helpful.

~~~
CuriouslyC
I totally admit that the inconsistency might be a result of misconfiguration
by maintainers at my company. That being said, being able to shoot yourself in
the foot so subtly and badly via configuration seems like a pretty strong
anti-feature.

~~~
manigandham
Since you aren't sharing any actual details and are switching between blaming
the product to blaming your devs and config (where no product can magically
keep you from breaking settings), your comments come across as rather
disingenuous.

Why not share a clear example of exactly what happened, or post on their forum
with details, so we can all judge for ourselves?

------
raitom
Is there a way on this release to create auto-scale rules like on RDS?

------
siscia
Sort of shameless plug, but if you are looking into such solution you could
have a look also at [https://redisql.com/](https://redisql.com/)

It is a redis module that embed SQLite, bringing on the table a lot of
advantages. Extremely fast and with it you can even upgrade your Redis
instance to be your only database.

There is not a huge company behind it, but I really do my best to support it.
I don't believe to have disappointed any of our users so far.

Also the tech documentation:
[http://redbeardlab.tech/rediSQL/references/](http://redbeardlab.tech/rediSQL/references/)

~~~
edoceo
This is awesome! Exactly what I was looking for. Basically a SQLite server-
node. Replace my homebrew Go thing.

~~~
siscia
I am very glad you find it interesting!

For any issues don't hesitate to contact me directly or through GitHub.

Also if you need it for some open source/(do the world a better place) kind of
project I am more than happy to provide the PRO version free of any charges or
obligations.

Of course it apply to anybody reading!

------
supperdave
Sweet!

