

Making MySQL 5,888.6x faster. Yep, you read that right! - titusblair
https://medium.com/bigobject-digest/making-mysql-5-888-6x-faster-yep-you-read-that-right-5ddbf4ce192b

======
bandrami
Why are you using a locking read on a million+ row table?

Why are you doing a naive join between million+ row tables?

Is the point here that your product is better than a badly-implemented
database on its own?

~~~
titusblair
Q: Why are you using a locking read on a million+ row table?

A: BigObject is row based storage as of current implementation. There is a RW
lock on the table to coordinate access. Going forward as BigObject adopts
advanced storage schemes i.e. column based storage, indexed row group, etc
concurrency will naturally improve. For this iteration, we felt that most
query are READ/SCAN access, and a simple RW lock is sufficient for the
performance gain we observed.

The bottom line is, BigObject is a collection of data structures that could
operate on structured data sets in an efficient manner. It is the foundation
that we build on to provide new ways to address data analytic problems.

Q: Why are you doing a naive join between million+ row tables?

A: Here we are demonstrating a heavy operation that will scan the entire data
set. You could improve the performance if you know first hand what subset of
data you will touch, and leave the rest behind, but so would BigObject's
performance.

Q: Is the point here that your product is better than a badly-implemented
database on its own?

A: MySQL, PostgreSQL and other database systems are robust and well-maintained
solutions.

Developers today develop applications with the premise that a database server
must be present in their service, but could be using it for the wrong purpose.
BigObject is a middle layer for extending the capabilities and performance of
databases whose role today are skewed towards general storage and retrieval
while the original idea was handling transactions.

Arguably, you could tell people to use the right tools for the type of data
and inbound frequency they are dealing with, but as a service provider and
partner in Big Data Analytics, we have a responsibility beyond RTFM and snide.

We hope this helps address your questions.

Note: Questions answered via Jeff our product architect here at BigObject.

Additional Material if you are interested:

[https://bitbucket.org/macrodata/benchmark-
rdbms](https://bitbucket.org/macrodata/benchmark-rdbms)

------
devhead
Do you have plans to support replication from sources? Having to manually load
data via a query takes away real time use and takes away the gains of the
query in bigobject.

------
sigmonsays
No explanation what so ever on how you did this.....

~~~
titusblair
Let me know what explanation you are looking for and I would be happy to go
into more detail.

------
chenyuanjen
Seriously? 5,888x or 5,888%?

~~~
titusblair
5,888x...here is the math I used:

MySQL Time: 588.86 seconds BigObject Time: 0.1 second

588.86/0.1 = 5888.6 which is 5888.6x :)

Hope this helps!

