
How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster - gesaint
https://pingcap.com/success-stories/lesson-learned-from-40-k-qps-and-20-billion-rows-of-data-in-a-single-scale-out-cluster/
======
chucky_z
Calling cockroachdb "not open source" is a stretch.

[https://github.com/cockroachdb/cockroach](https://github.com/cockroachdb/cockroach)

(The title is also wildly inaccurate to what's in the article, and should just
have the articles title)

~~~
jinqueeny
Thanks for the comment! CockroachDB is now under BSL and CCL licenses.
Different users have different opinions but you are right, we need to be
accurate about it. Please give me some time to consult our user and see
whether we can update this case study.

~~~
jinqueeny
This case study has been updated: [https://pingcap.com/success-stories/why-we-
chose-tidb-over-o...](https://pingcap.com/success-stories/why-we-chose-tidb-
over-other-mysql-based-and-newsql-storage-solutions/)

------
_tkzm
Pingcap is loosing hundreds and thousands of potential customers and millions
of dollars by not having a single-binary product. The current setup is over-
complicated and confusing and I would not touch it with a 10ft pole, as they
say. That's why it's worth rewriting the app from mysql syntax to postgre
syntax and use cockroachdb more than spending dev _, ops_ , cloud* hours on
making legacy code work with this thing. But they'll never learn. It's not
like this thing came out just yesterday. I have been keeping my eye on this
since it came out and I think it is maybe even superior product to cdb and
possibly the best in the category, but again, i am getting head-ache just
looking at the setup docs.

~~~
dongxu
PingCAP CTO here, thanks for these comments! We highly appreciate all the
feedback!

First, it’s true that the current setup/deployment of TiDB is not easy. This
is something that we're making serious moves to improve. For example,

A. We provide Ansible playbooks to simplify the deployment and rolling upgrade
for on-prem users;

B. We built and open-sourced TiDB Operator ([https://github.com/pingcap/tidb-
operator](https://github.com/pingcap/tidb-operator)) to enable TiDB on
Kubernetes. We are working on a fully managed service in the public cloud
(coming soon). Whether it is one binary or multiple binaries, it’ll be all
transparent at the user level;

C. We are improving the default or self-adaptative parameters and are
continuously refining the configuration process;

D. We are also trying to reduce the number of components. For example, the new
version of CDC is implemented directly inside TiKV.

E. We are developing TiOps tools in a single binary to improve the operating
and maintaining experience of the cluster. A fair amount of customers around
the world are using TiDB in their production environments and we are making
sure they get our help when needed in the setup so it would not be a deal-
breaker.

Second, TiDB’s multiple-component or highly-layered architecture is
challenging for deployment but the benefits are also obvious:

A. The separation of the storage and computing layers makes it flexible and
agile to scale/upgrade each layer as needed. Different layers need different
types or different number of hardware resources. If the computing resources
become the bottleneck, users can scale the SQL layer by adding more TiDB
instances in real-time; if the bottleneck is the storage layer, they can
easily add more TiKV instances to increase the storage capacity.

B. As is known to many that we have donated TiKV to CNCF last year. We are
fully committed to the open-source community and would like to see TiKV be the
building block and foundation of the next-generation infrastructure. For
example, we are happy to see some community users sit Redis on top of TiKV,
and we ourselves built the TiSpark
([https://github.com/pingcap/tispark](https://github.com/pingcap/tispark))
connector to run Apache Spark on TiKV.

For more thoughts about this, please take a look at my blog:
[https://pingcap.com/blog/9-whys-to-ask-when-evaluating-a-
dis...](https://pingcap.com/blog/9-whys-to-ask-when-evaluating-a-distributed-
database/#2-why-separate-sql-from-storage)

Feel free to give more feedback on
[https://github.com/pingcap/tidb](https://github.com/pingcap/tidb) and our
community Slack channel
[https://pingcap.com/tidbslack](https://pingcap.com/tidbslack). We're glad to
discuss more with you on this issue!

~~~
_tkzm
all those points sound more like quickfixes for bad architecture design. I've
been there, done that.. and i've learnt from it. instead of fixing the initial
problem, you are just throwing more code and complexity over it. which costs
you money and time. i think you made a BIG mistake byt going polyglot and not
sticking to single a language which now prevents you from merging the code
into a single binary in cost-efficient way.

if you decide to go with the single binary approach, i am curious to see if
you decide to go with Go or Rust :)

just fyi: cdb went with Go but their storage layer ended up being rewritten
with C, so they too are not much different from you, with the exception of
being able to run a single binary via Cgo which you cannot do with Rust.

~~~
gravypod
> all those points sound more like quickfixes for bad architecture design.

As an on looker the architecture of TiDB is anything but bad. It doesn't fit
into the "run one thing for everything" bucket of software because it's
designed to be horizontally scalable. Every part of the design is scoped well
enough to only do the bare minimum for it's goal. I have no doubt that
internally at Google things like BigTable look very architecturally similar.

> Big mistake going polyglot.

Why? Because you can't make a single binary? That's not really a use case that
makes sense for a cloud native DB. Everything is going to be a container
anyway, so it doesn't matter what artifacts are included. Also the individual
implementation layers of TiDB are individually useful (TiKV).

Making a single binary has no benifits unless you're running your hardware in
a pet mentality which manually curated software. In this case a horizontally
scalable db will not help you.

Also, providing an operator is a far cry from a "quick fix". I applaud them
for doing this because it essentially removes the operation burden of running
their DB.

~~~
_tkzm
[https://www.youtube.com/watch?v=pWdd6_ZxX8c](https://www.youtube.com/watch?v=pWdd6_ZxX8c)

------
rubyn00bie
Nice article, I just wish they went into more detail as to why TiKV was chosen
over other distributed stores like: Cockroach, Yugabyte, or Foundation. It'd
also be nice to know the size of those 40bn rows is it 10TB or 100TB?

~~~
Caitin_Chen
For database selection, there is a blog post about why choose TiDB over
CockroachDB and other MySQL-based solutions: [https://pingcap.com/success-
stories/why-we-chose-a-distribut...](https://pingcap.com/success-stories/why-
we-chose-a-distributed-sql-database-alongside-mysql-to-break-down-data-silos/)

~~~
jiggawatts
It's interesting to see how crazy complex some of these scale-out solutions
end up being, especially when they start off with something like MySQL and are
forced to maintain compatibility while scaling far beyond what it was designed
to do.

You've got to wonder if simply sharding by either player or game would have
worked well enough, even with MySQL. Worked well enough for Blizzard for over
a decade!

Similarly, I wonder if a single active write server paired with a handful of
readable secondaries could have taken this query volume in its stride if using
something like MS SQL Server on high-spec bare metal kit. Think In-Memory
processing on dual AMD EPYC 2 servers with 128 cores per box.

They mention billions of rows and terabytes of data, but I've heard of similar
scale systems as far back as over decade ago! The Australian phone company
Telstra did all their billing in a single IBM DB2 database, for example, and
the incoming data in that system is just as real time.

I'm probably wrong, but this smells like the company missed some basic
optimisation opportunities somewhere...

~~~
adev_
> You've got to wonder if simply sharding by either player or game would have
> worked well enough, even with MySQL. Worked well enough for Blizzard for
> over a decade!

With all the respect I have for blizzard, I would not take old Blizzard as an
example of scalability handling. I remember a time where there authentication
system was well known to give up under load after every update.

