
Kyoto Tycoon in 2019 - coleifer
http://charlesleifer.com/blog/kyoto-tycoon-in-2019/
======
HardLuckLabs
I keep hoping that the creators of fallabs make a triumphant return. They did
post a small bugfix for a build error not too long ago, so the lights aren't
completely out.

We used kyotocabinet exclusively for a project that ran for 3 years with ~1
billion txns/day in a big AWS cluster. Totally bomb-proof and a great piece of
software. There's no reason not to use it today, really. Tycoon, the server,
is good, but the library kyotocabinet is multithreaded and highly performant.

The flexibility of B+ tree or hash type stores, on-disk or in-memory with the
same interface is extremely useful for building distributed systems. Very
stable under load, and no data bloat as records pile up.

edit: nobody should feel guilty for using stable software with a reliable
track record. Just because a project isn't being printed on tee-shirts anymore
and breathlessly fawned over during the free happy hour at conferences doesn't
mean that it isn't still a perfectly useful piece of software.

~~~
diekhans
TK doesn't compile with modern C++ compilers; it has been a real headache for
us. It needs some TLC if it is to be used.

------
stakhanov
KyotoCabinet is one of the most important weapons in my holster as a data
scientist.

With that kind of a specialty, there are essentially two different kinds of
jobs / work environments. One is where it's all about infrastructure and
implementing the database to end all databases (and "finding stuff out" is a
mere afterthought to that. The other is where 100% of the focus is on "finding
stuff out BY NEXT WEEK" with zero mandate for any infrastructure investment of
any kind. When I find myself in the latter kind of work environment and I need
to quickly get sortable and/or indexable datastructures of any kind, then a
key-value store is the way to go and KyotoCabinet is a really good one that
I've used LOTS and it has never let me down.

Just don't let your boss find out about it if yours is of the pointy-haired
variety. If he finds out that it's an open source project that has seen its
last commit some 6 years ago and has been abandoned since, he will be less
than pleased. -- Personally I find that's a really bad argument. It's feature-
complete w.r.t. anything I've ever wanted to do with it, and its feature set
is actually way richer than most of the younger alternatives that are still
being actively developed (because they are still in _need_ of very active
development, and aren't nearly as well battle-tested). Plus what's the worst
that could happen? That you will one day want to move to an environment where
it will no longer work. Well in that case it shouldn't be so hard to
substitute another key-value store for this one. After all, the simplicity of
the API around a key-value store where get & set is basically all you need,
should make that really easy.

~~~
ims
> When I find myself in the latter kind of work environment and I need to
> quickly get sortable and/or indexable datastructures of any kind, then a
> key-value store is the way to go

Interesting, can you expand on this?

~~~
stakhanov
Cf my ansewr to zzleeper's comment below.

------
romwell
I've used Tokyo Tyrant / Tokyo Cabinet and Kyoto Tycoon / Kyoto Cabinet in
production in my previous job in 2018.

There was a need to cache a large number of computation results, hundreds of
gigabytes, over network, for a scientific computing process that could run for
days.

The cache would not fit in memory, so the KV store needed to be persistent.

I wrote a caching layer with switchable backend, with an API inspired by KT,
so that plugging Kyoto Tycoon in would be easy.

Tested with TT, KT and Aerospike. In the end, went with either TT or KT, both
being stable, easy to use, performant, and compiling on something as old as
RH5 (which some of our clients were still running, apparently).

Compared to KT, Aerospike had more features (ssd optimization, sharding) when
used as a KV store, but we didn't need them.

In the world of persistent KV-stores there have not been that many options as
of last year. Kyoto Tycoon is feature-complete and works well.

That's why you don't hear many people talking about it: it just works, there's
not much to talk about.

And the source/API/documentation it's a work of art.

~~~
continuations
How did KT's performance compare to that of Aerospike's?

~~~
romwell
Sadly, I don't recall off the bat, and I don't have access to those files
anymore.

IIRC Aerospike would be faster initially because it keep a lot in RAM, but
they'd be about the same once the DB fills up you can't get away with that. In
the end, either network or disk would be the bottleneck for the workload I
tried it with.

------
iten
We use KT's in-memory database in a scientific computing application I work on
(to store a graph 30-100GB in size accessed by 100-1000 workers). The
performance is very impressive, and it's been reliable for our use case. But I
don't think I'd recommend anyone use it in 2019 -- having an active community
(and someone to actively maintain the software!) is too important. A small
performance gain over alternatives like Redis is probably not worth the
tradeoff of using software that is (sadly) abandoned.

edit: That's not to say it is really in need of much development -- it's
pretty feature-complete. But it's undergone a bit of software rot: the Debian
package, for example, ships header files which fail to compile under many
recent gcc versions. And the network effect is just not there. If you run into
some database slowness, searching the Web for "Redis performance problem"
might get you some ideas. Searching for "kyototycoon performance problem" will
get you nowhere.

~~~
kevin_thibedeau
Seems like it should be put up on GitHub where people can casually fix issues
as needed.

~~~
romwell
The source is available, why does it need to be on GitHub specifically?

But hey, why not, done deal:

[https://github.com/cloudflare/kyotocabinet](https://github.com/cloudflare/kyotocabinet)

------
booleandilemma
Fun fact: “tycoon” is a loanword from Japanese.

[https://en.wiktionary.org/wiki/tycoon](https://en.wiktionary.org/wiki/tycoon)

~~~
ohbarye
That's new to me even though I'm Japanese. lol

~~~
escherplex
Jim Breen's EDICT dictionary lists 'great' and 'old boy' as core meanings for
associated kanji. Aside from an in-group honorific suffix the _kimi_ kanji is
also associated with _moro no kimi_ or a Shinto _wolf spirit_ which may be a
good definition of a tycoon. :)

~~~
ramchip
Another reading of 大君 is ookimi, which is how the Emperor was called in
ancient times. In this context the kanji means lord, prince.

~~~
unscaled
Yes, ookimi is the common reading for 大君 today. Taikun seems to have come from
Edo period. According to several Japanese dictionaries, this was the title
applied to the Shogun, especially in foreign correspondence, which I assume
was primarily with China and Korea from most of that period. This title was
probably more appropriate for use with Chinese or Korean speakers, who would
read Shogun (將軍) as General, a purely military title. In any case, it seems
that after Japan opened up to western countries following Commodore Perry's
gunboat diplomacy, the term was imported to the West referring to the Shogun.

------
jgrahamc
Cloudflare used to use KT. That's very true, but we removed it and replaced it
with our own distributed KV store and replication mechanism called
Quicksilver.

~~~
numbsafari
What were the engineering motivations for moving from KT to Quicksilver? Did
that have to do with performance/throughput, or more with tooling and
functionality?

~~~
majke
There was a number of problems, but mostly boiling down to the sheer scale of
our deployment. As an engineer I would say that KT gave us surprisingly good
mileage. We were able to grow the company from couple of servers to hundreds,
but at some point we just outgrew KT.

The main issues that triggered serious internal discussions about KT:

(1) Tuning the data structures was hard. Each data backend has many toggles,
and it's not obvious what they actually do. I ended up writing script to try
out all the different toggles just to figure out what the disk and performance
cost would be.

(2) If we decided to tweak the data backend toggles, it was not obvious how to
upgrade the storage. There isn't a single command to take the KT store and
rewrite it with different specific-optimization-setting. This meant our disks
stored stuff in non-optimal format, wasting our disk and memory.

(3) We added encryption to the replication protocol, it was fine for some
time, but not perfect. If I remember right the SSL handshake was blocking,
which meant the master stalled for some time on each replica connect. This
code was ours, sure, but the problem is: integrating non-blocking SSL accept()
into KT event loop was very much non trivial.

(4) The replication protocol is great (and trivial!), if you the lifetime of
the replicas is roughly in line with the lifetime of the master. This was not
the case in our deployment. We might bring a new colo up, or just format disks
in some region. In such case, how do you bootstrap the replica? This is an
unsolved problem. Sadly, you can't just rsync the data directory while keeping
KT master running: it will end up inconsistent. We had to: stop the master,
rsync the disk, start the replica, start the master, make sure the replica
picks up the log in right place. This is fine if you have two replicas, but
not hundreds.

(5) The HA story of the "KT master" was unsatisfactory. The docs say that you
should configure two masters, each replicating to each other. But I never
understood how it was supposed to work in case of conflicts, clock skew, or
disk failures. Think about problems in second or third layer of replicas.

Most importantly (4) and (5) lead to data inconsistencies, which is
unacceptable. Furthermore as we grew we were constantly toggling our KT
architecture, adding more and more replica layers. This made (4) only worse.

I personally love KT, I think it's great piece of software. Sadly, it looks
abandoned, lacking community. I guess at some point for us the pains of
operating it outgrew the benefits it gave.

Our replacement, Quicksilver, is brilliant. Perhaps we should blog about it
more.

~~~
jamesog
And to add to point (4): as Marek says KT's replication is great... to a
point. When Cloudflare was small and we only had to replicate to a relatively
small number of machines it coped fine.

As we scaled out each data centre we quickly hit limitations in KT's ability
to replicate to many clients. We ended up having to install many more
replication points in each data centre or region just to allow KT to replicate
to all downstream clients, otherwise it would regularly stall while trying to
service all replication requests, massively slowing down replication, or
sometimes even missing updates.

There's also two more points to be made:

(6) As our KC (Kyoto Cabinet) databases grew, KT had trouble keeping
consistency of the file on disk. Sometimes a restart of the service would
leave you with a corrupted KC database, causing us to have to resync the node
from scratch. This was a huge operational burden - especially if corruption
somehow replicated downstream.

(7) KT allows you to write to replicas. This one we always found crazy. You
can write (or overwrite) a key on a downstream replica and that only ever
stays on the replica - it can't propagate upwards (and nor should it). Until
the top-level writes to that key again it will remain with this erroneous
value. This made monitoring problematic because it would cause a keycount or
replication diff if it ever happened.

------
peatfreak
At a previous organizion I worked at, about 5-7 years ago, I made very heavy
use of Tokyo Tyrant and T. Cabinet (the predecessors of Kyoto Tycoon and KC).
We had many instances throughout our infrastructure, and many engineers had at
least some knowledge of it.

It was the fastest key-value store I'd ever used. I was floored by its speed
and relative stability.

And the article and comments rightly point out KT's terrible documentation and
thin user community on the ground. TT was just the same. This caused a lot of
surprising (and mostly undocumented) issues whenever somebody was trying to
get onboarded.

I was surprised that it was so obscure and so undocumented. Especially
considering its speed and simplicity (in terms of getting a basic k-v store
set up).

I depended a lot on DBA's and DB engineers, our sysadmins, and other engineers
with more experience of TT than I had. Usually, whenever I needed to
understand a configuration option in detail, deploy a new instance with
different characteristics, or optimize an existing instance, I would consult
them. I think we had just as much, if not, more, knowledge of this software
than any community on the Internet.

But all of this raw speed, at the cost of lack of support, lack of
documentation or community bit us in the ass, hard. Either the TT/TC service
itself crashed, or the host it was running on crashed (I can't remember).

Some data was lost because it was designed with no fault tolerance. Most data
was recovered from backups. An executive decision was immediately made to move
to a different DB, with proper resilience, etc. We quickly moved to a well
known, open source, commercially supported system with all the usual value-
added proprietary parts.

Then the $$$$$ started pouring in to the support contract. At least we had
something trustworthy that we could understand, lots of documentation, and
good upstream support engineers :-)

It's a bit sad to imagine what could have become of TT/TC. I've no idea about
KT/KC. They had a LOT of potential. Maybe the potential is still there? Maybe
they're project deserve resurrection, at least for educational purposes?

At least the source code still exists if anybody is interested enough.
Documentation generally consists of reading struct definitions and comments; a
couple of decent StackOverflow posts; and now-mostly defunct mailing lists on
Google Groups.

------
k-ian
I was really hoping that this was a videogame

------
anoother
Does anyone have comparisons (even anecdata) with (L)MDB?

~~~
jamesog
Cloudflare's replacement for KT that jgrahamc mentioned - Quicksilver - uses
LMDB for storage.

I don't have numbers, but it performs great for our needs. It's heavier on the
disk though because of fsync, however unlike KT we never get database
corruption on disk.

------
acd
When investing key value stores for trying to beat a nosql speed record Kyoto
Tycoon and LevelDB came up as low latency alternatives.

[http://blog.creapptives.com/post/8330476086/leveldb-vs-
kyoto...](http://blog.creapptives.com/post/8330476086/leveldb-vs-kyoto-
cabinet-my-findings)

Seems like KyotoDB is beating LevelDB on performance?

~~~
hyc_symas
Google's original LevelDB microbenchmark report shows LevelDB soundly beating
Kyoto.

[http://www.lmdb.tech/bench/microbench/benchmark.html](http://www.lmdb.tech/bench/microbench/benchmark.html)

But using the exact same code and microbenchmark, LMDB obliterates them all.

[http://www.lmdb.tech/bench/microbench/](http://www.lmdb.tech/bench/microbench/)

All with full ACID transactions and crash-proof reliability.

------
jnwatson
I used KT back in the day, but found LMDB to be much more reliable and
performant.

------
totoglazer
I was a huge Tokyo Cabinet fan back in the day. Was a big step up from
BerkeleyDB for some workloads we were doing a lot. Always disappointed it was
so fractured with successors and less adoption than I felt it deserved.

------
aasasd
The benchmark is likely limited by Python. You can't really test Redis' speed
with Python, not in one process.

~~~
coleifer
The purpose of the benchmark was to see the _relative_ performance of the two
databases.

~~~
aasasd
You won't see the performance of the database if the time is spent in the
Python driver.

