Vitess: Scaling MySQL databases for the web

[1010011010]

http://code.google.com/p/vitess/

[enneff]

I'm glad to see this released because it is great technology that showcases Go's strengths. Here's why they chose Go, in their own words:

"Go is miles ahead of C++ and Java in terms of expressibility and close in terms of performance. It is also relatively simple and has a straightforward interaction with Linux system calls."

It has always frustrating to say "We use Go at Google, I just can't say how." So I'm happy that now I can say that YouTube use Go in their core infrastructure, serving more than a billion requests per day with this service alone.

[tcard]

A quick glance at the sources shows immediately what Go offers per se in terms of readability and maintainability. Go strongly encourages, even forces at compile time, certain style constraints that results in such a delightful code to explore. Were it written in C, you'd have to deal first with arcane #whatever directives in .h and .c files and strange makefiles and such stuff.

I hope someday Go will get the acknowledgement it deserves as a real piece of genius. I really encourage recommend just to take a 15-min try at http://tour.golang.org.

[canop_fr → tcard]

The thing would probably look better if adapted to Go1, without all those makefiles and deps.

[canop_fr]

Interesting comments (in mysql.go) :

    // Muahahaha
    func strlen(str *C.char) int {
            gstr := (*[1 << 30]byte)(unsafe.Pointer(str))
            length := 0
            for gstr[length] != 0 {
                    length++
            }
            return length
    }

BTW, is Google planning to make a proper mysql driver in go (implementing database/sql API and not unsafe) ?

[enneff → canop_fr]

Here's a list of database/sql drivers: http://code.google.com/p/go-wiki/wiki/SQLDrivers

[canop_fr → enneff]

Yes, that's the driver I'm using. But my question was related to the fact that Google seems to have made (see source) a partial mysql driver, so I was wondering if they were planning to make it database/sql compatible.

[stevencorona]

This project looks awesome and really points out some of the major weaknesses with in mySQL/libmysql-

  * Impossible/difficult to gracefully handle connection/query timeouts.
  * Extremely poor query/row cache (the table's entire query cache is flushed anytime a write is made in the table)
  * Poor connection memory usage- Each open connection sucks up ~10MB of memory

[harlowja → stevencorona]

Odd, so instead of helping fix the underlying libraries they created yet another layer. Sadness++

[foobarbazetc]

How does this help "scale" databases?

It just looks like a connection pooler with a RPC interface?

Am I missing a feature?

[enneff → foobarbazetc]

The feature list on the linked page lists a lot more than that. Also see:

http://code.google.com/p/vitess/wiki/ProjectGoals

[jakejake]

At a glance this seems fantastic. I have an application where the database recently started to get overloaded at peak usage times.

It seems like this could be dropped in as a replacement with very little code re-factoring if the client library is similar to the native mysql client lib. (or an adapter could be written)

Does anybody know what would be involved in getting this working with PHP?

[haarts]

It is rather unclear to me whether or not Vtocc automagically takes care of sharding and replication.

[dkhenry]

What I don't understand is why you would use MySQL as your storage engine it seems to me you are stripping away most of the benefits of using MySQL by interfacing it with this library. Is there a compelling reason to not use google's own LevelDB?

[mattyb → dkhenry]

Is there a compelling reason to not use google's own LevelDB?

LevelDB is a key-value storage library. MySQL is a relational database server. They are about as different as data stores can get.

[stevencorona → dkhenry]

Don't really see what parts of mySQL you are stripping away by using this library. It's just extending/improving some weak spots of mySQL- you literally don't lose any features.

[dkhenry → stevencorona]

According to their website - """ The way we plan to achieve this is by providing a subset SQL front end with limited guarantees """

If your making something to parse and validate SQL yourself which is what it seems like they are doing then why go through all the hassle to run it a second time through another SQL parser. Also since your mandating everything have a "Key" why are we not using a "Key-Value" store.

I am not saying this is a bad project or a bad design decision I am just curious why they made those decisions since at first glance it makes no sense whatsoever.