
Google releases snappy, the compression library used in Bigtable - tonfa
http://code.google.com/p/snappy/
======
haberman
I did a double-take when I saw this -- the library is called "zippy"
internally, but there must have been some kind of trademark issue with that.

This is used in more than just BigTable; it's used extensively inside Google,
in particular for the RPC system. It's great to see it open-sourced!

~~~
stevenbedrick
Maybe the issue was regarding the comic strip, "Zippy the pinhead"?

~~~
pmjordan
My first association was with the character from the UK children's TV show
Rainbow: <http://en.wikipedia.org/wiki/Zippy_(Rainbow)>

I doubt it's a trademark issue, as trademarks tend to be usage-specific. (e.g.
Apple Computer vs. Apple Music) It's probably just a matter of Google feeling
there are too many different things already associated with the name 'Zippy.'

(I also now have the "Rainbow" theme tune stuck in my head.)

------
ot
I wonder if they had evaluated LZO
(<http://www.oberhumer.com/opensource/lzo/>) before writing this. It is quite
well-tested (a variant on it runs in the Mars Rovers) and very very fast: the
author reports 16MB/sec on a Pentium 133, on modern architectures it should
easily get to the 500MB/sec claimed by snappy.

~~~
lemming
"The LZO algorithms and implementations are copyrighted OpenSource distributed
under the GNU General Public License."

Does this even make sense? Can you apply the GPL to an algorithm? As I
understand it, if there's no patent I should be able to implement it with no
problems.

~~~
wmf
People who want to upsell you a commercial version tend to say such things.

~~~
bhickey

        Be warned: the main source code in the 'src' directory 
        is a real pain to understand as I've experimented with 
        hundreds of slightly different versions. It contains 
        many #if and some gotos, and is *completely optimized  
        for speed* and not for readability. Code sharing of the 
        different algorithms is implemented by stressing the 
        preprocessor - this can be really confusing. Lots of 
        marcos and assertions don't make things better.
    

Given the author's statements, I don't know that I'd feel comfortable using
LZO in a production environment.

~~~
malkia
We used it in our games since Ultimate Spiderman. I've got assembly version
working on the PS2 IOP chip (33Mhz, the chip used to run PS1 games, otherwise
I/O and sound for PS2).

I've got 5mb/s decompressed data speed, so that was speeding our disk access.
I had to tweak just a little bit the source code and make sure unaligned
writes (4 bytes) were used and that made it x2 or x3 faster.

------
nonameman
You should probably mention why someone would choose this library over another
compression library. I think good advice would be to use Snappy to compress
data that is meant to be kept _in memory_ , as Bigtable does with the
underlying SSTables. If you are reading from disk, a slower algorithm with a
better compression ratio is probably a better choice because the cost of the
disk seek will dominate the cost of the compression algorithm.

------
wladimir
Another incredible internal project open-sourced by Google. I really respect
Google's dedication to improving the speed of the internet in general, and to
open source.

Of course this benefits them as well, but it's a form of enlightened self-
interest that, to me, is very refreshing compared to for example Microsoft,
and other companies that only care about their own software/platforms and only
release stuff on need-to-know basis.

------
sanxiyn
A note from a developer:
<http://blog.sesse.net/blog/tech/2011-03-22-19-24_snappy.html>

------
endgame
IMHO, the build system could do with a little work:

* The various bits generated from and added by the autotools shouldn't be committed. autoreconf -i works really well these days. That's INSTALL Makefile.in aclocal.m4 compile config.guess config.h.in config.sub configure depcomp install-sh ltmain.sh missing mkinstalldirs.

configure.ac:

* Needs to call AC_SUBST([LIBTOOL_DEPS]) or else the rule to rebuild libtool in Makefile.am won't work.

* A lot of macro calls are underquoted. It'll probably work fine, but it's poor style.

* The dance with EXTRA_LIBSNAPPY_LDFLAGS seems odd. It'd be more conventional to do something like:
    
    
        SNAPPY_LTVERSION=snappy_version
        AC_SUBST([SNAPPY_LTVERSION])
    

and set the -version-info flag directly in Makefile.am. If it's to allow the
user to provide custom LDFLAGS, it's unnecessary: LDFLAGS is part of
libsnappy_la_LINK. Here's the snippet from Makefile.in:

    
    
        libsnappy_la_LINK = $(LIBTOOL) --tag=CXX $(AM_LIBTOOLFLAGS) \
                $(LIBTOOLFLAGS) --mode=link $(CXXLD) $(AM_CXXFLAGS) \
                $(CXXFLAGS) $(libsnappy_la_LDFLAGS) $(LDFLAGS) -o $@
    

* There should be an AC_ARG_WITH for gflags, because automagic dependencies aren't cool: <http://www.gentoo.org/proj/en/qa/automagic.xml>

* Shell variables starting with ac_ are in autoconf's namespace. Setting things like ac_have_builtin_ctz is therefore equally uncool. See <http://www.gnu.org/s/hello/manual/autoconf/Macro-Names.html> :

> To ensure that your macros don't conflict with present or future Autoconf
> macros, you should prefix your own macro names and any shell variables they
> use with some other sequence. Possibilities include your initials, or an
> abbreviation for the name of your organization or software package.

* Use AS_IF instead of directly using the shell's `if`: [http://www.gnu.org/software/hello/manual/autoconf/Limitation...](http://www.gnu.org/software/hello/manual/autoconf/Limitations-of-Builtins.html#Limitations-of-Builtins) and [http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Cons...](http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Constructs.html) .

* Consider adding -Wall to either AUTOMAKE_OPTIONS in Makefile.am or as an argument to AM_INIT_AUTOMAKE. If you don't mind using a modern automake (1.11 or later), also call AM_SILENT_RULES([yes]). Even MSYS has automake-1.11 these days.

Makefile.am:

* Adding $(GTEST_CPPFLAGS) to both snappy_unittest_CPPFLAGS and snappy_unittest_CXXFLAGS is redundant. See this part of Makefile.in:
    
    
        snappy_unittest-snappy-test.o: snappy-test.cc
        @am__fastdepCXX_TRUE@   $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -MT snappy_unittest-snappy-test.o -MD -MP -MF $(DEPDIR)/snappy_unittest-snappy-test.Tpo -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc
        @am__fastdepCXX_TRUE@   $(am__mv) $(DEPDIR)/snappy_unittest-snappy-test.Tpo $(DEPDIR)/snappy_unittest-snappy-test.Po
        @AMDEP_TRUE@@am__fastdepCXX_FALSE@      source='snappy-test.cc' object='snappy_unittest-snappy-test.o' libtool=no @AMDEPBACKSLASH@
        @AMDEP_TRUE@@am__fastdepCXX_FALSE@      DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
        @am__fastdepCXX_FALSE@  $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc
    

* snappy_unittest should be in check_PROGRAMS, not noinst_PROGRAMS. That way, it's built as part of `make check`, not `make all`.

~~~
seiji
Soon, somebody will convert the build system to CMake, move it to github,
clean up the insulting directory structure, then nobody will look at their
google code page ever again.

~~~
endgame
People act like CMake is an improvement, but it's not. The language is not
very good (lists as semicolon-separated strings, seriously?). For example: its
pkg-config support is completely broken. It takes the output of pkg-config,
parses it into a list (liberally sprinkling semicolons where the spaces should
be) and then the semicolons make it into the compiler command line, causing
all manner of cryptic errors.

Stick with automake. Seriously.

~~~
ootachi
On Windows, automake is an order of magnitude slower to compile than the
projects generated by CMake, not to mention that compiling with MSVC is very
difficult to make work at all with autotools. automake just isn't a viable
option if your projects need to be portable to Windows.

~~~
dspillett
Speed differences like this are often down to process creation.

A lot of automation routines designed on unix-a-like systems involve creating
short lived processes with reckless abandon because creating and tearing down
a process in most Unix environments is relatively efficient. When you
transplant these procedures to Windows you are hit by the fact that creating
or forking a process there is relatively expensive. IIRC the minimum per-
process memory footprint is higher under Windows too, though this doesn't
affect build scripts where generally each process (or group of processes if
chaining things together with pipes and redirection) is created and finished
with in turn rather than many running concurrently.

This is why a lot of Unix services out there are process based rather than
thread based but a Windows programmer would almost never consider a process
based arrangement over a threaded one. Under most unix-a-like OSs the
difference between thread creation and process creation is pretty small so
(unless you need very efficient communication between things once they are
created or might be running in a very small amount of RAM) a process based
modal can be easier to create/debug/maintain which is worth the small CPU
efficiency difference. Under Windows the balance is different: creating
processes and tearing them down after use is _much_ more work for the OS than
operating with threads is, so a threaded approach is generally preferred.

------
tptacek
When you can measure efficiency improvements like this in millions of dollars,
I'm sure this makes a whole hell of a lot of sense. But for anyone below, say,
Twitter's scale: is this ever an engineering win over zlib?

~~~
Andys
Perhaps a more pertinent question: Is it any better than LZJB or LZO?

~~~
wmf
It's got a better license than LZJB (and maybe LZO depending on your views).

~~~
ot
Doesn't Google use GPL2 all over the place?

~~~
wmf
Just to use one example, I would assume that the crawler/indexer/ranker
"secret sauce" in the appliance can't link against any GPL libraries.

~~~
tptacek
If you don't distribute your GPL-tainted code, I don't believe you need to do
anything to comply with the GPL. This is why the Affero GPL exists.

~~~
tonfa
The keyword in the parent comment was "appliance" :)

~~~
tptacek
Doh! Missed that. Yep.

------
tezza
Curious, it's written in C++ .

IMHO I think straight C would have been easier for World + Dog to link
against.

~~~
bajsejohannes
If exports are within `extern "C"`, it shouldn't be any more difficult.

------
chubs
Had a look at the code, it's quite neat and tidy, i'm really impressed and
surprised considering the need for speed/optimisation in libraries like this
tends to make the code unreadable...

------
MichaelGG
For pure speed, check out QuickLZ[1]. While it probably doesn't compress well
as Snappy, it does hit 300MB+/core. But, it's GPL instead of Apache.

1: <http://www.quicklz.com/>

~~~
_delirium
They claim in the README that Snappy is usually faster than QuickLZ. Haven't
tested myself, though.

~~~
Lazze
They compare with QuickLZ 1.0 from 2006 (you can see that from
snappy_unittest.cc).

I've done a quick benchmark with the files on quicklz.com on the same machine
with QuickLZ 1.5.0:

test file; library; compressed size; compression speed in mb/s

average QuickLZ 47.9% 308 snappy 53.0% 261

proteins.txt QuickLZ 1.5.0 35.6% 331 snappy 40.5% 232

plaintext.txt QuickLZ 1.5.0 48.1% 245 snappy 55.5% 193

gdb.exe QuickLZ 1.5.0 45.8% 270 snappy 51.1% 214

flower.bmp QuickLZ 1.5.0 86.7% snappy 91.5% 208

northwind.mdf QuickLZ 1.5.0 23.2% snappy 26.4% 456

~~~
Sesse_
Hi,

I work at Google, and I've been working on the Snappy open-sourcing. First of
all, let me state that when doing our benchmarks, we were definitely comparing
against the latest public version (1.5.0) of QuickLZ. I'm having problems
reproducing your benchmark results; could you provide some more details about
the exact settings, compiler version, architecture etc.? I can't even get all
of the compression ratios to match up exactly with the ones I'm seeing, so
there must be some sort of difference between the setups. Are you perchance
running Snappy with assertions enabled? (The microbenchmark will complain if
you do, so it's easy to check.)

For what it's worth, I ran the QuickLZ test suite on my own machine (you can
see the exact testing environment below), and got quite different results:

    
    
      qlz-testsuite/flower.bmp                 :
      QUICKLZ: [b 1M] bytes 5922816 -> 5087471 85.9%  comp 132.4 MB/s  uncomp 140.8 MB/s
      SNAPPY: [b 4M] bytes 5922816 -> 5369439 90.7%  comp 180.6 MB/s  uncomp 484.0 MB/s
      qlz-testsuite/gdb.exe                    :
      QUICKLZ: [b 1M] bytes 8872609 -> 4074990 45.9%  comp 177.4 MB/s  uncomp 213.3 MB/s
      SNAPPY: [b 4M] bytes 8872609 -> 4530851 51.1%  comp 224.7 MB/s  uncomp 468.7 MB/s
      qlz-testsuite/northwind.mdf              :
      QUICKLZ: [b 1M] bytes 2752512 -> 628607 22.8%  comp 291.3 MB/s  uncomp 409.6 MB/s
      SNAPPY: [b 4M] bytes 2752512 -> 726437 26.4%  comp 456.6 MB/s  uncomp 790.1 MB/s
      qlz-testsuite/plaintext.txt              :
      QUICKLZ: [b 1M] bytes 2899483 -> 1436466 49.5%  comp 155.4 MB/s  uncomp 182.7 MB/s
      SNAPPY: [b 4M] bytes 2899483 -> 1629408 56.2%  comp 195.2 MB/s  uncomp 487.5 MB/s
      qlz-testsuite/proteins.txt               :
      QUICKLZ: [b 1M] bytes 7254685 -> 2600129 35.8%  comp 218.2 MB/s  uncomp 261.2 MB/s
      SNAPPY: [b 4M] bytes 7254685 -> 2934406 40.4%  comp 269.4 MB/s  uncomp 556.9 MB/s
    
      Average compression ratio (geometric mean, lower is better): QuickLZ 43.7%, Snappy 48.9%.
      Average compression speed (harmonic mean): QuickLZ 180.9 MB/s, Snappy 238.0 MB/s.
      Average decompresion speed (harmonic mean): QuickLZ 212.5 MB/s, Snappy 536.9 MB/s.
    

In addition, there's one nearly-incompressible file listed on the same page,
which I also ran on (it's not included in the averages above):

    
    
      qlz-testsuite/NotTheMusic.mp4            :
      QUICKLZ: [b 1M] bytes 9832475 -> 9832565 100.0%  comp 289.3 MB/s  uncomp 2815.9 MB/s
      SNAPPY: [b 4M] bytes 9832475 -> 9485433 96.5%  comp 1308.4 MB/s  uncomp 2477.1 MB/s
    

This is on a Nehalem 2.27GHz, running Debian GNU/Linux 6.0 in 64-bit mode, GCC
version 4.4.5, with flags -O2 -g -DNDEBUG for both Snappy (1.0.0) and QuickLZ
(1.5.0), running snappy_unittest to compare. QuickLZ was left at the default
settings, ie. the files were exactly as downloaded from
<http://www.quicklz.com/quicklz.[ch]> as of today. In particular, this means
QuickLZ was running in unsafe mode, which means it could crash on corrupted
compressed input. (Safe mode, which is comparable to how Snappy runs, is,
according to <http://www.quicklz.com/download.html>, 15–20% slower at
decompression, so the difference would be bigger.)

Exact benchmark results will, as always, vary with differing compiler, set of
flags, CPU, and data set. The average numbers from this benchmark, however,
show QuickLZ 1.5.0 (in unsafe mode) compressing about 11% more densely than
Snappy 1.0.0 does, but Snappy compressing about 32% faster and decompressing
about 153% faster (ie., about 2.53 times as fast).

/* Steinar */

~~~
Lazze
Sorry, had assertions on. Getting same results now (using my own benchmark
function though)

Benchmarking: benchdata/northwind.mdf

snappy: Compressed 2752512 bytes into 726437 (26.4%) at 653.0 MiB/s

QuickLZ: Compressed 2752512 bytes into 620933 (22.6%) at 456.8 MiB/s

Benchmarking: benchdata/gdb.exe

snappy: Compressed 8872609 bytes into 4530844 (51.1%) at 333.1 MiB/s

QuickLZ: Compressed 8872609 bytes into 4056279 (45.7%) at 267.6 MiB/s

Benchmarking: benchdata/pic.bmp

snappy: Compressed 18108198 bytes into 16561772 (91.5%) at 292.5 MiB/s

QuickLZ: Compressed 18108198 bytes into 15784615 (87.2%) at 197.2 MiB/s

Benchmarking: benchdata/plaintext.txt

snappy: Compressed 2988604 bytes into 1657747 (55.5%) at 289.4 MiB/s

QuickLZ: Compressed 2988604 bytes into 1436646 (48.1%) at 238.5 MiB/s

Benchmarking: benchdata/proteins.txt

snappy: Compressed 7344249 bytes into 2964303 (40.4%) at 371.6 MiB/s

QuickLZ: Compressed 7344249 bytes into 2586659 (35.2%) at 321.9 MiB/s

------
delineal
I may convert my monthly archive of websites over to snappy; the speed of the
compression / decompression will allow me to implement a more consolidated
storage scheme than I'm using now.

------
swah
Sometimes its odd to think a thousand lines of C++ is something folks were
waiting to be released for years.

------
philf
I'm puzzled by snappy-stubs-internal.h l105-118 Why would one log by
instantiating a class, not using the result, therefore leading to the
destructor being called which writes the log message? Can anyone come up with
a reason for this?

~~~
daniel02216
It uses the same construct as VLOG above does, which allows you to << an error
string into CRASH_UNLESS and VLOG. The weirdness comes in when you want to
conditionalize the log stream, which isn't implemented in the stubs here. The
real Google logging classes are probably more sophisticated, and this is just
a shim to get the snappy code to work unchanged.

Specifically the answer to your question is that printing in the destructor is
there to print a newline right before calling abort but after having printed
whatever was <<'ed to CRASH_UNLESS.

Check out libjingle's logging implementation which has some similar weirdness
in it:
[http://code.google.com/p/libjingle/source/browse/trunk/talk/...](http://code.google.com/p/libjingle/source/browse/trunk/talk/base/logging.h)

------
patrickaljord
Anyone feels like writing a tiny C client? Looks like this comes only as a
lib.

~~~
tonfa
I think it is already a command line application (if you use the google
cmdline flag lib, see the README).

~~~
endgame
I don't believe so. Look at the Makefile.am. It only declares a libtool
library (lib_LTLIBRARIES = libsnappy.la, which causes libsnappy.so.x.y.z and
libsnappy.a to be built), and a unittest (TESTS =
snappy_unittest,noinst_PROGRAMS = $(TESTS)).

EDIT: I'm wrong:

> Actually, snappy_unittest is more than just a test, despite the name; in
> particular, if you have the gflags package installed, it can be used for
> benchmarking. Thus, it's useful to have it built as part of make all. I'm
> closing this one (but thanks for all the others!).

[http://code.google.com/p/snappy/issues/detail?id=11&can=...](http://code.google.com/p/snappy/issues/detail?id=11&can=1)

