This is used in more than just BigTable; it's used extensively inside Google, in particular for the RPC system. It's great to see it open-sourced!
I doubt it's a trademark issue, as trademarks tend to be usage-specific. (e.g. Apple Computer vs. Apple Music) It's probably just a matter of Google feeling there are too many different things already associated with the name 'Zippy.'
(I also now have the "Rainbow" theme tune stuck in my head.)
Does this even make sense? Can you apply the GPL to an algorithm? As I understand it, if there's no patent I should be able to implement it with no problems.
Be warned: the main source code in the 'src' directory
is a real pain to understand as I've experimented with
hundreds of slightly different versions. It contains
many #if and some gotos, and is *completely optimized
for speed* and not for readability. Code sharing of the
different algorithms is implemented by stressing the
preprocessor - this can be really confusing. Lots of
marcos and assertions don't make things better.
I've got 5mb/s decompressed data speed, so that was speeding our disk access. I had to tweak just a little bit the source code and make sure unaligned writes (4 bytes) were used and that made it x2 or x3 faster.
"In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios."
> Finally, snappy can benchmark Snappy against a few other compression libraries
(zlib, LZO, LZF, FastLZ and QuickLZ), if they were detected at configure time.
I'll do some tests, I'm curious about the results
~/snappy-read-only $ ./snappy_unittest -norun_microbenchmarks -lzo testdata/*
LZO: [b 1M] bytes 152089 -> 82691 54.4% comp 64.5 MB/s uncomp 206.4 MB/s
SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 171.5 MB/s uncomp 375.8 MB/s
LZO: [b 1M] bytes 125179 -> 73217 58.5% comp 52.5 MB/s uncomp 173.2 MB/s
SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 137.8 MB/s uncomp 301.7 MB/s
LZO: [b 1M] bytes 27512 -> 26487 96.3% comp 24.4 MB/s uncomp 491.6 MB/s
SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 305.2 MB/s uncomp 1465.7 MB/s
LZO: [b 1M] bytes 27483 -> 26528 96.5% comp 24.3 MB/s uncomp 499.1 MB/s
SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 331.8 MB/s uncomp 1660.4 MB/s
LZO: [b 1M] bytes 28384 -> 27380 96.5% comp 24.1 MB/s uncomp 488.2 MB/s
SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 275.7 MB/s uncomp 1346.5 MB/s
LZO: [b 1M] bytes 24603 -> 11621 47.2% comp 57.2 MB/s uncomp 258.5 MB/s
SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 190.0 MB/s uncomp 443.5 MB/s
LZO: [b 1M] bytes 11150 -> 4663 41.8% comp 73.8 MB/s uncomp 259.8 MB/s
SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 207.3 MB/s uncomp 431.7 MB/s
LZO: [b 1M] bytes 100000 -> 20423 20.4% comp 115.8 MB/s uncomp 429.9 MB/s
SNAPPY: [b 4M] bytes 100000 -> 23488 23.5% comp 359.0 MB/s uncomp 625.8 MB/s
LZO: [b 1M] bytes 3721 -> 1781 47.9% comp 66.8 MB/s uncomp 311.4 MB/s
SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 214.9 MB/s uncomp 461.2 MB/s
LZO: [b 1M] bytes 126958 -> 127173 100.2% comp 20.2 MB/s uncomp 1420.8 MB/s
SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2037.6 MB/s uncomp 7578.7 MB/s
LZO: [b 1M] bytes 102400 -> 21027 20.5% comp 115.1 MB/s uncomp 423.8 MB/s
SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 362.4 MB/s uncomp 703.5 MB/s
LZO: [b 1M] bytes 409600 -> 82980 20.3% comp 120.9 MB/s uncomp 416.4 MB/s
SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 359.0 MB/s uncomp 698.0 MB/s
LZO: [b 1M] bytes 1029744 -> 357315 34.7% comp 133.5 MB/s uncomp 533.6 MB/s
SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 294.1 MB/s uncomp 432.6 MB/s
LZO: [b 1M] bytes 184320 -> 71671 38.9% comp 84.3 MB/s uncomp 235.9 MB/s
SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 231.7 MB/s uncomp 373.8 MB/s
LZO: [b 1M] bytes 426754 -> 221290 51.9% comp 57.6 MB/s uncomp 181.9 MB/s
SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 153.7 MB/s uncomp 341.4 MB/s
LZO: [b 1M] bytes 94330 -> 76999 81.6% comp 24.9 MB/s uncomp 810.9 MB/s
SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 709.0 MB/s uncomp 1669.5 MB/s
LZO: [b 1M] bytes 481861 -> 294610 61.1% comp 51.4 MB/s uncomp 164.8 MB/s
SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 129.8 MB/s uncomp 281.9 MB/s
LZO: [b 1M] bytes 513216 -> 86232 16.8% comp 119.0 MB/s uncomp 506.1 MB/s
SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 479.8 MB/s uncomp 670.8 MB/s
LZO: [b 1M] bytes 38240 -> 17686 46.2% comp 57.8 MB/s uncomp 267.0 MB/s
SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 194.5 MB/s uncomp 401.0 MB/s
LZO: [b 1M] bytes 702087 -> 309320 44.1% comp 55.7 MB/s uncomp 265.1 MB/s
SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 216.9 MB/s uncomp 499.3 MB/s
LZO: [b 1M] bytes 4227 -> 2450 58.0% comp 56.3 MB/s uncomp 286.2 MB/s
SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 173.8 MB/s uncomp 397.3 MB/s
Of course this benefits them as well, but it's a form of enlightened self-interest that, to me, is very refreshing compared to for example Microsoft, and other companies that only care about their own software/platforms and only release stuff on need-to-know basis.
* The various bits generated from and added by the autotools shouldn't be committed. autoreconf -i works really well these days. That's INSTALL Makefile.in aclocal.m4 compile config.guess config.h.in config.sub configure depcomp install-sh ltmain.sh missing mkinstalldirs.
* Needs to call AC_SUBST([LIBTOOL_DEPS]) or else the rule to rebuild libtool in Makefile.am won't work.
* A lot of macro calls are underquoted. It'll probably work fine, but it's poor style.
* The dance with EXTRA_LIBSNAPPY_LDFLAGS seems odd. It'd be more conventional to do something like:
libsnappy_la_LINK = $(LIBTOOL) --tag=CXX $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXXLD) $(AM_CXXFLAGS) \
$(CXXFLAGS) $(libsnappy_la_LDFLAGS) $(LDFLAGS) -o $@
* Shell variables starting with ac_ are in autoconf's namespace. Setting things like ac_have_builtin_ctz is therefore equally uncool. See http://www.gnu.org/s/hello/manual/autoconf/Macro-Names.html :
> To ensure that your macros don't conflict with present or future Autoconf macros, you should prefix your own macro names and any shell variables they use with some other sequence. Possibilities include your initials, or an abbreviation for the name of your organization or software package.
* Use AS_IF instead of directly using the shell's `if`: http://www.gnu.org/software/hello/manual/autoconf/Limitation... and http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Cons... .
* Consider adding -Wall to either AUTOMAKE_OPTIONS in Makefile.am or as an argument to AM_INIT_AUTOMAKE. If you don't mind using a modern automake (1.11 or later), also call AM_SILENT_RULES([yes]). Even MSYS has automake-1.11 these days.
* Adding $(GTEST_CPPFLAGS) to both snappy_unittest_CPPFLAGS and snappy_unittest_CXXFLAGS is redundant. See this part of Makefile.in:
@am__fastdepCXX_TRUE@ $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -MT snappy_unittest-snappy-test.o -MD -MP -MF $(DEPDIR)/snappy_unittest-snappy-test.Tpo -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc
@am__fastdepCXX_TRUE@ $(am__mv) $(DEPDIR)/snappy_unittest-snappy-test.Tpo $(DEPDIR)/snappy_unittest-snappy-test.Po
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ source='snappy-test.cc' object='snappy_unittest-snappy-test.o' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc
I've also found Diego Pettenò's "Autotools Mythbuster" to be quite good: http://www.flameeyes.eu/autotools-mythbuster/index.html . His old article "Best practices with autotools" isn't bad, but a little light: http://www.linux.com/archive/articles/114061
You may also want to browse the autoconf and automake tags of Diego's blog: http://blog.flameeyes.eu/tag/autoconf and http://blog.flameeyes.eu/tag/automake
A thorough reading of the autoconf and automake manuals also points out common pitfalls.
I started by spending a few hours forcing myself to read the following. It explains how autotools works and (importantly) why.
It is critical to understand what the autotools are actually doing before you just go and copy'n'paste somebody else's configure.ac and/or Makefile.am.
When you are ready to get your feet wet, here is a really good quick summary from the GNOME project:
Once you understand what is going on, finding specifics in the GNU manuals (and being able to interpret them) is much easier. Yes, you should understand how m4 works (it's not rocket science).
Stick with automake. Seriously.
I think it might actually be worse than autoconf in every way, which is surprising considering how bad autoconf is. The handwritten non-macro-expanding not-much-autogenerating configure/makefile in ffmpeg/libav/x264/vp8 is easier to deal with than either.
To me, it's not perfect, but it's pretty good. Then again, I'm known to my friends as "that guy who knows automake" :-).
A lot of automation routines designed on unix-a-like systems involve creating short lived processes with reckless abandon because creating and tearing down a process in most Unix environments is relatively efficient. When you transplant these procedures to Windows you are hit by the fact that creating or forking a process there is relatively expensive. IIRC the minimum per-process memory footprint is higher under Windows too, though this doesn't affect build scripts where generally each process (or group of processes if chaining things together with pipes and redirection) is created and finished with in turn rather than many running concurrently.
This is why a lot of Unix services out there are process based rather than thread based but a Windows programmer would almost never consider a process based arrangement over a threaded one. Under most unix-a-like OSs the difference between thread creation and process creation is pretty small so (unless you need very efficient communication between things once they are created or might be running in a very small amount of RAM) a process based modal can be easier to create/debug/maintain which is worth the small CPU efficiency difference. Under Windows the balance is different: creating processes and tearing them down after use is much more work for the OS than operating with threads is, so a threaded approach is generally preferred.
For a Win/OS X/Linux portable build, a Makefile should suffice. Example: https://github.com/MatzeB/cparser/blob/master/Makefile
Sure, if your build is simple, make works fine. But, the projects I've worked on where autotools was used, simply using make would have been a horrible experience. And, in most cases, the projects started out using make by itself and then moved to using autotools when the number of platform specific makefiles became too big to maintain.
This is a matter of opinion. When you have dependencies which rely on specialist tools, it is a good idea (and accepted practice, though this obviously could be argued), to commit your generated files too. This means that the files don't change depending on the version of autoconf/bison/etc that's installed on a user's machine.
As I recall, years ago some folks wrote up all these "version control best practices" for some conference, and this was one of the "rules". But it's common sense too - autotools is deep enough magic that most people won't know to run bootstrap.sh, or autoreconf -i, or whatever.
(Ironically, google code doesn't agree with this, and won't let you trim generated code like this from the diffs they send. See http://code.google.com/p/support/issues/detail?id=197).)
I disagree. At this point in the game, autotooled systems are pretty common, and to assert that they are deep magic like saying "CMake is deep enough magic that most people won't know to run mkdir build && cd build && cmake .., or whatever".
Developers are not like most people. A release tarball (created with `make distcheck`, say) will absolutely contain configure and other generated files. That's one of the points of the autotools - to have minimal dependencies at build time.
Developers, on the other hand, will need to look up a project's dependencies, install other special tools (parser generators, for instance). This should be documented somewhere, along with the bootstrap instructions.
Maybe you can require your developers to install esoteric tools, but that's no way to solicit contributions. For a compiler I worked on, we had a dependency on libphp5, which was enough to discourage anybody. But we also had dependencies on a particular version of flex, gengetopt, bison, autoconf, automake and _maketea_. The final one was a tool we wrote and had spun out into a separate project, which was written in Haskell, and required exactly ghc 6.10. Do you think we were going to get contributions from people who needed to install all of those just to fix a minor issue?
What I mean is that you're using a different version than is being tested by the other authors. So for example, if developer A used flex 2.35, but developer B has flex 2.36 installed on their system, they might get a different result (or a bug, or a security flaw in the resulting binary) etc.
I'll add that this isn't without cause - it has caused me personally many hours of errors.
Note that you can use autoconf to require exact versions, but then you require that the developer install an exact version of bison, even though they aren't touching the parser (or exact versions of autoconf even though they aren't touching the build system).
The problem I encounter regularly is that people have slightly different versions of autotools (e.g., autoconf 2.62 and autoconf 2.63).
When they commit with svn (and are not super careful), several commits have small diffs in the generated files (added newline, '\n' vs. real newline, etc.), making merges painful.
Don't think of this as, "We can cheaply compress large amounts of data to save space." Think of this as, "We can compress/decompress stuff on the fly just because it is convenient for us." For the latter kind of usage the efficiency is an enabler, and the fact that the compression is supposed to be temporary makes interacting with other things a complete non-issue.
These use cases are normally pretty application specific though, so I imagine a lot of in-house code gets written for things like this. Seems like a pretty similar use case to this, I'd be interested in seeing details of their algorithm - I can't see it anywhere obvious on their site.
Note: have you seen the LZO code? I bet not.
IMHO I think straight C would have been easier for World + Dog to link against.
I've done a quick benchmark with the files on quicklz.com on the same machine with QuickLZ 1.5.0:
test file; library; compressed size; compression speed in mb/s
QuickLZ 47.9% 308
snappy 53.0% 261
QuickLZ 1.5.0 35.6% 331
snappy 40.5% 232
QuickLZ 1.5.0 48.1% 245
snappy 55.5% 193
QuickLZ 1.5.0 45.8% 270
snappy 51.1% 214
QuickLZ 1.5.0 86.7%
snappy 91.5% 208
QuickLZ 1.5.0 23.2%
snappy 26.4% 456
I work at Google, and I've been working on the Snappy open-sourcing. First of all, let me state that when doing our benchmarks, we were definitely comparing against the latest public version (1.5.0) of QuickLZ. I'm having problems reproducing your benchmark results; could you provide some more details about the exact settings, compiler version, architecture etc.? I can't even get all of the compression ratios to match up exactly with the ones I'm seeing, so there must be some sort of difference between the setups. Are you perchance running Snappy with assertions enabled? (The microbenchmark will complain if you do, so it's easy to check.)
For what it's worth, I ran the QuickLZ test suite on my own machine (you can see the exact testing environment below), and got quite different results:
QUICKLZ: [b 1M] bytes 5922816 -> 5087471 85.9% comp 132.4 MB/s uncomp 140.8 MB/s
SNAPPY: [b 4M] bytes 5922816 -> 5369439 90.7% comp 180.6 MB/s uncomp 484.0 MB/s
QUICKLZ: [b 1M] bytes 8872609 -> 4074990 45.9% comp 177.4 MB/s uncomp 213.3 MB/s
SNAPPY: [b 4M] bytes 8872609 -> 4530851 51.1% comp 224.7 MB/s uncomp 468.7 MB/s
QUICKLZ: [b 1M] bytes 2752512 -> 628607 22.8% comp 291.3 MB/s uncomp 409.6 MB/s
SNAPPY: [b 4M] bytes 2752512 -> 726437 26.4% comp 456.6 MB/s uncomp 790.1 MB/s
QUICKLZ: [b 1M] bytes 2899483 -> 1436466 49.5% comp 155.4 MB/s uncomp 182.7 MB/s
SNAPPY: [b 4M] bytes 2899483 -> 1629408 56.2% comp 195.2 MB/s uncomp 487.5 MB/s
QUICKLZ: [b 1M] bytes 7254685 -> 2600129 35.8% comp 218.2 MB/s uncomp 261.2 MB/s
SNAPPY: [b 4M] bytes 7254685 -> 2934406 40.4% comp 269.4 MB/s uncomp 556.9 MB/s
Average compression ratio (geometric mean, lower is better): QuickLZ 43.7%, Snappy 48.9%.
Average compression speed (harmonic mean): QuickLZ 180.9 MB/s, Snappy 238.0 MB/s.
Average decompresion speed (harmonic mean): QuickLZ 212.5 MB/s, Snappy 536.9 MB/s.
QUICKLZ: [b 1M] bytes 9832475 -> 9832565 100.0% comp 289.3 MB/s uncomp 2815.9 MB/s
SNAPPY: [b 4M] bytes 9832475 -> 9485433 96.5% comp 1308.4 MB/s uncomp 2477.1 MB/s
Exact benchmark results will, as always, vary with differing compiler, set of flags, CPU, and data set. The average numbers from this benchmark, however, show QuickLZ 1.5.0 (in unsafe mode) compressing about 11% more densely than Snappy 1.0.0 does, but Snappy compressing about 32% faster and decompressing about 153% faster (ie., about 2.53 times as fast).
/* Steinar */
snappy: Compressed 2752512 bytes into 726437 (26.4%) at 653.0 MiB/s
QuickLZ: Compressed 2752512 bytes into 620933 (22.6%) at 456.8 MiB/s
snappy: Compressed 8872609 bytes into 4530844 (51.1%) at 333.1 MiB/s
QuickLZ: Compressed 8872609 bytes into 4056279 (45.7%) at 267.6 MiB/s
snappy: Compressed 18108198 bytes into 16561772 (91.5%) at 292.5 MiB/s
QuickLZ: Compressed 18108198 bytes into 15784615 (87.2%) at 197.2 MiB/s
snappy: Compressed 2988604 bytes into 1657747 (55.5%) at 289.4 MiB/s
QuickLZ: Compressed 2988604 bytes into 1436646 (48.1%) at 238.5 MiB/s
snappy: Compressed 7344249 bytes into 2964303 (40.4%) at 371.6 MiB/s
QuickLZ: Compressed 7344249 bytes into 2586659 (35.2%) at 321.9 MiB/s
Specifically the answer to your question is that printing in the destructor is there to print a newline right before calling abort but after having printed whatever was <<'ed to CRASH_UNLESS.
Check out libjingle's logging implementation which has some similar weirdness in it: http://code.google.com/p/libjingle/source/browse/trunk/talk/...
EDIT: I'm wrong:
> Actually, snappy_unittest is more than just a test, despite the name; in particular, if you have the gflags package installed, it can be used for benchmarking. Thus, it's useful to have it built as part of make all. I'm closing this one (but thanks for all the others!).