

MessagePack: Fast and Compact Serialization - tilt
http://msgpack.org/

======
jstephens
For perl it's fast as long as you have a small payload. Here's my benchmarks
using progressively bigger data structures.

    
    
        perl: 5.008008
        Storable: 2.21
        JSON::XS: 2.25
        Data::MessagePack: 0.34
    
        ==== Size ====
    
        .-----------------+-----------+---------+---------.
        | src             | storable  | json    | msgpack |
        +-----------------+-----------+---------+---------+
        | 1               | 4         | fail    | 1       |
        | 3.14            | 8         | fail    | 5       |
        | {}              | 7         | 2       | 1       |
        | []              | 7         | 2       | 1       |
        | [('a')x10]      | 37        | 41      | 21      |
        | {('a')x10}      | 14        | fail    | 11      |
        | +{1,+{1,+{}}}   | 29        | 14      | 7       |
        | +[+[+[]]]       | 19        | 6       | 3       |
        '-----------------+-----------+---------+---------'
        ==========================
    
        Testing: theirs:HASH (4995 bytes)
    
        ==== Serialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.14 usr +  0.00 sys =  1.14 CPU) @ 26947.37/s (n=30720)
                mp:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 70446.85/s (n=78196)
          storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 18442.45/s (n=19549)
                    Rate storable     json       mp
        storable 18442/s       --     -32%     -74%
        json     26947/s      46%       --     -62%
        mp       70447/s     282%     161%       --
    
        ==== Deserialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 23424.51/s (n=23893)
                mp:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 35136.27/s (n=35839)
          storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 25357.55/s (n=26879)
                    Rate     json storable       mp
        json     23425/s       --      -8%     -33%
        storable 25358/s       8%       --     -28%
        mp       35136/s      50%      39%       --
    
    
        ==========================
    
        Testing: xdbi:HASH (24138 bytes)
    
        ==== Serialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 2414.15/s (n=2559)
                mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 8373.83/s (n=8960)
          storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 7244.34/s (n=7679)
                   Rate     json storable       mp
        json     2414/s       --     -67%     -71%
        storable 7244/s     200%       --     -13%
        mp       8374/s     247%      16%       --
    
        ==== Deserialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 4842.34/s (n=5375)
                mp:  2 wallclock secs ( 1.13 usr +  0.00 sys =  1.13 CPU) @ 4324.78/s (n=4887)
          storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 4654.29/s (n=4887)
                   Rate       mp storable     json
        mp       4325/s       --      -7%     -11%
        storable 4654/s       8%       --      -4%
        json     4842/s      12%       4%       --
    
    
        ==========================
    
        Testing: pdf:HASH (35239 bytes)
    
        ==== Serialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.04 usr +  0.00 sys =  1.04 CPU) @ 9845.19/s (n=10239)
                mp:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 5119.05/s (n=5375)
          storable:  1 wallclock secs ( 1.10 usr +  0.00 sys =  1.10 CPU) @ 7518.18/s (n=8270)
                   Rate       mp storable     json
        mp       5119/s       --     -32%     -48%
        storable 7518/s      47%       --     -24%
        json     9845/s      92%      31%       --
    
        ==== Deserialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3110.19/s (n=3359)
                mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 2790.65/s (n=2986)
          storable:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3554.63/s (n=3839)
                   Rate       mp     json storable
        mp       2791/s       --     -10%     -21%
        json     3110/s      11%       --     -13%
        storable 3555/s      27%      14%       --
    
    
        ==========================
    
        Testing: globallib:HASH (197319 bytes)
    
        ==== Serialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.09 usr +  0.00 sys =  1.09 CPU) @ 1760.55/s (n=1919)
                mp:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 791.51/s (n=839)
          storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 1279.05/s (n=1343)
                   Rate       mp storable     json
        mp        792/s       --     -38%     -55%
        storable 1279/s      62%       --     -27%
        json     1761/s     122%      38%       --
    
        ==== Deserialize ====
    
        Benchmark: running json, mp, storable for at least 1 CPU seconds...
              json:  1 wallclock secs ( 1.71 usr +  0.00 sys =  1.71 CPU) @ 301.75/s (n=516)
                mp:  1 wallclock secs ( 1.03 usr +  0.00 sys =  1.03 CPU) @ 404.85/s (n=417)
          storable:  1 wallclock secs ( 1.12 usr +  0.00 sys =  1.12 CPU) @ 544.64/s (n=610)
                  Rate     json       mp storable
        json     302/s       --     -25%     -45%
        mp       405/s      34%       --     -26%
        storable 545/s      80%      35%       --

------
dimva
In node.js, JSON is actually faster than msgpack:
<https://github.com/aikar/wormhole/issues/3>

Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64
bit integers.

~~~
mraleph
> Also, due to V8's poor C++ API, msgpack doesn't support deserialization of
> 64 bit integers.

JavaScript _does_not_ have a number type wide enough to hold 64-bit integer.
V8 API has nothing to do with it.

------
Ralith
This sounds an awful lot like yet another ProtocolBuffers. What, if anything,
new does it bring to the table?

~~~
ericmoritz
This is schemaless. Protobufs require a .proto file that describes the data
being serialized. A .proto file serves a similar purpose as the `struct`
keyword in C.

MessagePack uses type prefixes to describe the data. For instance an unsigned
8-bit int starts with a `0xcc` followed by a byte.

------
xal
We use this as data exchange format for our RPC services at Shopify.
Incredibly fast, stable, easy to use.

------
mdonahoe
Is this actually better than gzipped JSON?

~~~
rabidsnail
This appears to be mmap-friendly. Since arrays have length values it's
possible to jump around in a large messagepack-serialized structure without
having to scan it. In JSON if you want to get to element 2 of an array you
have to scan to the end of element 1. If there's a length value at the
beginning of element 1 (as it looks like there is with msgpack) then you can
seek straight past it.

------
zdw
Does it have it suitably low memory/CPU requirements to run on embedded
devices?

How well does it work on ARM/MIPS?

~~~
frsyuki
It works very well at least on ARM.

I've used MessagePack on iPhone 3GS for the format of a dictionary file (>
100MB). I utilized mmap and zero-copy deserialization feature of MessagePack.
And I received (and merged) some patches sent by those who use it on
iPhone4/iPad2.

I've never tried to run it on MIPS. But it probably works if you use gcc.

(I'm author of MessagePack)

------
erluser
Would be nice if suuport for RPC (and generally) for Erlang was better. It's
"currently" in alpha.

