
How to save multiple numbers into one byte - seycombi
http://www.allinsight.de/how-to-save-multiple-numbers-into-one-byte-int/
======
new299
I'm a bit shocked to see this on the front page. That's possibly a comment on
me as much as anything.

But this is just really simple bit packing. Is this something many engineers
(maybe frontend/JS) don't know about now?

~~~
Walkman
I guess we never do bit shifting in higher level languages at all. E.g. in
Python, there is no point to do this, there is only one type of int (64 bit)
and it still creates a new int object, which is not very fast.

~~~
seiferteric
You certainly can do this in python and often will need to when dealing with
binary formats. The struct library is a great help for this.
([https://docs.python.org/2/library/struct.html](https://docs.python.org/2/library/struct.html))

~~~
Walkman
> This module performs conversions between Python values and C structs
> represented as Python strings.

You can, but probably it's even slower than doing regular bit shifting
operations (<<, >>, &, |, ^, ~)

~~~
deathanatos
Where you can, you should use struct. For example, decoding a 32-bit integer
in struct is faster than doing it manually:

    
    
      In [11]: %timeit hex(b[0] | (b[1] << 8) | (b[2] << 16) | (b[3] << 24))
      The slowest run took 6.89 times longer than the fastest. This could mean that an intermediate result is being cached.
      1000000 loops, best of 3: 715 ns per loop
    
      In [12]: %timeit hex(struct.unpack('<I', b)[0])
      The slowest run took 8.77 times longer than the fastest. This could mean that an intermediate result is being cached.
      1000000 loops, best of 3: 587 ns per loop
    

The resulting code is also going to be cleaner, especially if you end up
needing to unpack multiple members at once, which is common when parsing a
binary file.

The struct module is, I believe, written in C. (Pedantically, the struct
module isn't; the C module _struct is hidden/private, but if you open the
source for struct.py, you'll see it referenced.)

------
bitcharmer
This is simple binary packing approach to storing small-type data in wider
types. Delta compression is a natural next step from here. I once published a
blog post on that topic: [http://bitcharmer.blogspot.co.uk/2013/12/how-to-
serialise-ar...](http://bitcharmer.blogspot.co.uk/2013/12/how-to-serialise-
array-of-doubles-with.html)

Mind you, the reference implementation from the blog is really poor in terms
of code quality and performance because I was not allowed to open-source what
I actually developed for my client, but the ideas still stand. Delta
compression can make a huge difference in some cases; because cpu cycles are
cheap and bitwise operations are very fast it has the potential to bring
serious benefits in some scenarios, ie. lower latency of transmitting
(numeric) data over networks.

------
BurningFrog
Since 3⁵ (243) is less than 256, you can actually fit 5 "trinary" numbers into
a byte.

The code gets more complex and slow, but not really by much.

~~~
kazinator
Just table lookup.

The word is "ternary", by the way.

A useful intermediate representation would be "binary coded ternary" (BCT?)
whereby 121 gets represented as 011001. The code 11 isn't used, just like 1010
through 1111 are not used in binary-coded decimal (BCD).

The 10 bit BCT representation of a five digit ternary number could index into
an array of 1024 entries, each yielding its numeric value, that fits into a
byte.

Likewise, a reverse table of 243 values can map to a 10 bit BCT.

Conversions between BCT and textual digits are trivial.

~~~
BurningFrog
I don't understand this fully, but it seems complicated.

The way I have in mind is the same way you'd separate out the digits in a
decimal number, except you'd use 3 instead of 10 in the math.

~~~
kazinator
Suppose you have the ternary number "12102". We can encode each digit in two
bits: 01,10,01,00,10 very easily. That gives us 0110010010, a ten bit number:
a "binary coded ternary" whose value, in decimal, is 402. Now, we use that as
an index into a table bct_value[402]. That table just looks up the numeric
value of this code as an integer value in the range 0 to 242, which is also an
8 bit binary number: 146. That's our encoding of the five ternary digits in
one byte. With the precomputed table, we just look up. We can also have a
reverse table that takes 146 to 402, which we can break into groups of two
bits and map to digits.

~~~
BurningFrog
OK. That's the super optimized way to read the encoded data. Makes sense if
performance is important.

Writing is the harder part. I suppose you could just make 15 such tables, one
for each possible value you could write.

------
gefh
Alternatively, just store it naively and gzip. Or use a sparse array. Or,
best, don't do anything until storage size is an actual, measured bottleneck

------
pjscott
See also bitfields, which make the compiler generate the shifting and masking
code for you:

[https://en.wikipedia.org/wiki/Bit_field](https://en.wikipedia.org/wiki/Bit_field)

~~~
davidw
Erlang also has pretty nice syntax for handling bits & bytes.

[http://erlang.org/doc/programming_examples/bit_syntax.html](http://erlang.org/doc/programming_examples/bit_syntax.html)

~~~
pjscott
Whoa, that's some practical syntax. Suddenly I'm filled with envy that none of
the C-family languages have that.

------
Walkman
Either I don't understand something or the Resulting bytes after c should be
'11100100' and after d should be '11100110'.

~~~
BillBohan
I agree with you. His Resulting Byte is incorrect in both partC and partD.
Otherwise it was a good article.

------
smaddox
And if you need to store values that are not powers of 2, you can use the
following kind of encoding/decoding. In python3:

    
    
        from math import *
    
        def number_to_trits(n):
            # little-endian encoding
            # (i.e. smallest precision trit first)
            trits = []
            v = n
            for i in range(1,9):
                v, d = divmod(v, 3**i)
                trits.append(d)
            return trits
    
        print(number_to_trits(8))
        # [2, 2, 0, 0, 0, 0, 0, 0]
    
        def trits_to_number(trits):
            n = 0
            for i, trit in enumerate(trits):
                n += trit*3**i
            return n
    
        print(trits_to_number([2,2]))
        # 8

------
mamisp
I also really liked this page on bitpacking, which also offers a technique to
implement them using only math operations.

[http://number-none.com/product/Packing%20Integers/index.html](http://number-
none.com/product/Packing%20Integers/index.html)

I recently implemented a bitpacker in a game VM that didn't have bitwise
operations, and the math-based version worked perfectly.

------
striking
Isn't it possible to use SQLite or something for something like this? That
would greatly simplify most of these tasks, making your dataset reachable
through a fast declarative language.

As many games of Go as 100000 is, it's not so big you couldn't use a regular,
untuned database.

~~~
planteen
Bit fields certainly have lots of uses that can't be replaced by databases.

~~~
striking
I'm not questioning the usefulness of bitfields. But in this case, it would be
silly not to use a database instead.

The method that the article describes (storing two-dimensional arrays of
"empty", "black", and "white" in 2 bits) is also not the best one to use to
store Go games. Chopping up games and creating deltas from them, or storing
the moves rather than the sheets themselves and recreating those sheets on the
fly, are best for data compression.

Bitfields are great, but not for this.

------
bill45
This method is only optimal if your numbers have ranges that fit nicely into
powers of two. See here for optimal bit packing:
[https://codeplea.com/optimal-bit-packing](https://codeplea.com/optimal-bit-
packing)

------
deutronium
Neat :)

I wrote some very slow code to set/get the Nth bit in an array of bytes.

I was planning on using it for Chess bitboards -
[https://en.wikipedia.org/wiki/Bitboard](https://en.wikipedia.org/wiki/Bitboard),
but it could be used for say 8 booleans to the byte.

unsigned char getbit(unsigned char * bits,unsigned long n){

    
    
        return (bits[n/8] & (unsigned char)pow(2,n%8)) >> n%8; 

}

void setbit(unsigned char * bits,unsigned long n, unsigned char val){

    
    
        bits[n/8] = (bits[n/8] & ~(unsigned char)pow(2,n%8)) | ((unsigned char)pow(2,n%8) * val);
    
    }

~~~
pjscott
You should be able to replace the pow(2, x) part with (1 << x), so e.g.
getbit() becomes

    
    
        return (bits[n/8] & (1 << (n%8))) >> n%8;
    

The compiler will turn the division and modulo into shifting and masking.

~~~
deutronium
Yeah that's a good point! Don't know why I didn't think about that.

------
amingilani
Great read, but you should really move the motivation up to the top before you
start the explanation.

------
ChicagoDave
"a lot" is two words.

~~~
hinkley
I don't know what alot of numbers looks like, but this is what a regular alot
looks like:

[http://hyperboleandahalf.wikia.com/wiki/Alot](http://hyperboleandahalf.wikia.com/wiki/Alot)

