
Low Level Bit Hacks You Must Know - jonpaul
http://www.catonmat.net/blog/low-level-bit-hacks-you-absolutely-must-know
======
mahmud
Here is a fun challenge for language snobs: implement bit-streams for your
favorite language. Here is a simple signature:

    
    
      bitvec read_nbits (unsigned int count, bitstream input);
    
      bool write_nbits (unsigned int count,bitvec bits,
                       bitstream output);
    

Then add a multirecord I/O. That is, read a record of bitvecs, each N bits
wide, where the length is not given but encoded in the bitstream in this
manner: read a bitvec record of N bits, if the high-bit is set (take it in
whatever endian you like), read another record and repeat, if the high bit is
not set, return whatever you read thus far.

Then add the ability to return the bitvec records as single integers (bignums
even) both signed and unsigned, taking them in this manner. If the bitvecs are
unsigned, each record contributes its 7 least significant bits, and the high
bit is dropped. Taking the bit on either end of the first record as the most
signficant, iterate over the rest and shift and OR according to your endian
needs to construct an integer from the sum of all the bits.

When I first did this exercise, I was very close to gouging my own eye-balls
out, but after I did it, I started to think of bits as something very natural,
and not to be feared.

~~~
fleitz

      let nthbit (bitContainer:'T) nth : bool = (bitContainer &&& ((1 :> 'T) <<<nth))
    
      let bitstream (bitContainerSeq:seq<'T>) : seq<bool> =
       let sz = sizeof<'T>
       bitContainerSeq 
       |> Seq.map (fun b -> [0..sz] 
                            |> nthbit b 
                            |> Seq.ofArray )
       |> Seq.concat
    

I think that should do it unfortunatley I'm away from an F# compiler

~~~
mahmud
This looks magical :-)

Are you sure you're decoding bits from an octet stream (normal 8-bit bytes)
and not using logical booleans? I ask this because I don't see any bit-
manipulation operators, except maybe for "1 :> 'T" which looks like the SML
module type casting, but I could be wrong.

~~~
fleitz
Yes very sure. nthbit will extract the nth bit from an integer type. The
bitwise logic is the &&& and <<< in F# you use triples for bitwise. But there
is a bug in there, i forgot to times sz by 8 <http://msdn.microsoft.com/en-
us/library/dd469495.aspx>

bitstream turns a sequence of integer types into a sequence of bits.

So if you wanted to turn a stream into bits you'd do something like the
following.

    
    
      new System.IO.FileStream('foo.txt')
      |> Seq.unfold (fun s -> (s , match s.read with
                                   | -1 -> None
                                   | x -> Some(x)))
      |> bitstream
    

I just realized that the bitstream function is unnecessary and you could
instead write

    
    
      let bits (bitContainer:'T) : seq<bool> = 
         [0..(sizeof<'T>*8)]
         |> Seq.ofArray
         |> Seq.map (fun nth -> (bitContainer &&& ((1 :> 'T) <<<nth))
    

And change the above code to:

    
    
      new System.IO.FileStream('foo.txt')
      |> Seq.unfold (fun s -> (s , match s.read with
                                   | -1 -> None
                                   | x -> Some(x)))
      |> Seq.map bits
      |> Seq.concat

~~~
fleitz
Just for fun here's the reverse to turn a sequence of booleans into a sequence
of integer types (byte,int,long,etc). Note that the length of sequence s must
be a multiple of sizeof<'T> times 8

    
    
      let bitsToBitContainer (s:seq<bool>) : seq<'T> =
        let sz = sizeof<'T>*8
        s
        |> Seq.scan (fun (i,n) x -> (i++,(n<<<1)|||(x&&&1)) (0,0)
        |> Seq.filter (fun (i,n) -> i = sz)
        |> Seq.map (fun (i,n) -> n)

~~~
mahmud
I am liking this F# thing. Thanks! :-)

------
bhavin
As much as I love bit manipulation, I have to say that most (not all) of them
don't provide any real performance gain for C/C++ code (only some geeky
satisfaction of writing difficult to interpret code!).. e.g. with any decent
compiler...

* if ((x & 1) == 0) performs same as, if ((x % 2) == 0)

* if (x & (1<<n)) same as, if (x% (pow(2,n)))

so on.. on the other hand, i find bit operations really handy when the
variables in question are to be treated as separate bits than normal base-10..

~~~
wingo
Agreed regarding modulo, but `pow' returns a floating-point number.

~~~
bhavin
sorry, what i meant there wasn't literally pow function per se, but a function
which does the job..

------
TimMontague
Here's a much more comprehensive collection of bit hacks:

<http://graphics.stanford.edu/~seander/bithacks.html>

~~~
IgorPartola
Ha. Next time someone asks how to swap two values at an interview question (I
think this is pretty common):

    
    
      #define SWAP(a, b) (((a) ^= (b)), ((b) ^= (a)), ((a) ^= (b)))

~~~
cs_loser
Is it pretty common? As someone who does a fair number of software engineer
interviews, that's a trick question. The real answer to "swap two vars with no
temps" is:

1\. Don't be clever in our code base. Use a temp variable. 2\. There's various
dumb tricks with XOR, and possibly add/subtract if overflows don't break. 3\.
A sequence of several instructions where each of them requires the result of
the previous one may not execute particularly fast on modern processors.
Instruction/cycle counts -- like 3 -- are great when there's no pipeline and
no cache, but otherwise pretty much useless. 4\. The things you're swapping
might be local variables, and when the compiler has -O <anything> specified,
local variables start getting weird, and "swap" can sometimes be done in
_zero_ instructions, namely by the compiler noting that they have now been
swapped and using the other one for the rest of the basic block. (or further
dominated basic blocks for that matter) 5\. If the things you're swapping are
in main memory, or even if it's not in L1, you're going to be incurring a cost
much greater than the temporary use of a register. (and, if you don't know
where they are and it might be main memory, this might dominate the average
runtime)

The answer is _definitely_ not "three xors".

~~~
mentat
Embedded and firmware engineering questions expect this as an answer. Anyone
following your advice will not be taken seriously. This is true regardless of
whether you're correct factually. Readers of this thread deserve to know that.

~~~
leif
3 is actually a quite valid point for embedded. Any swap actually will be
expensive when it comes to keeping cache lines clean. The correct answer in
that case is just to not swap the variables, and instead swap their uses later
on:

    
    
        int x, y;
        ...
        SWAP(x, y);
        foo(x, y);
    

becomes

    
    
        int x, y;
        ...
        foo(y, x);
    

(naturally, this is why I still eagerly await the arrival of a C compiler that
has macros with LISP power)

~~~
pjscott
In that case, you can get the right behavior by just swapping the variables
using a temporary variable. If the compiler is decent, it'll automatically
swap their uses later on.

I don't know how every compiler works, but if you use Clang (or anything LLVM-
based), it converts everything to Single Static Assignment (SSA) form:

    
    
        int x1 = 42, y1 = 666;
        ...
        int tmp = x1; x2 = y1; y2 = tmp;  // SWAP(x, y)
        foo(x2, y2);
    

In SSA form, the value of a variable does not change, so it ends up creating a
bunch of "imaginary" variables to hold intermediate values. From there, it
does optimizations, then figures out how best to allocate registers, and what
needs to be stack-allocated.

~~~
leif
Cool, thanks. I know far too little about compiler optimizations, it's always
nice to hear about them.

------
CountHackulus
If you're interested in reading about more of these, and how they work I'd
highly suggest reading the book Hacker's Delight by Henry S. Warren[1]. I know
that I've used more than a few of the tricks in my day-to-day work.

[1] [http://www.amazon.com/Hackers-Delight-Henry-S-
Warren/dp/0201...](http://www.amazon.com/Hackers-Delight-Henry-S-
Warren/dp/0201914654/ref=sr_1_1?ie=UTF8&s=books&qid=1287586055&sr=1-1)

~~~
swaits
Indeed, a great book. It's also mentioned in the article.

------
darwinGod
And, Here's a draft of Bitwise Tricks and Techniques, by Knuth.

<http://www-cs-faculty.stanford.edu/~uno/fasc1a.ps.gz>

------
larsu
_Bit Hack #6. Turn off the rightmost 1-bit.

Now it finally gets more interesting!!! Bit hacks #1 - #5 were kind of boring
to be honest._

Does anybody know a practical use case for that? I have personally never
encountered a situation were I needed to manipulate the right most 1-bit.

Otherwise it's a nice introduction to bit hacking.

~~~
barrkel
Let's say you use a bitmask for keeping track of free registers in a compiler.
To "allocate" a register, you'll want to clear a bit, to indicate that the
register is no longer free.

That's the context I learned this trick in. It's used in the Embarcadero (ex-
Borland) compilers (Delphi, C++, etc.).

------
yason
These aren't _hacks_ really: they're just basic operations in binary
arithmetic that anyone who has seriously coded C or assembly must have bumped
into, probably quite early.

Other's have already mentioned but for real hacks in the unintuitive sense,
check out Hacker's Delight and
<http://graphics.stanford.edu/~seander/bithacks.html>.

------
Dilpil
Is working on embedded systems really this fun? I might have to change
careers...

~~~
bryanlarsen
It was 10-20 years ago. The funnest project I ever worked on was to implement
a 17-function motor controller with 40-bit precision onto an MCU with 2K of
RAM and 64 BYTES of RAM. That was in '93.

Today embedded systems generally run Linux. You get to write a bit of assembly
language code in your bootloader, and then it's just bog standard Unix
programming. It's even likely that you'll be doing most of your coding in a
scripting language, although it's more likely to be Lua than Ruby...

~~~
oiuytgfrtgyhuji
There are still a lot of 8051s and MSP430s out there.

The fun thing about embedded is that since everything is done as a result of
interrupts or clock signals it's all the joys of multitasking without a
threading library.

I just got handed a project where the 'app' was just main() { while(1) {} } !
Everything happens as the result of functions that get magically called when
certain bit patterns appear

~~~
mahmud
Look into Esterel for some theoretical grounding. It will help you reason
about reactive systems.

------
nait
Here are some more bit hack algorithms <http://www.aggregate.org/MAGIC/>

The intersection of tricks on all those sites is quite large but on every one
of them there's something you haven't seen before.

By the way if you like to wrap your mind around such tricks you might also
find some gems here: <http://www.azillionmonkeys.com/qed/asmexample.html>

------
js2
Previous submission <http://news.ycombinator.com/item?id=681282>

------
grobolom
Great tutorial, but I wish he'd given some context for these. The even/odd is
a well-known one, but propagation of rightmost 1-bit? I have no idea what
that, or most of these, are used for.

------
golgo13
Also works in SQL Server: <http://msdn.microsoft.com/en-
us/library/ms176122.aspx> Sweet!

------
makmanalp
If you like this stuff, you'll love the book "Hacker's Delight" which is just
full of these:

<http://www.hackersdelight.org/>

------
newgrad
oo~ number 7 is interesting

