
eBPF Can't Count? - jgrahamc
https://blog.cloudflare.com/ebpf-cant-count/
======
zeusk
The reason why volatile variables aren't extended to 64 bit is because in
embedded and systems programming, it is often used to access memory mapped IO
registers - so any sign extension or truncation might lead to a wrong value
being sent over the bus. GCC also has a keyword (register) to make sure that
the variable is sized exactly right and fits in a register (iirc, there was a
way to even ask the compiler for a specific register to be used for the
variable in case of inline assembly being used).

~~~
jkbs
The MMIO restrictions make sense. Thanks for the explanation!

------
justinclift
There's a part which seems strange, where clang is used with "-O2" to generate
code:

    
    
      $ clang -O2 -target bpf -Xclang -target-feature -Xclang +alu32 -c sub64.c -o - | llvm-objdump -S -
    

> Apparently the compiler decided it was better to operate on 64-bit registers
> and discard the upper 32 bits.

The workaround was to use the `volatile` keyword.

The problem kind of sounds like one of the LLVM optimisation passes made the
change.

[http://releases.llvm.org/8.0.0/docs/Passes.html#transform-
pa...](http://releases.llvm.org/8.0.0/docs/Passes.html#transform-passes)

Wonder if disabling optimisations ("-O0") would have also worked?

~~~
majke
Compiling to eBPF is hard! The compiler must: \- avoid loops (backwards jumps)
- unroll everything \- inline everything - no function calls

Basically, it's impossible to use clang to generate eBPF without -O2. Sorry.

~~~
sreque
I'm new to this area of work, but has anyone stepped back and thought: "this
is not the right way to build eBPF programs"? Would it be better to create a
new high-level language and toolset? All this voodoo hackery to try to trick a
C compiler into making eBPF-compatible code feels unsustainable.

~~~
asgeir
bcc [https://github.com/iovisor/bcc](https://github.com/iovisor/bcc) if you
need low-level abstractions or bpftrace
[https://github.com/iovisor/bpftrace](https://github.com/iovisor/bpftrace) if
you need something simpler (similar to dtrace)

------
setheron
Great post. Detailed enough to keep it informative but light enough to read
along.

I've been doing some eBPF work at the moment but the CloudFlare guys are at
another level.

I'd love to take it up a notch like them. eBPF is very exciting and I'm hoping
to remain in the space.

------
yoodenvranx
I love stories like this, thx for posting!

------
antpls
If you made it works, does it mean the Spectre mitigation is basically
useless, as it can be bypassed ?

~~~
andreareina
The verifier rewrites pointer arithmetic to be safe. There's a bug where it
misclassifies 64-bit integer arithmetic to be pointer arithmetic, AIUI the bug
doesn't affect 32-bit arithmetic (i.e. it should still detect and rewrite
32-bit pointer arithmetic, so the Spectre mitigation isn't being bypassed)

------
adontz
Looks like there are no unit tests in place? Sad.

~~~
lixtra
Depending on the numbers in the test case it wouldn’t have triggered.
Depending on the user, it wouldn’t have triggered.

The other problem is that eBPF is a far way from KISS. But then that’s what’s
needed for power and performance.

EDIT after RTFA: and security against spectre and friends.

~~~
adontz
I am pretty sure that if you rewrite code, you should test rewritten version
for equivalence. GCC has about 300K tests for a reason. I would not say kernel
is less important.

~~~
lixtra
Testing for equivalence? Don’t you need to solve the halting problem for that?

EDIT: I’m curious if a down voter can provide a piece of code that tests
whether two compilates are equivalent.

The root cause of the bug in the article was introduced by a patch to
automatically modify an arbitrary program to harden against spectre.

~~~
adontz
Also, I'm not talking about theoretical proof of equivalence, I'm talking
about simple test like

    
    
        assert func(params) == rewrite(func)(params)
    

for some reasonable set of func and params. Totally enough to catch problems
like this. I'd say [unit-]testing that subtraction operator works is not that
hard.

~~~
lixtra
Except, that in this case params is a string containing an arbitrary program.

It would help the discussion if you read the TFA (edit: more carefully).

Rewrite() in this case modified the params program to harden it against
Spectre.

~~~
marcinzm
That argument applies to pretty much every single unit test ever written. A
function running on a single long can take 2^64 possible values. Impossible to
test by your logic. Yet they're tested without issues constantly.

What you do is put together a long list of sample functions and sample
arguments that covers the expected edge cases and then test those for
equivalence. Hardly impossible. Just takes time. Not bulletproof but better
than nothing.

