
Measuring execution performance of C++ exceptions vs. error codes - BuuQu9hu
http://nibblestew.blogspot.com/2017/01/measuring-execution-performance-of-c.html
======
arcticbull
IMO the reason not to use them isn't performance or size (although those were
legitimate factors in the past AFAIK) -- rather that they make control flow
very difficult to reason about. They're worse than a go-to statement, they're
a comes-from statement. They can pop out of anywhere and be handled anywhere
up the stack. They always seem like a good idea at the time, and end up biting
you in the butt.

Not to mention, that's old-school error handling, Result<T,E> types are what
the kids seem into these days.

Curious how destructors will affect the numbers.

~~~
delhanty
+1 for this!

Unchecked exceptions subvert the type system, so they are convenient because
they are cheating when compared with Result<T,E> types.

Result<T,E> types are the one feature that have me thinking of moving from C++
to Rust. A lot of my experimental C++ code with callbacks looks like a bit
like this now:

template <OnSuccess, OnFailure> void someFunction(..., OnSuccess onSuccess,
OnFailure onFailure) { ...

Using the above style:

1\. Allows error handling to be type-checked 2\. Uses the heap less

But it's very clunky! If anybody has a better way to achieve these two things
in C++ I'd like to know ...

Edit:

Erlang, which is known for its reliability, gets this right: kill the whole
Erlang process in a clean fashion so that we can get transactional semantics.
(It's not that performant though.)

~~~
staticassertion
> Erlang, which is known for its reliability gets this right: kill the whole
> Erlang process in a clean fashion so that we can get transactional
> semantics. (It's not that performant though.)

This is also the rust way - panics kill the thread they occur in.

~~~
delhanty
Thanks - that's good to know! Another reason for me to try rust then ...

------
juliangoldsmith
I forked this on Github and changed the C code to be more idiomatic, i.e.
using error codes instead of allocating a struct (!) and strduping an error
string (!!) the way it was originally done. It causes the C version to perform
significantly better than the C++ version.

Code is here: [https://github.com/julian-
goldsmith/excspeed](https://github.com/julian-goldsmith/excspeed)

~~~
pixel_fcker
Do you have numbers? I want to try doing the same test against the proposed
expected<T> as well but haven't had the time yet.

EDIT: From your version on a 2013 rMBP, Apple clang 8, I get (top-left 5x5
corner of the matrix)

eccCC ecccc ceCcC eceCc ceeec

compared to the original:

EEEec EEEEE EEEEE EEEEE EEEEE

EDIT2: And switching out 'throw std::runtime_error("Error");' for 'throw
int();'

eccCC eEecc eeeec ecCec Eeeee

~~~
juliangoldsmith
I ran it on my x86_64 Linux server, using gcc. The full results can be found
at
[https://juliangoldsmith.com/excspeed%20results.txt](https://juliangoldsmith.com/excspeed%20results.txt),
but the matrix output looked like this:

    
    
      ecCCCCCCCC
      ccCCCCCCCC
      ccccCCCCCC
      cccccccCCC
      ecccccccCC
      cccccccCCC
      ecccccCCcc
      cccccccccC
      ecccccCccc
      cccccccccc

------
barrkel
This is a useful topic for empirical study, because there are a lot of factors
that can change the outcome.

It's not just Java that has big stack traces; it's more complex programs that
combine multiple different abstraction layers that have big stack traces. The
easier it is to use third-party code, the more likely it is you'll have big
stack traces, because it encourages more modularity, and modules tend to be
better factored and less flattened with respect to the original programmer's
intention and code that gets things done.

Another variable not talked about here is hot code paths. Exceptions can be
slow when first used, because they can touch code or data that has been
(optimistically) removed from the main code path. This is usually done under
the rationale that exceptions are very rare, and that making code have zero-
cost exceptions is very important. E.g. exceptions on Windows x64 use table
information, while exceptions on Windows x86 use a linked list of stack-
allocated structures with callbacks, pushed and popped by exception handlers.
So you'd expect the first exception of many thrown to have a higher cost in
the x64 model, while the throughput without exceptions to be lower in the x86
model.

Ironically, manual error propagation up the stack can't be so easily taken out
of line, so it affects throughput more.

~~~
glandium
Another thing that is missing is cleanup. When you have objects that need to
be destructed at different levels of the stack, the destructors need to be
called during the stack unwinding that happens when an exception is thrown,
and I'd expect it to fill up the gap between exception throwing and error
propagation.

------
Sharlin
I wonder how often the argument "exceptions are slower than error codes"
actually means "exceptions are slower than having no error handling at all"...

~~~
krylon
My semi-educated guess: Far more often than any of us is comfortable
admitting...

------
vancan1ty
The title is misleading. Their "error codes" are not simple C error codes but
are rather structs which they are dynamically allocating upon error occurence.
Their "create_error" function below is very inefficient compared to using
traditional int error codes with a lookup table for the messages.

    
    
       struct Error* create_error(const char *msg) {
        struct Error *e = malloc(sizeof(struct Error));
        e->msg = strdup(msg);
        return e;
       }

------
stinos
Nice to know but arguably real-life C++ code using error codes written today
won't (shouln't) look anything like their C code. Ok it might not change the
outcome of the performance tests but still it would imo be more interesting to
see the results of something like

    
    
      llvm::ErrorOr< int > func0() {
        const int num = random() % 5;
        if(num == 0) {
          return func1();
        }
        //etc
      }
    

Also I'm not too sure the 'because call stacks are usually not this deep' (for
depth of about 66) claim is justified, depending on the type of application.

~~~
baq
interestingly this is similar to how Rust does it: it has a Result<Type,
Error> trait.

[https://doc.rust-lang.org/std/result/](https://doc.rust-lang.org/std/result/)

~~~
barrkel
With the try! macro, it's got the test and return built in. Further, if the
return value is large, the caller will allocate space for it and pass the
address of the space as a hidden parameter. If you've got rich return values,
the final code will be pretty much isomorphic with the example C code (it
wouldn't be a pointer to a pointer, though; it would be a pointer to a
struct).

~~~
danieldk
Note that Rust now also has the ? operator, which can be a bit easier on the
eyes than try!:

[https://github.com/rust-
lang/rfcs/blob/master/text/0243-trai...](https://github.com/rust-
lang/rfcs/blob/master/text/0243-trait-based-exception-handling.md)

[https://blog.rust-lang.org/2016/11/10/Rust-1.13.html](https://blog.rust-
lang.org/2016/11/10/Rust-1.13.html)

~~~
saurik
FWIW, C++ has this really cool feature they call "exceptions", which is a
language-level implementation of Result, try!, and ? that is automatically
included as part of any function that would otherwise have to manually unwrap
and pass along an error in languages like Rust; a really great bonus is that
because the compiler has implemented this feature and abstracted it from the
developer, in addition to it removing a lot of boilerplate from your code, it
allows for an extremely fast table-based implementation they call "zero-cost
exceptions" that can almost always completely eliminate the CPU cost of having
to constantly check and forward error information.

~~~
izym
Yes, people are quite aware of that, and if you check the comments you'll see
points made against it. Also, ? is very much language level.

------
omgtehlion

      struct Error **error
    

This is the most asinine error handling strategy I ever met. I'd choose C++
exceptions over it any day, no matter if exceptions are slower.

On the other end of spectrum is using good old error codes, not reinventing
exceptions on your own.

~~~
barrkel
If you allocate data for your errors dynamically (perhaps they have formatted
strings for informative messages), you'll want to return a pointer, otherwise
you'll have copies throughout; and if you get passed a pointer to a pointer,
you can save lots of shuffling, as I pointed out elsewhere on this thread.

There is a solid rationale behind it.

~~~
stinos
_If you allocate data for your errors dynamically (perhaps they have formatted
strings for informative messages), you 'll want to return a pointer, otherwise
you'll have copies throughout_

Not sure what you mean exactly but I don't think you need copies all over the
place since this is C++ so instances can be moved and there's also copy
elision. And should you use 'int fun(Error& error)' with error containing an
std::string for the message then at least the principle of 'you need a
pointer' is dealt with by std::string insetad of having to do it manually.

 _There is a solid rationale behind it._

Might be but from a usage point of view it seems harder to deal with than
alternatives (see 'ErrorOr' in another comment, or passing 'Error& error' or
so: these days raw pointers are frowned upon, imo with reason, and raw pointer
to pointer doesn't make that any better). Suppose I write an

    
    
      int fun(struct Error **error) {
        if(somethingWentWrong) {
          StoreErrorCode(error);
        }
      }
    

In StoreErrorCode I'd first have to check if '* error == nullptr' or not. If
not I'd have to '* error = new Error()', I guess. If '* error' does point to
an instance, what do I do? 'delete(* error)' followed by new? Or reuse the
current instance? That's imo too many questions already and I didn't even
store an error code yet. Sure the answers to those questions can be dealt with
in an error handling API used by the library. Or one could rely on some
convention that '* error' always is NULL and hope everyone follows that
convention. But unless somehowe proven that this is really really needed for
performance reasons I'd pick any easier to work with (and hence less likely to
introduce bugs) C++-style alternative over pointer to pointer anytime.

------
saurik
The obvious performance advantages of zero-cost exceptions combined with the
equally obvious benefits of "the vast majority of code, hopefully even almost
all code, doesn't need to have error checks all over the place" is why I am
really annoyed at the Rust crowd for forsaking them, which to me sounds mostly
like "I am scared of language features in a similar manner to the stereotype
of a Java developer" (which is almost ironic given that when you write Rust
you are supposed to make your code exception safe anyway, which kind of
defeats the only benefit to having avoided them).

~~~
chaotic-good
Error checks are almost free and can be considered zero cost because branch
predictor works really well with error checks (because errors are rare).
Branch predictor uses some space in L1 cache to store tables so it's not 100%
free but very close to that. Exceptions aren't zero cost either but you're
paying for them only when an error occurs most of the time.

------
jpkst19
Exceptions are for exceptional events. If exceptions are effecting the
performance of your application, you have done something horribly wrong.

~~~
scott_s
My sentiment is a relative of yours: I care more about the performance of the
two methods _when there is no error_ , as that is the common path. Hopefully.
In my own brief micro benchmarks, exceptions do have a performance penalty
when there is no error. With that said, I still use them, but I sometimes have
to think carefully about when.

~~~
alfalfasprout
If you're seeing performance degradation as a result of including exceptions
in your application the exceptions are generally not themselves to blame.

The surrounding code can be a culprit. For instance, if you have a conditional
that's not so easily predicted that in turn throws an exception then the cost
comes not from the exception but from the branch misprediction.

It all boils down to orders of magnitude. The overhead of a branch + exception
is going to be a few cycles. That said, only in a really hot code path (eg;
vectorized matrix operations) where I'm really pushing FLOPS then I'll avoid
throwing exceptions. Realtime code is another place where avoiding exceptions
is wise (since their cost is usually negligible but non-constant due to the
exception frame). Otherwise, I have yet to see a benchmark that shows
detrimental performance loss in all but the hottest loops.

~~~
scott_s
I work on (among other things) a soft-realtime streaming runtime where we care
about microsecond latency, so I care about the cost of the common data path.
You made a good point about the surrounding code being more simplistic making
it more amenable to optimizations. But, I still see that as the "cost" of
exceptions.

~~~
alfalfasprout
Yeah, I also work on systems where we care about microsecond latency (market
data handling). In most cases, we still leave the exception handling in. It's
maybe a few cycles of cost and we definitely need the whole system to stop if
any of those exceptions come up. Then again, the costs are minimal b/c our
conditional to check for an exception is easy to predict (simple comparison to
a stack variable probably on the register already).

In a hard realtime system you really have to move all the exception handling
upstream and use error codes downstream. Problem is this does hurt performance
in the average case.

------
maxpolun
I wonder what the performance difference is when you're actually handling the
errors though. I'd imagine that

    
    
        try {
           doThing();
        } catch (Exception1 e) {
            handleError1();
        } catch (Exception2 e) {
            handleError2();
        } catch (Exception3 e) {
            handleError3();
        }
    

Might be slower than

    
    
        Error err = doThing();
        if (err.code === ERROR1) {
            handleError1();
        } else if (err.code === ERROR2) {
            handleError2();
        } else if (err.code === ERROR3) {
            handleError3();
        }
    

Exceptions are best in my mind when they are very coarsely-grained, and error-
codes best when you need to handle very fine-grained errors. This is certainly
true in the code ergonomics, but I'd be interested in seeing if it's true for
performance as well.

------
delegate
Ok, so sometimes one is slower, sometimes faster. One interesting variable to
look at is _how much_ faster/slower exceptions are. Are we talking orders of
magnitude ?

Last time I used exceptions was in the year 2000 (I kid you not). I remember
that after removing the exceptions, the socket code was _several times
faster_.

I still don't use them mainly because of the unpredictability of the control
flow.

~~~
mikeash
These days, exceptions are usually implemented so that code with exception-
handling code is no slower than code without it, in the case where no
exceptions are thrown, since that's the common case. These are called "zero
cost exceptions."

It's a rather misleading term, because the "zero cost" only applies when there
are no exceptions thrown! You only get "zero cost" when you go through a try
block without throwing.

The tradeoff is that throwing an exception is pretty slow. Since all of the
normal code is built as if no exceptions happen, throwing an exception has to
go through and carefully unwind all the stuff currently in flight before it
can start executing the catch.

If your code very rarely throws, you'll be faster using exceptions. If it
throws frequently, then you could easily end up losing an order of magnitude
or three in performance.

------
pjmlp
It would be more interesting to have this measurements take place across a
wider selection of C++ compilers[1}, not just the two usual open source ones.

[1} -
[https://en.wikipedia.org/wiki/List_of_compilers#C.2B.2B_comp...](https://en.wikipedia.org/wiki/List_of_compilers#C.2B.2B_compilers)

~~~
proaralyst
If you have access to more compilers, the test code is here:
[https://github.com/jpakkane/excspeed](https://github.com/jpakkane/excspeed)

~~~
pjmlp
Doesn't seem something that would work out of the box on Windows, I will need
to look at it in a few weeks time for the VSC++ compilers I have installed.

------
matthewaveryusa
The problem is throwing. Once you start throwing on your critical path, which
will happen as the codebase and number of developers expands, you're in for a
rude awakening.

