On: RTX 2080 Ti Deep Learning Benchmarks

[sabalaba]

Stephen from Lambda here. These are, to our knowledge, the first public Deep Learning benchmarks for the 2080 Ti on actual hardware. We've provided all of the code and tooling to reproduce these results yourself so, if you have a 2080 Ti or another GPU you want to benchmark, please feel free to reproduce these results yourself:

https://github.com/lambdal/lambda-tensorflow-benchmark

[nabla9 → sabalaba]

Thank you.

You compared each model fixed number of times (images/sec). What was the training accuracy difference between FP32 and FP16? Do they match?

I would like find out the efficiency gain when training he same model to same accuracy with different floating points types and the same model size.

[sabalaba → nabla9]

The training accuracy won't match. Here's an excerpt from a presentation on FP16 training to give you an idea of the loss in accuracy:

    Mode                     | Top1 accuracy, % | Top5 accuracy, %
    -------------------------+------------------+---------------------
    Fp32                     | 58.62            | 81.25
    Mixed precision training | 58.12            | 80.71
    FP16 training            | 54.89            | 78.12

[1] http://on-demand.gputechconf.com/gtc/2017/presentation/s7218...

[wmf → sabalaba]

Were tensor cores used?

[sabalaba → wmf]

yes

[wmf → sabalaba]

Interesting. I would have expected FP16 with tensor cores to give at least a 100% speedup, not just 60%.