
Hardware Architectures for Deep Neural Networks [pdf] - blopeur
http://www.rle.mit.edu/eems/wp-content/uploads/2017/06/ISCA-2017-Hardware-Architectures-for-DNN-Tutorial.pdf
======
neolefty
I've been wanting to know how electronics' power efficiency compares to
biological neurons, and this paper gives a clue. The most efficient hardware
it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16.

If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5
kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which
is contributing one FLOP, then I _think_ a human neuron is about _500 times_
as efficient as a GV100 [3], at 200,000 GFLOPS per watt.

[1] [https://www.extremetech.com/extreme/185984-the-human-
brains-...](https://www.extremetech.com/extreme/185984-the-human-brains-
remarkably-low-power-consumption-and-how-computers-might-mimic-its-efficiency)

[2]
[https://en.wikipedia.org/wiki/Adenosine_triphosphate](https://en.wikipedia.org/wiki/Adenosine_triphosphate)

[3]

    
    
      Neuron:
      10E6 ATP = 1 activation = 100 FLOP
      30.5 kJ = 1 mole ATP = 6E23 ATP
      1 kJ = 0.28 Wh
      ...
      2E5 GFLOPS = 1 W
    
      GV100 "Tensor Core":
      4E2 GFLOPS = 1 W

~~~
bga
Keep in mind the energy costs of the supporting hardware for that brain,
though.

~~~
TeMPOraL
Computers have supporting hardware too - powerplants (plus related
infrastructure) and cooling systems, at minimum.

~~~
mannigfaltig
Humans consume 0.7-1.3 kW per capita. A network of these chips scaled to human
brain capacity would consume around 10 kW. It is really not a big difference.
Our robot overlords are going to be quite efficient!

------
nshm
There is also a paper from same authors with about the same content:

Efficient Processing of Deep Neural Networks: A Tutorial and Survey Vivienne
Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel Emer

[https://arxiv.org/abs/1703.09039](https://arxiv.org/abs/1703.09039)

------
Shishram
link seems broken!

~~~
hacker_9
It's a 32MB PDF, as it's image heavy, and takes a while to download from their
servers.

