
Neural network chip built using memristors - Grognak
http://arstechnica.com/science/2015/05/neural-network-chip-built-using-memristors/
======
themeek
This is one of the exotic devices in DARPA's UPSIDE competition for exascale
computing. This initiative seeks to find non-state (non-transistor) based
approaches to computation: exploitation of nanoscale response properties of
discrete components to perform some restricted, non-binary, forms of
computation. Essentially, exotic ways to abuse silicon lithography to get
analog computation.

The idea, and this can be seen on DARPA's slides
([http://www.darpa.mil/workarea/downloadasset.aspx?id=21474857...](http://www.darpa.mil/workarea/downloadasset.aspx?id=2147485714)),
is to get computation that is several orders of magnitude higher for their
specialized sets of problems than what can theoretically be reached by
traditional computing models even if Moore's law continues.

DARPA would like to first apply this technology to ARGUS drone systems
([https://www.youtube.com/watch?v=QGxNyaXfJsA](https://www.youtube.com/watch?v=QGxNyaXfJsA))
and related technology because streaming video can't be done to the ground,
tracking and decision making must be done on board - yet traditional
processing platforms can only track a few orders of magnitude fewer targets
that what the military would like.

In a more advanced phase, if memristor or coupled oscillator (etc) approaches
to building inference models become possible, then programs written in DARPA's
other initiative (Probablistic Programming) could be programmed into these
exotic solid state devices to compute in a way more analogous to today's
generic computation. And indeed, eventually the adoption of Probablistic
Programming will train programmers to write code for quantum computers - while
more complicated, replacing Probablistic Programming's PDFs with probability
amplitudes almost get one there.

I hope to see more journalistic coverage of some of the other exotic devices.

~~~
simi_
I find it disturbing how everyone seems relatively unfazed by DARPA's intended
use of this technology.

~~~
jraines
People have, I think, internalized the logic of arms races. Someone's going to
have [scary technology X], so why shouldn't we have it first/best? (Sure,
there are answers to that, but none that I think most people would stick to
when push comes to shove).

------
FractalNerve
I'm very excited about this! Is there any possibility for me to work in a
company working on similar stuff? That would make a dream come true for me! I
live in Germany and would love to write my masters thesis a related topic!

This is so amazing! Btw. I have found an HP Invent sign in my town, but the
security guard didn't answer any questions about it. The only thing he said
was that I won't find any address or telephone number for it. That made me
curious, because HP is working on a memristor based Computer, but I doubt that
they produce it in Germany.

~~~
p1esk
Are you an EE? What exactly would you like to work on?

~~~
FractalNerve
I've previous EE experience, but I study CS. I'm open for project suggestions
and would love to do research in self-assembly for mass fabrication and
study/develop AI Models.

~~~
p1esk
Well, the obvious project suggestion is to read about challenges of building a
larger crossbar, then work on overcoming those challenges. Literature list is
provided in the paper.

This type of work all about mass fabrication, but has nothing to do with AI
models. Which direction you want to go?

~~~
FractalNerve
I think given my background I'm a better match for AI, than for mass
fabrication. That's what I'd really enjoy working on.

------
superfx
Link to original paper: [http://rdcu.be/cJ0x](http://rdcu.be/cJ0x)

~~~
forketta
[http://arxiv.org/abs/1412.0611](http://arxiv.org/abs/1412.0611)

~~~
Balgair
THANK YOU! Also, Dec 1 2014, this has been out a while then

~~~
walterbell
More recent papers at
[https://scholar.google.com/scholar?&q=leon+chua+memristor](https://scholar.google.com/scholar?&q=leon+chua+memristor)

------
meric
A neural network chip semi-conductor startup:
[http://brainchipinc.com/technology/](http://brainchipinc.com/technology/)

They "backdoor listed" on to an Australian mining company, share price went
from 1 cent to 27 cents:
[https://www.google.com/finance?cid=11163357](https://www.google.com/finance?cid=11163357)

Valued at $57m.

~~~
modeless
Man, why are all the silicon people so fixated on spiking nets? Maybe in 20
years once we figure out how the brain works they'll be great but if you built
a convnet chip instead then it could be smashing records in real problems like
speech recognition, translation, image identification, etc, _today_. Where are
the convnet chip startups?

~~~
p1esk
First of all, what makes you think a hardware convnet will perform better than
a software convnet?

Second, you can implement a convnet with a spiking circuit:
[http://link.springer.com/article/10.1007%2Fs11263-014-0788-3](http://link.springer.com/article/10.1007%2Fs11263-014-0788-3)

~~~
modeless
That paper does not implement training, only testing. Low-power testing is
good to have e.g. for mobile applications, but it doesn't increase the
capabilities of our algorithms. What we need to advance the field is faster
training of larger nets. That's where the really interesting applications will
be discovered.

A convnet-optimized chip could clearly be much faster and more power efficient
than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x
speedup already, but GPUs are hardly ideal for running convnets. For one
thing, they have tons of graphics-specific hardware that's useless for
convnets and could just be deleted in a convnet chip. For another, GPUs are
much more flexible than necessary for convnets. The main operation you need to
perform is convolution and you could make fixed-function convolution units
that would be much more power and area efficient than generalized GPU shader
cores. For yet another thing, there's no reason to believe that 32-bit IEEE
754 floating point is the best power/precision tradeoff for convnets. I'm
willing to bet that you could go much lower. You could even experiment with
approximate arithmetic; 0.5 ULP precision is probably not necessary.

~~~
p1esk
Building a hardware convnet is only beneficial when you figured out the exact
parameters of the network. Everything is hardwired. Therefore, it's useless if
you want to experiment with lots of different parameters, tricks, or
architectures to "advance the field".

Moreover, building such a chip is an expensive and long process, and given how
fast GPUs are improving, it's not clear that by the time you build it, it will
still be competitive.

Finally, if you want to speed up training, try to figure out a better
algorithm. For example, humans can learn from very few training examples.
Current neural networks need many thousands, or even millions. There's a
potential million-fold speed up in training time right here - and to find it,
you need the flexibility of the software.

~~~
sushirain
Maybe the brain can do one shot learning only after it has trained for years?

~~~
p1esk
Maybe. We want to reduce the amount of supervised training. I think that's
what Hinton is trying to do with his capsules.

------
p1esk
Memristor crossbars are exciting even outside of neural network applications:
it can be used as a very dense, non-volatile memory. If this design can be
scaled up, it could potentially replace both flash memory storage, and RAM.

~~~
Balgair
These things are really not well respected for what they can do. IBM
introduced their TrueNorth tech late last year. Those used a 'neural' design
for the chips to overcome the von Newman barriers for computer design. Along
the way, the TrueNorth chips also reduced power consumption by a LOT. However,
those designs are still digital. With a memristor in the TrueNorth set-up you
can have a similar system for input, processing, storage, and output in an
_analog_ system.

I feel that I need to emphasize that. The memristor is the component that will
easily allow for analog logic to occur at digital speeds and with digital
logic type systems ( _very_ grossly speaking). What these little guys can do
is under-sold.

~~~
p1esk
There have been dozens of analog neuromorphic chips built in the last 30
years. Latest ones use floating gate transistors for synapses. Memristors, in
theory, are better devices that flash memory (faster, lower power, more
dense), however that's just in theory. In practice, they are very hard to
scale. This crossbar is 12x12. No one knows how to build anything much larger
than that, on a mass scale.

------
noobiemcfoob
I have nothing to say but that this is a cool application of memristors. My
graduate research is in cognitive computing and the thought of using circuitry
to represent the synaptic weights as opposed to hardware/software based adders
and multipliers is pretty awesome.

------
Lozi
The general theory of memristors, meminductors, memcapacitors of any order
(first order memristor is the genuine one invented in 1971 by Professor Leon
Chua, however he generalized recently his discover to second, third order
memristor, etc.)is published in a paper I wrote with him on september 2014 in
International Journal of Bifurcation and Chaos. One can download it freely
from the site
[https://www.researchgate.net/publication/261676241_MEMFRACTA...](https://www.researchgate.net/publication/261676241_MEMFRACTANCE_A_MATHEMATICAL_PARADIGM_FOR_CIRCUIT_ELEMENTS_WITH_MEMORY)

------
chimtim
This looks cool but I'm somewhat skeptical. I would be more interested in
seeing what problem the system solves better or decently (say even MNIST)
rather than how it was built using memristors.

There is a lesson from IBM trying to mimic a rat's brain -- that is you try to
solve a problem rather than just burn power.

~~~
p1esk
>>>I would be more interested in seeing what problem the system solves better

Better than what?

~~~
modeless
Better than previous state of the art on a standard machine learning dataset.
Check some leaderboards here:
[http://rodrigob.github.io/are_we_there_yet/build/#datasets](http://rodrigob.github.io/are_we_there_yet/build/#datasets)

~~~
p1esk
What are you talking about? They build a 12x12 crossbar. The best you can do
with it is to implement a single layer perceptron to classify 3x3 pixel
patterns. Once they figure out how to scale it up, they will implement a
larger network.

------
return0
The technology sounds very promising but if the goal is to simulate the brain,
the ANN models we have today are inadequate. Current evidence suggests that it
needs to incorporate dendritic dynamics and , soon, molecular computation.

~~~
neolefty
Is the goal simulation or functional equivalence?

For example, to simulate a horse, is it necessary to create legs, or is it
okay to build a road and use wheels?

~~~
return0
possibly, we don't know yet, but ANNs are not isomorphic to real neurons. If
we are talking about a bottom-up approach to intelligence, we have to opt for
realistic simulation. If, on the other hand we knew what intelligence is, we
can simulate it any way we like.

------
peter303
Memristers are supposed to be the main memory of HP's future computing project
called The Machine. It is supposed to be as fast as register memory and
compact as flash.

------
CamperBob2
Lemme know when I can order a few on cut tape from DigiKey.

~~~
badsock
I know it's not what you're asking, but for the record you can buy memristor-
memory-based microcontrollers in single quantities from Mouser today.

The reason I mention it is that memristors are being accused of being
vaporware when really they're in production already.

~~~
Balgair
LINK! You just can't say that and not provide a link to the catalog for us!

~~~
leohutson
[http://www.mouser.com/ProductDetail/Panasonic/MN101LR05DXW/?...](http://www.mouser.com/ProductDetail/Panasonic/MN101LR05DXW/?qs=OeBdveGBEcQzg65tbPujiw%3D%3D)

~~~
lowglow
For anyone interested: [http://en.wikipedia.org/wiki/Resistive_random-
access_memory](http://en.wikipedia.org/wiki/Resistive_random-access_memory)

------
andyl
"Even on a 30 nm process, it would be possible to place 25 million cells in a
square centimeter, with 10,000 synapses on each cell. And all that would
dissipate about a Watt."

Wow - seems like a lot.

Human brain by comparison (sourced by google): \- 12 watts \- 100 billion
neurons \- 1000 trillion connections

Computing with memsisters is going to be very interesting.

~~~
thangalin
The human brain has between 100 and 500 trillion synapses and consumes a lowly
12 watts. (In contrast, a 12.6 megawatt supercomputer, in 2013, took 40
minutes to simulate one second of biological brain activity.)

The article cites 250 billion synapses per watt.

For the same 12 watts as a human brain eats up, a set of memristors could
simulate three trillion synapses. A cat, in comparison, has 10 trillion. To
get 100 trillion synapses, multiply those 12 watts by 33.3 to get 400 watts
(the draw of nearly seven 60-watt incandescent bulbs).

Since one watt bags us 250 billion synapses and 400 watts is equivalent to a
memristor-based human brain, then 400 cm^2 is the area needed to emulate the
meekest of human minds.

That's nearly the same area as half of a medium Domino's pizza.

Certainly is... food for thought.

~~~
sz4kerto
"In contrast, a 12.6 megawatt supercomputer, in 2013, took 40 minutes to
simulate one second of biological brain activity."

Just as a note: we are not sure whether that large computer really simulated
brain activity or not. The tricky thing in brain research is that we have
practically zero* idea about what matters and what can be omitted from the
simulation. (For example, the glia cells seem to be important -- until
recently, we have disregarded their role.)

So at this point even we had an infinitely big computer we could not simulate
the brain properly because we don't know what exactly to simulate.

*zero means that there's much more we don't know than what we know.

~~~
samatman
As I never tire of quipping, if you can't get it high on neurotransmitters, it
isn't a neural network.

