
'Memtransistor' Forms Foundational Circuit Element to Neuromorphic Computing - charlysl
https://spectrum.ieee.org/nanoclast/semiconductors/devices/memtransistor-forms-foundational-circuit-element-to-neuromorphic-computing
======
cr0sh
I'm curious about something.

Now we have something called a "memtransistor". We also have something called
a "memristor".

But invariably in these discussions, there's another device that exists that
almost never seems to get a mention. It's called a "memistor":

[https://en.wikipedia.org/wiki/Memistor](https://en.wikipedia.org/wiki/Memistor)

It was first developed in 1960, and used for a couple of related hardware
neural network architectures - ADALINE and MADALINE.

What is also interesting about this device, is that it can be relatively
easily built by a hobbyist, as shown by:

[http://www-isl.stanford.edu/~widrow/papers/t1960anadaptive.p...](http://www-
isl.stanford.edu/~widrow/papers/t1960anadaptive.pdf)

Strangely, though - the memistor is hardly ever mentioned; it does have some
downsides (mainly difficulty to miniaturize), so maybe that's it? Maybe
somebody here can play with it...

Interestingly, on a side note - it's possible to homebrew a memristor as well:

[http://sparkbangbuzz.com/memristor/memristor.htm](http://sparkbangbuzz.com/memristor/memristor.htm)

~~~
ravenstine
You just helped me rediscover a lost aspect of my childhood. Sparkbangbuzz.com
was one of a handful of science websites my 14 year old self couldn't get
enough of. I'm pretty sure that's where I first learned of the leyden jar and
was inspired to build one. I'm so glad to see that site still exists in all
its Web 1.0 glory. :)

------
cs702
I'm excited by the potential of memristors, which are already being used in
the lab for toy deep learning tasks -- for example, see
[https://www.nature.com/articles/s41467-017-02337-y](https://www.nature.com/articles/s41467-017-02337-y)
. Over time, I would expect memristors to be used for increasingly larger,
more challenging AI tasks.

The big deal about memristors for AI is that they have memory and therefore
produce different readout outputs for different input sequences over time --
out-of-the-box, without requiring any training -- and in a way that can make
sequences linearly separable under fairly general conditions. For instance, in
the Nature paper I linked to above, the researchers took a network of
memristors, which they call the "reservoir," added a linear layer with a
SoftMax on top of the reservoir, trained this hybrid network on a lower-
resolution variant of MNIST (feeding pixel values over time, as varying
voltages), and achieved classification accuracy superior to a tiny neural net
despite having only 1/90th the number of neurons.[1] Note that they only
trained the added layer; they did not have to train the reservoir. Figure (a)
in this image has a simplified diagram of the reservoir + added layer
architecture:
[https://www.nature.com/articles/s41467-017-02337-y/figures/1](https://www.nature.com/articles/s41467-017-02337-y/figures/1)
\-- only the matrix Θ had to be learned. The potential, over time, is for
having highly scalable hardware neural-net components for learning to
recognize and work with sequences.

PS. For clarity's sake, I'm ignoring a lot of important details and playing
fast and loose with language. If you're really curious about this, please read
the IEEE article and the Nature paper.

[1] [https://news.engin.umich.edu/2017/12/new-quick-learning-
neur...](https://news.engin.umich.edu/2017/12/new-quick-learning-neural-
network-powered-by-memristors/)

~~~
jimrandomh
All of the MNIST accuracy scores mentioned in these links are _terrible_ ,
even by the standards of untuned novelty algorithms.

~~~
cs702
Of course, because the nets we're talking about are really, really _tiny_.

As you can see for yourself in figure [1]c, the memristor net used in that
paper has only 15 units (yes, _fifteen_ ): five memristors in the "reservoir"
and 10 output neurons. The entire net is one linear transformation followed by
a SoftMax, with a total of 5 × 10 = 50 parameters.

It's an impressive result that only hints at the things that will be possible
when we have nets with millions or billions of memristor units.

[1]
[https://www.nature.com/articles/s41467-017-02337-y/figures/1](https://www.nature.com/articles/s41467-017-02337-y/figures/1)

~~~
jimrandomh
I don't think it says anything about what a million or billion memristor
network would be able to do. I don't think they scale in the same way normal
machine learning algorithms do.

------
forapurpose
I was speaking recently to a veteran SV hardware engineer, with a masters in
EE from Berkeley, who has worked at leading SV companies. They had never heard
of a memristor, neither the device nor the term. Do I live in a bubble, or are
you as surprised as I am?

~~~
cycrutchfield
Given that the existence of memristors was first predicted by a professor at
Berkeley (Leon Chua), yes I am surprised. Surely he had to have heard of them.

~~~
wl
Memristors are a pretty niche topic. Until somebody made some, they were an
obscure topic in circuit theory. Even now that they exist, they've yet to be
commercialized. It makes sense for a practicing EE who may not follow the tech
press all that closely to not know about them.

~~~
cycrutchfield
I studied EECS at Berkeley, but almost entirely just CS. I took maybe 1
circuits class. Even I have heard of memristors and Leon Chua.

------
crb002
Physics. You don't have to pay the speed of light transfer latency when you
store a bit locally. Not sure of how much more efficient the memristor is over
having transistors + local storage.

------
bdamm
Hopefully someone close to neural networks can describe how useful this
primitive is as a node in neural networks. If it is useful then, is this a
better primitive than simply doing it in "software" using a GPU, for example?

~~~
jokoon
I'm not an expert, but my basic knowledge of electronic and with a little
intuition, a neuron can hold memory while a transistor cannot. Since
processors are based on a separated memory and data calculation, it makes
sense that a standard unit of a processor that can do both retain memory and
do calculation should scale better, just like it seem to do in a biological
brain.

Even when you look at CPU performance, you can often pinpoint bottlenecks
right at the amount of available L1 or L2 cache. Cache locality has almost
always been the limitations of performance, because to process data, you must
first access and write data. So if memory is more closely available, then
everything should always be fast.

Also, remember you cannot do software on a GPU, because GPUs, even with CUDA
or OpenCL, are not built to run software for the simple fact that GPUs don't
do error correction. OpenCL and CUDA will only help when processing data that
can be parallelized, so where the result will not risk to be jeopardized if
errors accumulate.

~~~
twtw
Could you please clarify your last paragraph? You can definitely "do software
on a GPU," and GPUs have as much error correction as anything else.

~~~
jokoon
Read this comment:

[https://news.ycombinator.com/item?id=16436487](https://news.ycombinator.com/item?id=16436487)

You can do software on a GPU, but you cannot have good guarantees.

~~~
twtw
I don't know about AMD, but NVIDIA GPUs also have internal protection. Tesla
and Quadro GPUs have had internal error detection on registers and cache since
Fermi at least.

But this is totally irrelevant anyway, the comparison we started with is to an
analog alternative. Anything analog will have strictly worse noise and error
problems. In neural networks, errors are probably not even a problem for
analog implementation, so they definitely aren't a problem for GPU
implementations.

~~~
jokoon
I was not talking about neuron networks.

But ok, I was not aware that gpu had error correction, how much additional
transistor does it take?

------
xpuente
The real issue is not to have a few devices, but having tens of billions of
them in the same substrate.

------
crankylinuxuser
ieee articles are usually pretty good, but I like more direct results.

To purchase:
[http://www.bioinspired.net/products-1.html](http://www.bioinspired.net/products-1.html)

