
Darts: Differentiable architecture search for convolutional, recurrent networks - kumaranvpl
https://github.com/quark0/darts
======
JustFinishedBSG
I'm pleasantly surprised by how well it works! No offense to Barret Zoph, Quoc
Le et al but the "let's throw 10000 GPUs and TPUs at the problem" approach was
frankly absurd.

And a huge +1 for the inclusion of random architectures in the results.

~~~
osipov
Why do you think that throwing compute at the problem is absurd? Arguably
that's the approach that produced the resurgence of interest in CNNs.

~~~
chadmeister
Yes I think the whole point of Zoph and Le's research is to empirically
demonstrate the desired trade off of compute cycles vs feature engineering

------
ashelmire
Cool, was thinking about this subject earlier today.

Abstract:

>This paper addresses the scalability challenge of architecture search by
formulating the task in a differentiable manner. Unlike conventional
approaches of applying evolution or reinforcement learning over a discrete and
non-differentiable search space, our method is based on the continuous
relaxation of the architecture representation, allowing efficient search of
the architecture using gradient descent. Extensive experiments on CIFAR-10,
ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in
discovering high-performance convolutional architectures for image
classification and recurrent architectures for language modeling, while being
orders of magnitude faster than state-of-the-art non-differentiable
techniques.

Zoph's paper on architecture search:
[https://arxiv.org/abs/1611.01578](https://arxiv.org/abs/1611.01578)

Concept of architecture search from ages ago:
[https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...](https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886cea3/t/5909113c1b631b40f8137956/1493766462349/1989+neural+networks.pdf)

------
aabajian
Not to get existential here, but the analogy to human neural networks is
compelling. The algorithm efficiently redesigns itself to fit the data, just
like neural synapses reconnect one another as knowledge is learned, to
optimize the storage and retrieval of the information.

