
Evolving Normalization-Activation Layers - memexy
https://arxiv.org/abs/2004.02967
======
memexy
> Normalization layers and activation functions are critical components in
> deep neural networks that frequently co-locate with each other. Instead of
> designing them separately, we unify them into a single computation graph,
> and evolve its structure starting from low-level primitives. Our layer
> search algorithm leads to the discovery of EvoNorms, a set of new
> normalization-activation layers that go beyond existing design patterns.
> Several of these layers enjoy the property of being independent from the
> batch statistics. Our experiments show that EvoNorms not only work well on a
> variety of image classification models including ResNets, MobileNets and
> EfficientNets but also transfer well to Mask R-CNN, SpineNet for instance
> segmentation and BigGAN for image synthesis, significantly outperforming
> BatchNorm and GroupNorm based layers in many cases.

I recently was wondering why evolutionary tactics are not used to evolve
neural network architectures and this paper answers that question. People are
looking at evolving novel architectures using evolutionary tactics.

