Google breaks the trillion-parameter ceiling with the Switch Transformer

[groar]

https://arxiv.org/abs/2101.03961

[mensetmanusman]

It’s an interesting thought experiment to consider models that have more parameters than there are data points being analyzed.

What does that mean?

[igorkraw → mensetmanusman]

It basically means you need to learn about double descent:

https://twitter.com/hippopedoid/status/1243229024085835779

and regularize your models well, implicitly or explicitly. Your model might otherwise memorize the data instead of learning features.

[rdlecler1 → mensetmanusman]

All networks are like this. In a network of N nodes you have N^2 potential relationships. That’s just simple paired relationships. You can still go to higher order groups.

[panpanna]

This is impressive but also requires a lot of power to train.

If this trend continues ML will soon surpass bitcoin as the worst polluter.