
The Cost of Training NLP Models: A Concise Overview - jonbaer
https://arxiv.org/abs/2004.08900
======
asdf_snar
I don't understand how the conversion from FLOPs to dollars is done. A
footnote says that "These $ figures come with substantial error bars, but we
believe they are in the right ballpark." Where are the estimates themselves
coming from? Cost of cloud computing? Cooling and power consumption estimates?
I feel the article would benefit from adding a sentence fragment in the
abstract or introduction that specifies this as I have no idea what I'm
looking at right now.

------
chroem-
I think the biggest advancement that could come out of the ML space right now
is the equivalent of Big O notation for ML. Our current algorithms suck, and
the first step to making them suck less is to precisely measure just how much
they suck.

~~~
tixocloud
It’s similar to our viewpoint as well. Currently, it’s easy to see gains
because we’re mainly going from highly manual processes to 1st generation AI
applications. Future generation AI applications will mostly be optimization
and squeezing out very tiny performance gains. We want to be able to highlight
eventually how much things will cost and whether it’s even worth spending so
much for a tiny amount of gain.

There’s so much hype right now that silly money is just being thrown at new
tech without fully understanding the value you’re getting.

