
Google open sourced GPipe, a library for training large ML models - ipsum2
https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html
======
visarga
Interesting, if you're a large company that can afford the compute.

~~~
zak
Actually, anyone can use a Cloud TPU v2 (one of the accelerators mentioned in
the blog post) for free via Colab:
[https://colab.research.google.com/notebooks/tpu.ipynb](https://colab.research.google.com/notebooks/tpu.ipynb)

Several other TPU-enabled Colabs are linked from the TFRC website:
[https://www.tensorflow.org/tfrc/](https://www.tensorflow.org/tfrc/)

~~~
EForEndeavour
Colaboratory is a great resource for learning and prototyping code, but it has
drawbacks that render it useless to anybody who needs to deploy anything
compute-intensive. The grandparent commenter has a point despite the
availability of free GPU/TPU compute like Colab.

Here are two dealbreakers that come to mind, with quotations from
[https://research.google.com/colaboratory/faq.html](https://research.google.com/colaboratory/faq.html):

1\. Vague disclaimers that your VM is reset (even when it's running? Probably,
but who knows?) after some unspecified maximum lifetime, and GPUs are
"sometimes unavailable" (What does "sometimes" mean? They don't tell you. Does
this apply to TPUs as well? Probably, but they don't say). Also, the mere act
of manually resetting the VM is "sometimes unavailable"!

Relevant passages: "Code is executed in a virtual machine dedicated to your
account. Virtual machines are recycled when idle for a while, and have a
maximum lifetime enforced by the system."

"Colaboratory limits how often [resetting all your managed VMs] can be done to
prevent undue resource consumption. If an attempt fails please try again
later."

"How may I use GPUs and why are they sometimes unavailable? [...] _Long-
running background computations, particularly on GPUs, may be stopped._"

2\. The only way currently to use Google's free GPU/TPU resources is to
connect through a browser running a Colab Notebook, which is Google's homemade
flavour of Jupyter Notebook, which adds Google-Drive-like live editing of the
same document, but (in my opinion) ruins critical UX/UI features that make
vanilla Jupyter Notebooks tolerable to use. Colab has worse keyboard
shortcuts, and _sort of_ offers command and edit modes, but half the time you
can't see which mode you're in, and it's not obvious or consistent at all how
and when to switch modes. Not only does Colab force over-reliance on notebooks
and Google Drive as opposed to "proper" version control, but it limits you to
a nonstandard Notebook version that makes the code-writing experience far less
fluent than it has to be.

~~~
tylerl
Yeah, if those are your deal breakers, then you seem to be a prime candidate
for the "pay for your own damn resources" tier of pricing model, that way you
can have whatever you want.

~~~
EForEndeavour
I fully agree! The purpose of my mini-rant was simply to address the parent
comment, which seemed to imply that thanks to Colab, paying for compute is
obsolete.

------
novaRom
Tried similar approach several years ago but was unsatisfied with final
accuracy results. Pipelining is obvious method to accelerate a sequence of
computations, but with respect to SGD it requires some accuracy tolerance
because weight updates go out of sync.

~~~
aoeusnth1
It seems they're not actually increasing the minibatch size here. I don't see
how the updates for N microbatches of size M will be any more out of sync than
a single large minibatch of size N*M.

