
TensorFlow on Mobile: TensorFlow Lite - gk1
https://towardsdatascience.com/tensorflow-on-mobile-tensorflow-lite-a5303eef77eb
======
zitterbewegung
Some people are wondering if Tensorflow Lite would support CoreML / iPhone's
neural engine. Tensorflow is not supported with coremltools (but I suppose
Google could contribute a patch for this). I'm not sure about the CoreML
libraries on the phone but from my understanding it may work.

Keras is supported out of the box with the Tensorflow backend though. See
[https://medium.com/@JMangia/super-simple-end-to-end-test-
of-...](https://medium.com/@JMangia/super-simple-end-to-end-test-of-keras-
tensorflow-and-coreml-f247ab73fb42) and
[https://github.com/apple/coremltools](https://github.com/apple/coremltools)

~~~
rambossa
not supported (or is this the patch you mention)?
[https://developers.googleblog.com/2017/12/announcing-core-
ml...](https://developers.googleblog.com/2017/12/announcing-core-ml-
support.html)

------
JepZ
Would be super cool if they would even port their stuff to Web Assembly. Now
we have some Android and some iOS solution, but webasm would be truly platform
independent.

~~~
jahewson
Without SIMD I’m guessing the performance wouldn’t be great.

~~~
deepsun
Doesn't JIT automatically SIMDifies some operations on arrays? Although I
agree it would be better to have a control over that, instead of hoping JIT
compiler would do the right thing.

~~~
dmitrygr
Generic automatic vectorization remains very much an unsolved problem. Outside
of very specific cases, even modern compilers very much suck at it. That's why
most vectorizable libraries have hand-coded SSE/NEON assembly blocks. I would
not expect a JIT, which has to be fast, to have time to auto-vectorize
anything at all, much less to do it well.

------
Quenty
I’m surprised it doesn’t take advantage of the iPhone’s “neural engine” if
it’s available.

~~~
RubenSandwich
Apple uses a different model format[0], so tensorflow models need to be
converted first before they can be used. So Apple would need to add support
for this tensorflow lite/mobile model format to work in Core ML.

[0][https://developer.apple.com/documentation/coreml/converting_...](https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml)

~~~
J0-nas
Another format is onnx [0], where Apple and Goodle don't seem to participate.
I don't know the politics behind it but there should be a common format for
all libraries/platforms.

[0] [https://onnx.ai/](https://onnx.ai/)

~~~
stochastic_monk
onnx seems to be an initiative to try to allow Microsoft's and Facebook's AI
platforms to compete with Tensorflow.

Considering Tensorflow is more a grab at developer mindshare than an ideal
platform [for example, its performance lags by a factor of two behind MXNet
and Torch], I think it's a smart plan.

~~~
J0-nas
I don't think the last sentence is fair to trensorflow. Torch has been around
for ~15 years compared to the 3 of TF. You'd expect TF to catch up in terms of
performance in the future.

~~~
stochastic_monk
I think it's more than fair. I'm certain that more developer-hours have been
spent on TensorFlow than on Torch.

They very well may catch up, but that doesn't mean that hype and the Google
name aren't the reason it's gotten popular.

------
sandGorgon
The training export from Google CloudML -> Tensorflow Lite is still not clear.
Since the freeze_graph scripts use Bazel, it is hard to pull this off on the
cloud

~~~
reubenmorais
You can call freeze_graph directly from Python code, it's in
tensorflow.python.tools.freeze_graph.
[https://github.com/tensorflow/tensorflow/blob/ba032db1362794...](https://github.com/tensorflow/tensorflow/blob/ba032db13627945e7cc772dbd3d85d257aef3ab9/tensorflow/python/tools/freeze_graph.py#L209)

------
n8henrie
I’m looking forward to some tflite on Raspberry Pi tutorials.

~~~
bengalister
An issue has been created on github:
[https://github.com/tensorflow/tensorflow/issues/14589](https://github.com/tensorflow/tensorflow/issues/14589)

It seems that it will be supported but there's no timeline.

~~~
lovelearning
It looks like this issue has been resolved to a reasonable extent.

I just tried building TFLite for a lark on a Raspberry Pi 2 and succeeded
without much hassle and any of that bazel nonsense. It's a standalone
component - they provide a Makefile and I just had to run make.

The example label_image inference app took a bit more work - I had to write my
own Makefile - but wasn't difficult. I'm able to run inference on images on
RPi2 with both floating and quantized MobileNets tflite models. Will try
videos now.

For anybody else interested in trying out TFLite on your desktop or Raspberry
Pi or any SBC, I've put up instructions and Makefile at
[https://pastebin.com/LDEGyG46](https://pastebin.com/LDEGyG46)

~~~
alex_hirner
Thanks for providing those instructions. What kind of performance do you get
for both MobileNet versions?

~~~
lovelearning
I did not do any performance profiling yesterday. I am currently interested in
deploying object detection models for video streams, and plan to do detailed
profiling of those when ready.

That said, the label_image classification example does provide some timing
information. On the 500x600px test image that comes with it, non-quantized
classifier for 10 iterations took ~900 ms on a RPi2 while quantized took ~300
ms. The confidence values were much lower in the quantized version, but I
didn't spot any major misclassification.

------
makapuf
Just out of Fosdem and was attending a very interesting talk on tensorflow
running on microcontrollers (uTensor project) not dissimilar to this.

------
jacksmith21006
Be curious if Google will add support for the PVC in the Pixel 2 phones?

