
Tensorflow.js: Machine Learning in JavaScript - memexy
https://www.tensorflow.org/js/
======
ml_basics
I recently wanted to learn how to use tf.js, so had a couple of mini-projects.
The first is a minimal working example showing how to train a model in python
and then load it in javascript.

[https://github.com/paruby/mnist](https://github.com/paruby/mnist)

The second is the classic game of snake, controlled by pointing your head in
the direction you want to steer the snake.

[https://github.com/paruby/snake-face/](https://github.com/paruby/snake-face/)

This uses the MediaPipe Facemesh model with the device camera to work out
which direction your head is pointing in.

~~~
songzme
I have a hard time sorting through the ml logic vs the game logic, it would
really help my learning (and others too) if the game logic was decoupled from
the ml logic into 2 separate files.

------
aerovistae
As a non-ML practitioner who has an interest in the field, I have a question I
was hoping someone could answer.

Looking at the page this links to, it gives pretrained models for "common use
cases", among other things. Can you actually do anything interesting or novel
with those? I feel like any meaningful new use of ML will not be possible with
such "beginner building blocks" and you would just wind up with a weaker
version of existing offerings. Could someone give me any examples of
meaningful ways this can be used?

~~~
nl
Sure. The innovation often comes from how you use the model, not trying to
build new ones (although that happens too).

They have pose detection: [https://github.com/tensorflow/tfjs-
models/tree/master/posene...](https://github.com/tensorflow/tfjs-
models/tree/master/posenet). One day during a talk with someone who works in
aged care I worked up a demo of fall detection using this.

~~~
mncharity
And some years back I use posenet in node.js, just for its head tracking,
while exploring 3D UIs. Because it was fairly robust, at least as long as an
eye was visible. Even now, it degrades more gracefully with heads turned away
than some face nets. Though its positions are a bit noisy, which hurt multi-
camera-parallax 3D pose estimation.

------
ghj
Any example of this being used in the wild? I feel like it could save with
backend costs (especially for hobby projects) but the bandwidth/loading time
might make it infeasible?

Also if you have any novel model it will be trivial to reverse engineer it.
You gotta send the weights over to the client and they can just run
tensorflow.js themselves right?

~~~
GistNoesis
I used it in the wild for some week-end project [1][2]. To get around the
bandwidth issue, I used it as a browser extension. This means that weights are
downloaded and installed only once during the setup. But I also used it in the
past in a website for real-time audio processing when model are small enough
[3].

Novel models will probably need to have some custom layers, which are quite
painful to write. The weights for the web version will probably be a low
quality of what you achieve on a more powerful machine. And you don't need to
provide the code for the training, nor the dataset used, so you will still
have an edge over people copying you.

The main problem I have with tensorflow.js is that it's not production ready
yet. It's probably 10 years ahead of it's time. Most people don't have recent
GPU or recent video drivers. As it's doing some tricks with the GPU to have
some acceleration, a fraction of your clients will encounter random bugs and
crashes. I even got some bad review for causing reboots on Firefox :). For
some people it will be so slow it will feel unresponsive, and others will find
that it's unimpressive because you would have to use the minimal model. In a
day and age where you need 99% satisfaction to exist, picking tensorflow.js is
a mistake.

[1] chrome :
[https://chrome.google.com/webstore/detail/colorify/pipnfjhpb...](https://chrome.google.com/webstore/detail/colorify/pipnfjhpbdpejoemmkhoglekbjakmnkc)

[2] firefox : [https://addons.mozilla.org/en-
US/firefox/addon/colorify/](https://addons.mozilla.org/en-
US/firefox/addon/colorify/)

[3] Wisteria : [https://gistnoesis.github.io/](https://gistnoesis.github.io/)

~~~
ijpsud
I was playing with tf.js last year and there were some core components still
being converted to WASM, which gave me some big performance boosts once I
switched. I had some problems with WebGL-related bugs too. It seems that
they're currently in the process of writing WebGPU-based back-ends. I'm hoping
that when WebGPU is on-by-default in most browsers, and there's been a year or
two to sort out all the major teething problems, we'll see some wide-spread
uptake.

Internet speeds are still increasing exponentially, so hopefully the model
size problems will become less of an issue - perhaps aided by some CDN-served
(and thus cached across domains) "base" models that are fine-tuned with some
parameters downloaded from the server. I think I could start playing with it
seriously if I could get the model sizes under 30mb or so. In a few years
(with increasing internet speeds) that might be 50mb. I think huggingface's
distilled GPT-2 model is a couple of hundred megabytes, for reference, so
we're certainly not going to be doing anything revolutionary in the browser,
but I have a bunch of neat little ideas that I think would be useful.

For bringing the "big" stuff to the web we're probably stuck with APIs, like
OpenAI's new GPT-3 offering. I think access to SOTA models on hard problems is
going to be almost exclusively via APIs for the foreseeable future.

------
Joe-Z-L
Super noob question, I have a awesome use for image classification on my job,
can I make it on Teachable Machine export as TensorFlow Js and make a simple
page/app with simplistic UI just to drop/choose a image for the model to
compare. What would be a roadmap or path to do this ? (I started programming
literally yesterday With the intention to make what I wrote real)

~~~
ijpsud
Yep! It looks like that's exactly what the new version of Teachable Machine is
for:
[https://teachablemachine.withgoogle.com/](https://teachablemachine.withgoogle.com/)

I might have misunderstood your comment, but if you've just started learning
to code, I'd probably start with simpler tutorials and projects - just to get
a handle on the basics. Depending on your background it could take a good 6
months of daily practice to get comfortable with the basics. It took me and my
friends at least that long - similar time frame to learning a spoken language
like Japanese, say, to a basic level. Learning to code is absolutely worth it
though - 6 months is a very cheap price in my opinion. I'd probably start with
a tutorial like this one:
[https://www.youtube.com/watch?v=yPWkPOfnGsw](https://www.youtube.com/watch?v=yPWkPOfnGsw)
Or the Khan Academy computer science course.

------
jwilber
Here’s something I made awhile back that uses it to generate Bob Ross drawings
live from your sketches. (Never wrapped it up into an app but maybe I should):

[https://twitter.com/jdwlbr/status/1131244682317484032?s=20](https://twitter.com/jdwlbr/status/1131244682317484032?s=20)

~~~
gmanis
This is fantastic. Any way to see and learn from source?

------
mrojecki
Maybe here someone can help me with my struggle.

I'm trying to tackle the topic of ML / AI for some time but I don't want to
just use any existing tool and go with the flow - what I would like to do, is
to understand how everything works, including underlying math.

Just to give you an example, I would like to build my own image recogniton
tool from scratch and understand every part of it.

Where can I start? Any book recomendations would be greatly appreciated.

------
epmaybe
For an image classification project I wanted to complete, I had to do a bunch
of matrix/tensor manipulation, and I thought it would be interesting to do it
all client side in JavaScript. Things like changing the color space, taking
slices/patches of images, etc. I'd be happy to share some of that if people
are truly interested!

~~~
memexy
Yes, I'm interested. If you have notes please share. The more documented
examples there are of how to use Tensorflow.js the better.

------
jansan
I will have to purchase a new PC soon. Provided I want to play around with
Tensorflow.js, is there anything that I shoudl avoid o urgently include in the
spec? Any specific graphics card or processor?

~~~
epmaybe
If you can wait, wait as the other commenter stated. However, if you're just
interested in playing with machine learning, you don't need to get the latest
and greatest tech! Even the raspberry pi can do some machine learning
experiments or inferences.

------
revskill
Do you know how to consume model produced by python using tensorflow.js ?

~~~
ml_basics
I played around with tf.js a few weeks ago and learning to use python-trained
models was the first hurdle. I made a minimal working example of this here,
feel free to check it out:

[https://github.com/paruby/mnist](https://github.com/paruby/mnist)

See model/mnist_js.ipynb to see how the model is trained and exported to a js
readable format, and you can see in lines 13-15 in index.html how the model is
loaded.

Note: I have a decent amount of ML experience but almost no javascript
experience, so YMMV

~~~
johnx123-up
TF Lite to TFJS might be a good option. But, this ticket is pending for a long
time
[https://github.com/tensorflow/tfjs/issues/991](https://github.com/tensorflow/tfjs/issues/991)
Do you have any solution for this?

------
aphroz
Will it be blocked by Chrome as resource heavy ?
[https://blog.chromium.org/2020/05/resource-heavy-ads-in-
chro...](https://blog.chromium.org/2020/05/resource-heavy-ads-in-
chrome.html?m=1)

~~~
onion2k
That's only relevant to ads. Hopefully no one is putting AI models in adverts.

~~~
paozac
I'm afraid that will happen soon

