
What I learned from looking at every AI/ML tool I could find - amrrs
https://huyenchip.com/2020/06/22/mlops.html
======
proverbialbunny
>A large portion of AI investment is in self-driving cars, and as fully
autonomous vehicles are still far from being a commodity, some hypothesize
that investors will lose hope in AI altogether. Google has freezed hiring for
ML researchers. Uber laid off the research half of their AI team. There’s
rumor that due to a large number of people taking ML courses, there will be
far more people with ML skills than ML jobs.

Ouch >.<

As someone who does ML for a living this hits. When switching companies in
2019 for the first time I had a hard time despite having 10+ years experience.
Linkedin maxed out saying over 200 people had applied to every job I was
looking at.

Thankfully I have a rare skill set. It's not found in classes, books, or boot
camps. On the downside there are not many jobs to choose from as companies
don't know what I do can be done. (Time series classification and prediction,
usually sensor data.)

Hopefully the industry matures. I've bumped into data scientists who don't
know how to do feature engineering as well as ones who can't grok ML. It makes
the whole industry look bad.

~~~
TrackerFF
But keep in mind, the VAST majority of those people have only toyed around
with ML libraries or frameworks on the most trivial data. There's A LOT of
script kiddies right now, that just copy/paste the same tutorials, maybe even
publish yet another "How to classify MNIST dataset with NN / SVM / Naive Bayes
/ etc." on medium or whatever, just to come off as authority.

So while there's a lot more competition, the majority of these don't even meet
the minimum standards to get called into interviews.

If you have solid work experience to show for, then that alone should
differentiate you from the the masses. Again, a lot of these people have NEVER
worked with anything relevant, and can only show you a bunch of tutorials
they've followed. IF lucky, they've actually taken on more exciting datasets
from sources like Kaggle, and tried to come up with something.

edit: Not to mention the data - I get extremely impressed when someone
actually shows a project which involves them collecting data from the wild,
wrangling it, and then doing meaningful work.

Too many beginners _only_ work on neat / ready datasets. You'd be surprised
how many candidates get completely blindsided when asked to do some trivial
tasks on pre-processing data.

~~~
bulldoa
> minimum standards to get called into interviews.

I am a bit curious on what is the minimum standard? It does feel to me (an
outsider that took 1 class of ML in college) that you need at least Master in
ML to get intuition on the probabilistic and linear algebra theory behind ML
concepts.

~~~
proverbialbunny
>I am a bit curious on what is the minimum standard? It does feel to me (an
outsider that took 1 class of ML in college) that you need at least Master in
ML to get intuition on the probabilistic and linear algebra theory behind ML
concepts.

On the computer science side it makes sense to understand how to build a tree,
not just know how to use one. In computer science you're learning how to write
algorithms, not just use them. On the MLE or Machine Learning Software
Engineer side, the same rule applies and knowing linear algebra as well as
probability theory is helpful so you can write your own ML.

On the data science side you're rarely inventing new ML. What you want to
specialize in is: 1) data mining using ML, 2) data cleaning, by knowing what
kind of input ML needs, 3) feature engineering, also by knowing what kind of
input ML needs, and 4) What kind of ML is ideal to choose, eg, due to the
bias/variance trade off.

A lot of these boot camps, classes, and books teach the underlying structure
of ML to become an MLE, though for some sort of reason they often advertise it
as data science. I find this odd, because MLE pays better and is in higher
demand.

On the data science side we rarely need to know these finer details. We just
need to know the ML's characteristics so we know when it is the right tool for
the job, similar to knowing how to use a data structure but not needing to
know how to invent new data structures.

A data scientist needs to be a research specialist which is more of a phd than
masters skill, so knowing the underlying math also doesn't matter as much
because we know when it is necessary to research it. It's not that a data
scientist can't know it, and many do know it out of hobby or classes, but it's
far from mandatory. Knowing probability theory and how to digest a problem
into multiple paths forward, like how to collect data, is far more valuable as
a skill.

And finally, many data scientists barely know how to write code. They're a
kind of analyst. I feel like the job title isn't sufficiently explained so
software engineers make a lot of assumptions mixing up MLE with DS.

edit: Also, Linear Algebra isn't that bad and is an undergrad class.
Probability theory is taught lightly in DS101 freshman year, but the first
year of getting a masters probability theory is often taught again to a much
more rigorous. This can get a bit harder, but if you understand the basics
it's not bad.

------
peatmoss
I love this kind of software meta-analysis. I have idly dreamed of setting up
a site that essentially did literature review of existing solutions and
approaches.

I thought it would be a good contribution for a language with strong academic
roots like Racket. I suspect a lot of good libraries remain unwritten in part
because ecosystems tend to grow organically. Organic is good, but I’m positive
that even basic meta-analysis like linked article would lead to better
libraries in more languages, with less labor expended.

~~~
kfk
Same thing for enterprise solutions. How does Redshift stack up against
Snowflake? What about choosing a dashboard tool within the various ones that
exist today? Etc.

------
nl
I do this stuff foe my day job, and everything here rings true.

Every part of the "build and use ML in production" workflow is horrible
(unless maybe you work at Google).

Firstly, the Datascience workflow is NOT the same as software engineering.
Things like version control tools don't work properly (in every part: git on
Jupyter notebooks doesn't work without hacks, versioning data is horrible,
versioning models is horrible).

Deployment is horrible. Sagemaker (and equivalents) provide the very base
level of functionality needed, but are so separated from the feature
engineering side that everyone ends up doing vast amounts of work to get
something useful.

Frameworks are horrible. TensorFlow did the upgrade to TF2, so half the
examples on the web don't work anymore. The TF data loading abstractions are
great - if you work at Google but so complicated to get basic examples going.

PyTorch has a horrible deployment story.

Every other frame work are either experimental research things and take months
to make progress (JAX) or are so far behind modern work they are useless
(MXNet).

But the thing is dealing with all these issues is worth it because ultimately
it does actually work.

~~~
jiggunjer
Any experience with ML in MATLAB?

~~~
nl
Yes.

I mean it's fine, but I don't see any reason to use it instead of Python, and
lots of reasons not to. But I'm not a mathematician by training.

I do quite like RStudio though, and I do see places where that is useful. So
maybe MALBAB fits somewhere in between - less stats than R, less programmign
that Python.

~~~
langitbiru
What do you think of Julia?

~~~
nl
> Julia

The Haskell of Machine Learning.

[https://www.jwz.org/doc/worse-is-better.html](https://www.jwz.org/doc/worse-
is-better.html)

------
kjw
A great survey of the current landscape of ML tool innovation.

This reminds me of the explosion of "big data" tech, which feels like it
started exploding in 2010 and peaked maybe five years later.

If ML follows a similar cycle, then in perhaps five years, most tools will
have receded into obscurity but a few frameworks and approaches will become
dominant. Big stakes and justifies the prevalence and investment in OSS.

Comparison of "big data" and "machine learning" in Google Trends --
[https://trends.google.com/trends/explore?date=all&geo=US&q=b...](https://trends.google.com/trends/explore?date=all&geo=US&q=big%20data,machine%20learning)

~~~
etaioinshrdlu
I literally worked at a company that did a major project called "Big Data". It
was in 2013/2014\. It used Hadoop. It was pretty useless :)

------
stephc_int13
Aside from the irrational but probably inevitable over-hype, I find this whole
AI/ML space quite fascinating and frustrating at the same time.

Most of the successful algorithms are much less complex/sophisticated than you
might believe from the outside, and I believe that's a very good sign because
simplicity scales.

But on the other hand, there is a huge amount of brute-forcing, you can't
really do serious research in this field without a few dozen millions to
spare, and I think that's always a limit on the innovation potential.

Unfortunately writing good code/finding new algorithms is only a small part of
the problem.

I also think that many people don't completely understand how weak this model
is to process real-world data when there no reliable way for the system to
self-train on it.

~~~
cambalache
I agree with you. I was in the Michael Jordan camp, as in: Machine Learning is
just statistical inference rebranded and with better marketing. There are lots
of cool applications for sure but there is no "magic sauce" beyond your usual
models obtained by error minimization. But lately, especially seeing the
progress with the Attention/Transformer models I have been slowly letting
myself convinced that maybe that is all there is behind human intelligence:
Lots of priors encoded by evolution in the brain, and general quasi-brute-
force mechanism for inference, deduction and imagination. Of course this is
probably a naive view, but I think the main point will remain: Intelligence
can be created or simulated by giving an agent the capability to extrapolate
in a way that "makes sense" in the environment the agent will live.

~~~
ieatpies
Attention based models are still extremely data inefficient and fail to
generalize. I recommend reading the "On the Measure of Intelligence" paper by
Chollet if this topic interests you.

~~~
cambalache
I read the paper, I like Chollet, but he comes across as too pesimistic in
that paper. I agree that current models are still very data inefficient, but,
despite this, I think they are not that far away from the mark if you consider
2 things. 1)The brain probably has hardcoded a lot information. Things like an
universal grammar a la Chomsky, the concept of continuity of time and
space.Just add water and boom a 2 year old understands. The other thing is
that humans are great extrapolating from few data points for a limited set of
cognitive tasks, not all.

------
reureu
Awesome work - it must have been super interesting to dive into so many
difference companies! The current state work that you describe is super
fascinating.

Be wary of survivorship bias when you're using today's landscape to draw
historical conclusions. If you're only looking at companies/tools that exist
today, taking their founding date, and using that as a sign of what existed
circa 2012, then you're going to miss the plethora of companies/tools that
failed prior to today. Similarly, much of the recent explosion may only be
because they haven't existed long enough to have failed.

------
wickerman
I'm a data engineer, stopped trying to "upgrade" into ML years ago when I
realised my skills were a lot more in need in the longer term than any ML
specialists.

------
ww520
This is a good overview of the landscape. From what I see, even if there's an
AI bubble burst in funding, a lot of the ML techniques and algorithms can
still be really useful for general computing with a smaller budget. Deep
learning with neural net might be too costly to run for most projects. Shallow
learning algorithms are a lot more cost effective to add to general computing.
Algorithms like Support Vector Machine or Decision Tree learning work
surprisingly well in targeted areas.

~~~
visarga
They are OK if your problem is based on tabular data, but not for images and
text. But the current paradigm is to reuse a pre-trained model, it's less more
data intensive. In some tasks if you have a good backbone model you just need
one or a few training examples.

~~~
IanCal
SVMs absolutely work on text, TF-IDF + SVM is a very classic (and pretty
solid) approach for classification. It's easily explainable and has known
classes of problems.

------
evilelectron
Great write up. Thanks for posting this.

 _There’s rumor that due to a large number of people taking ML courses, there
will be far more people with ML skills than ML jobs._

I hope this is true. There are so many areas where ML skills could be useful.
The sad part would be that some industries would be changed forever. The
animation industry for example might not even exist the way it is today.

~~~
visarga
Don't think it's the case - there will be more desired ML projects than people
capable of implementing them. ML is like electricity 100 years ago or
programming 40 years ago, we haven't applied it yet to most problems of
society.

~~~
alephu5
The problem is it's not as useful as many people seem to think. I often hear
my colleagues suggest ML for anything remotely complicated, even something
like "measuring body fat percentage using electricity" that in reality only
needs a physical equation.

I've even heard people suggest it for web scraping which seems absolutely
crazy to me.

~~~
ieatpies
I could actually see a use case for web scraping. If you're after particular
pieces of content that aren't accessed in a structured way, on a site that
rate limits you to the point of being restrictive, maybe using a bit of NLP
could help you rank links to click.

~~~
hadeson
I tried using OCR to scrap Facebook profiles by simulate web browsing
behavior. It helps a lot in avoid account blocking but still too slow to be
practical.

~~~
esperent
What's your reason for scraping so many people's personal data?

~~~
hadeson
Really just curious about this approach and want to test it since most old
scraping methods failed on Facebook data. My take is that it is possible with
enough resources since it is actually pretty hard to separate this from real
usages.

------
tedk-42
I did a ML/AI course in university in 2011 or 2012. After the course, I had a
much greater appreciation for how clever and complex the decision making
process in our brain is and how difficult it is to get a computer to even do
basic decision making based on a set of inputs. There's human creativity as
well which a machine cannot truely do.

I'm always cautious and a bit pessimistic when I hear companies advertising AI
skills or functionality built into their product. Most of them are probably
driven by people willing to be the next facebook and hype, rather than real
technology.

That same part of me is rejoicing in seeing the industry pay for this hype
train.

I personally think the AI revolution will come from a hardware innovation
(quantum computing or something like that) rather than building these large
datasets to train models. We don't fully understand how our brains work so
trying to replicate this with a computer program is a tough ask.

------
TrackerFF
The problem with investors is that they've been oversold on the "AI" part, and
are expecting magic robots that will replace humans in a few years.

Remember, the key motivation for investing in AI or ML is to become a more
competitive business, by reducing (or eliminating) human labor costs. That's
it.

Luckily, ML has become very incorporated in the modern workforce, but mostly
as part of specific tools that can eliminate easy and tedious tasks - just as
intended.

So yeah, it's sad that the big investor bucks have to come from more
"exciting" projects, because there's a vast landscape of problems that can be
aided my ML, even though they're not as exciting as self-driving cars, robot
doctors, or whatever - and don't require some monstrous deep learning model to
be solved.

------
thundergolfer
> If you have to choose between engineering and ML, choose engineering. It’s
> easier for great engineers to pick up ML knowledge, but it’s a lot harder
> for ML experts to become great engineers. If you become an engineer who
> builds great tools for ML, I’d forever be in your debt.

Is this true, or at least the majority view? Maybe it is because the
comparison is unbalanced, because "great engineer" is on both sides but the ML
part is "pick up ML knowledge" and "ML experts".

I'd choose engineering because the employment opportunities are so much more
favourable, not because it's easy to 'pick up ML knowledge'. I'm in the
beginning of many hours of mathematics and statistics study so that I can
become competent at ML, and in comparison to say, becoming good at git and
python and AWS, this feels harder.

~~~
cdavid
I agree 100 %, and I started as a PhD in ML 15 years ago.

DL, thanks to its "lego aspect", is getting ML commoditized fast for most
applications out there. It hollows the gap between applying more or less
directly existing models and pure research. There is a glut of engineers who
think ML engineering is playing with TF or pytorch all day, retraining models
on well defined datasets.

The reality: you want to use computer vision to do something cool server side
? Just using resnet/etc. and fine tuning on your data, maybe using as an
embedded space, will get you 90 & there.

What's hard ? Identifying opportunities, and especially data. I know it is
cliché, but identifying what kind of problems you can solve w/ the data you
have, and being able to convince the business side that what you do is useful,
all of that is very hard. Same for the ability to solve 'big problems' while
delivering regular deliverable so that you're not seen as a cost center. How
do you link model improvements to business impact ?

The system side of it is still immature as well.

I say that to all my reports, directs or indirects: hone your SWE skills, make
sure you understand how you bring value to your company. Unless you're in the
top 1 %, you're not going to survive with ML skills alone.

------
hivacruz
Thanks for the list. In my case, when I started doing ML/DL projects, I was
looking for many many tools that could help me organize/train/deploy my model.
I remembered looking at Kedro for days which looks great, or MetaFlow from
Netflix.

But at the end, I did my own project's structure with a enough Docker monolith
that includes MLFlow as well. This is the only tool I use (with sklearn) for
something else that training and serving my model.

I think it's nice to do all of these by yourself at first and later use some
of the tools listed there to enhance our workflow. It was really hard for me
to get started with Kedro when your project is not as simple as MNIST or
CIFAR.

------
timClicks
When is "tooling" worth buying? Perhaps it's a reflection of the kind of
person that I am, but I would always rather build my pipeline internally
rather than outsource it. That pipeline will probably become part of the
critical path.

~~~
knodi123
> Perhaps it's a reflection of the kind of person that I am

Probably.

Lots of companies just want the shortest path from here to profitability. If
the cost/benefit equations justify it, then eventually they can justify the
outlay of building their own tooling to replace outsourced pieces a bit at a
time.

I'm thinking of things like AWS Lambda. Once, I built a slim version that was
enough to support my own company's use case. I was told "that's good. save the
git repo. one day it may be worth it to add the redundancy and scaling we
already get from aws. until then, there's more profitable things for you to be
working on."

~~~
anonytrary
> until then, there's more profitable things for you to be working on.

Sorry, but this gave me a slight chuckle. I think I can empathize I guess, but
found your wording hilariously blunt.

------
guildmaster
I've have been working on software engineering, web development primarily over
the last few years. It's always felt that use of AI in industry was hyped up
and soon the hype wind down. Finally there would be done very specific domains
were AI can do exceptionally well.

But recently I've been looking into the works of OpenAI and I can say that all
my assumptions were out right wrong. The rate of growth in AI is increasing
definitely. Just consider the voice tech landscape, with the new generative
models around speech I can totally foresee the numbers of industries that are
going to be disrupted from the ground up.

------
spicyramen
A good read for beginners. I believe the literature about ML state seems to be
a little dated with current times. People and companies are at the phase of
already implementing ML in their companies, versus figuring out things: How?
The ML field gives you options for every user and different levels of
expertise. Advanced user: TF or Pytorch, preprocessing: Beam or Spark.
Accelerators: GPUs. Tools such as Bigquery ML and AutoML have helped users to
get into ML by generating MVP in an easy way. In would love to hear more from
people in the field.

------
jlbnjmn
What a great write up. The high level survey of the landscape was insightful,
and directly comparing the challenges of ML/AI with their "solved"
counterparts was great.

Excellent read. Bookmarked.

------
svaha1728
`The pretrained large BERT model has 340M parameters and is 1.35GB. Even if it
can fit on a consumer device (e.g. your phone), the time it takes for BERT to
run inference on a new sample makes it useless for many real world
applications. `

This is true, but a fine-tuned Distilbert model will give close to that
accuracy with a ~260MB file size.

------
darepublic
I don't think AI is overhyped, though certainly there are many bubble
companies within it. I believe like the value and applications for cutting
edge models is still being discovered, and I would only feel relief if less
people piled on the bandwagon, out of a selfish impulse; more for me.

------
jean-malo
Interesting article! However as a software dev working in this field it is
missing quite a few tools.

The first that comes to mind is Dataiku which is already a pretty big startup
(~$150M raised) so I'm surprised the author missed it.

------
baxtr
I am kinda irritated that this analysis was not done using ML...

------
awinter-py
I'd consume this style of outline for almost any topic

Regardless of my interest in it

So useful

------
SomeoneFromCA
The most important AI company nowadays is almighty Nvidia.

