
Michelangelo: Uber’s Machine Learning Platform - holografix
https://eng.uber.com/michelangelo/
======
atonse
This reminds me of LivingSocial. For years, LivingSocial was like a mini black
hole of really good startup-like talent in the DC area.

But when you have that many smart people under one roof, you have to keep them
busy. So you get these kinds of reinventing-a-beautiful-wheel projects. Not
knocking the quality, just knocking the growing distraction and temptation to
build in-house rather than use existing tech.

LivingSocial had such good quality output, but they were all over the place.
They had to keep all those productive people busy after all. And the best way
was to keep churning out new products.

~~~
hitekker
I agree. "Keeping them busy" also raises an uncomfortable question: if Uber
and other valuable Unicorns can be executed 99.999% with 25% of the engineers
they hire, do they need the next 75% of engineers?

Even considering crunch-time and potential diversification/expansion, full-
time employment of non-critical engineers is obviously inefficient. And, as
Uber proved, politically toxic and consequently harmful to the productivity of
the business.

More pressing is the existential tension underlying this quandary. That our
economy will generally and increasingly forgo the skilled and unskilled labor
provided by our society.

Framed concretely: one day, we may define a "10x engineer" as an engineer who
can send ten other engineers to the soup kitchen.

I'm still looking for a well-thought-out solution.

~~~
xixi77
This is not really new; just think of all the stuff that came out of AT&T
labs...

------
andrewfong
Are there any ML-as-a-service platforms that offers something comparable to
what Uber has here? And importantly, do you actually like using them?

That is, suppose I'm willing to do the following:

* Send said service a firehose of data

* Write code for feature selection, data cleanup, transformations, etc. -- i.e. the sort of code you'd expect to find in a data scientist's Jupyter notebook.

* Specify the type of ML model and parameters I'd like to experiment with.

Is there a service that handles data warehousing, serving up predictions via a
REST API, deployment of new models, reporting model performance over time, and
general scaling issues? I know it's possible to set up a large portion of this
using AWS services, but ideally, it'd be nice to have something with a Heroku-
like ease-of-use, but for ML.

~~~
rasmi
Disclosure: I work for Google. Google Cloud does this with Dataflow and ML
Engine in a fairly simple way, and the internals are open sourced as Apache
Beam and TensorFlow so you can migrate to other platforms if you want.

------
philliphaydon
I wonder if this thing is used for Uber Eats in Singapore... Because I use
User Eats all the time, the estimated time is MASSIVELY inaccurate. It's not
even within 30% to be close to actual time.

The estimated time is always less than the actual time, but every minutes the
estimated time re-adjusts. By the time it's actually delivered it's about 15
to 20 minutes more than the original estimate.

Note: I'm just complaining because the article uses User Eats as an example
and as a service I use often I wonder if this ML Platform is used at all...

~~~
strin
Maybe they trained on global data and it biases against sample portion of the
dataset that represents Singapore?

~~~
philliphaydon
Wouldn't the ML learn to that times differ globally tho?

~~~
d4rti
Only with the right models and features.

------
nl
This solves a real problem. ML pipelines are complicated, and often involve
multiple languages and platforms (just as described in this post).

But.. there are existing solutions in this area. Notable Luigi[1] (from
Spotify) and Airflow[2] (from AirBNB) both seem to have a lot of overlap with
this.

I'm most familiar with Luigi, and it does many of the things that are listed
here.

Some of the model visualisations look pretty nice, and don't come out-of-the-
box in the other platforms.

So I'm not really sure what makes this unique.

[1] [https://github.com/spotify/luigi](https://github.com/spotify/luigi)

[2]
[https://airflow.incubator.apache.org/](https://airflow.incubator.apache.org/)

~~~
gipp
What? Luigi and Airflow are ETL frameworks. They express dependencies between
tasks, schedule their execution, and that's about it. Model storage and
versioning, feature definition and extraction (batch and real-time), model
deployment... Luigi and Airflow do almost nothing described in this post.

~~~
nl
They do model/feature storage and versioning just fine. Or at least Luigi does
- I use it daily.

And feature extraction and modelling is my main usecase. It works really well,
across Spark, Scikit and R, saving data in HDFS.

Some examples:

[https://blog.dominodatalab.com/luigi-pipelines-
domino/](https://blog.dominodatalab.com/luigi-pipelines-domino/)

[http://blog.richardweiss.org/2016/10/13/kaggle-with-
luigi.ht...](http://blog.richardweiss.org/2016/10/13/kaggle-with-luigi.html)

[https://github.com/Atreya22/luigi_rosmann_sales](https://github.com/Atreya22/luigi_rosmann_sales)

~~~
gipp
Yeah, it does all those things, as long as you write all the code, set up all
the systems for it to interact with, and manage all the logic yourself. That's
like saying Python does all those things.

~~~
nl
It's not exactly clear that Michelangelo does anything different?

A quote:

 _We provide containers and scheduling to run regular jobs to compute features
which can be made private to a project or published to the Feature Store (see
below) and shared across teams, while batch jobs run on a schedule or a
trigger and are integrated with data quality monitoring tools to quickly
detect regressions in the pipeline–either due to local or upstream code or
data issues._

That certainly sounds like you write code to run inside the containers. The
deep integration with the standardized feature store etc sounds nice, but not
radically different.

 _we created a DSL (domain specific language) that modelers use to select,
transform, and combine the features that are sent to the model at training and
prediction times. The DSL is implemented as sub-set of Scala._

So this is pretty much the equivalent of Spark Dataset API, or maybe the
RFormula stuff in Spark ML[1] except in Scala right?

[1] [https://spark.apache.org/docs/latest/ml-
features.html#rformu...](https://spark.apache.org/docs/latest/ml-
features.html#rformula)

------
sandGorgon
> _For every model that is trained in Michelangelo, we store a versioned
> object in our model repository in Cassandra.._

> _For important model types, we provide sophisticated visualization tools to
> help modelers understand why a model behaves as it does, as well as to help
> debug it if necessary_

How is this done - what serialisation format (pickle, pmml,etc) is being used?
And more importantly, how are you tracing into the model ? It looks like this
is a Spark based framework

~~~
Eridrus
It's probably being "serialized" as a JAR file which just contains the code
you call into. That was Spark's online deployment story last I heard, and
jives with them saying the library mode has a Java API.

------
strictnein
SAP has Leonardo. Uber has Michelangelo. Who has Donatello and Raphael?

~~~
oculusthrift
Personally i don't get the obsession with ninja turtles.... /s

~~~
omot
Funny you thought of Ninja Turtles, I was thinking Renaissance artists.

~~~
strictnein
Then this Epic Rap Battle of History is for you (salty language warning):

[https://www.youtube.com/watch?v=6HZ5V9rT96M](https://www.youtube.com/watch?v=6HZ5V9rT96M)

------
jackyinger
I expect this will be an unpopular opinion, but a proliferation of frameworks
does not create value for anyone besides the developers employed in making it.

Sure different frameworks handle different uses better than others, but it
doesn't change the fact that in a given genre (machine learning, web frontend,
ORM, etc.) that there is a lot of duplication going on.

If you want to look cool and impress people, make a tool that doesn't exist,
develop a new algorithm, or contribute a great feature to an existing
framework.

~~~
lwansbrough
That's how I feel about most projects I see on here. Toy languages are my
personal pet peeve. So much talent being wasted on useless projects. I'm not
really one to speak - none of my projects make a substantially positive impact
on society, but they have their niches.

~~~
gfodor
Do you really feel this way? It seems like a pretty strange world view to
think that someone posting a thing they built on a site called hacker news
implores you to judge their talent as being wasted, or their project as being
useless.

Not only does it seem wrong to judge things people post here that way, anyone
out there doing good work knows that the path to doing meaningful work begins
with "useless projects."

~~~
lwansbrough
The reality for the people who post on here is that most of us aren't doing
anything truly substantial. I include myself in that group. You can believe
your IOT AI messenger bot is going to revolutionize an industry, but the more
likely situation is that it doesn't. That being said, such a project is not
something I'd consider useless, but it's in the same range as the things I do
consider useless. Take toy languages for example: could it revolutionize
software development? Maybe. Will it? Almost certainly not. How many languages
are commonly used in software engineering, and how much ability have you
gained by doing it?

It's not for me to say where you apply your time and from what you get your
enjoyment from, but I do believe there are objectively more important problems
that could be solved. There are plenty of problems out there that need to be
solved, and engineers here have the capability to solve them -- I think it's
fair to say we're more well equipped than most to solve some of the largest
problems humans face.

Here's a quick list of problems that are worth solving (in my opinion), vs.
things that are figuratively _useless_ :

\- Rising global temperatures will disrupt agriculture and food supplies, how
will we farm when desertification consumes the croplands?

\- Thousands of people walk by homeless people on the street every day, if a
small portion of those people stopped and engaged with the homeless, how many
lives could be improved?

\- Children in third world countries struggle to receive proper education, how
can we reach them and improve their education?

\- Governments around the world are failing to represent the needs and desires
of their people, what can be done to help governments succeed?

\- Growing levels of automation are replacing a staggering number of jobs, and
if the growth continues, the _great_ depression will be ahead of us, not
behind us.

\- Global financial markets are controlled by companies which you and I have
no say in, yet they hold most of the worlds wealth.

In contrast:

\- I want X programming language that does Y because Z doesn't do Y, in all
likelihood, I will be the only one who ever uses this language, but it'll be a
personal accomplishment.

\- I don't want to leave the house or stop working for half an hour to make a
meal, how can I improve/speed up the time it takes for me to get a meal from
my favourite restaurant?

\- Making X sucks, I want a robot to do it

\- Bad AI for X

\- IOT for X

\- Framework for X

\- Uber for X

How often do I see the latter vs. the former? Imagine if you saw FOSS projects
attempting to solve problems on the first list as often as you saw projects
attempting to solve problems on the second list. I truly believe the world
_would_ be a better place if that was the case.

I said earlier I'm guilty of this myself. I build consumer facing supporting
software for entertainment media. I like what I do, but I don't think I'm
contributing much to society, and that makes me uneasy.

~~~
bko
> Take toy languages for example: could it revolutionize software development?
> Maybe. Will it? Almost certainly not. How many languages are commonly used
> in software engineering, and how much ability have you gained by doing it?

Maybe revolutionizing the world is not the point. Maybe the people who create
toy languages gain a deeper understanding of some issue from pursuing the
project. Maybe sharing it helps others discover that deeper issue or hidden
complexities and elegant solutions. Or maybe they are just an inspiration and
serve as a reminder that everything that exists is built by humans. Everything
can be understood. There is no magic.

------
visarga
It would be more useful to publish anonymized datasets, if they got any useful
data.

------
strin
A random idea: in this article, they talked about a Feature Store shared
across teams. Feature extraction also seems a major bottleneck in their
modeling. It seems a very natural and valuable thing to have a platform to
share "generic features".

Yann LeCun said once a deep neural network predicts many targets, say 1000
classes in ImageNet, it is possible for the model to learn quite generic
features. So it makes sense to pre-train on a large amount of data and a
reasonable number of targets, and then share the learned feature extractors
with others.

Could this be a business? Or a community? Thoughts?

~~~
Eridrus
Pretrained models for generic things exist publicly, e.g. models trained on
ImageNet, or pretrained word vectors.

Not a whole lot of general use in having a feature extractor tuned to Uber's
data.

------
aorloff
I am surprised by this - h2o.ai seems to have many if not all of the features
of this, and is open source.

"Specifically, there were no systems in place to build reliable, uniform, and
reproducible pipelines for creating and managing training and prediction data
at scale. Prior to Michelangelo, it was not possible to train models larger
than what would fit on data scientists’ desktop machines, and there was
neither a standard place to store the results of training experiments nor an
easy way to compare one experiment to another."

Seems like h2o.ai fits a lot of that bill.

~~~
JPKab
Something else that throws me off on this:

My team's experience with MLlib has been bad. Especially compared to H2O
Sparkling Water.

Anyone else on here find MLlib to not be as good as advertised? I was
surprised to see Uber using it.

~~~
thecity2
Spark is good if it fits your use case (or if you can make it fit your use
case). We have definitely found that scaling can be a real issue though.

------
linkregister
The "subscribe for updates" modal is extremely distracting. It has a long lag
before it repositions at the top of the viewport, at least in my browser.

------
ahead
Is Michelangelo the name of their previously published paper "Scaling Machine
Learning as a Service"
[https://news.ycombinator.com/item?id=14708761](https://news.ycombinator.com/item?id=14708761)
, or is this something separate?

~~~
rockinghigh
Yes, it's the same platform.

------
rodionos
> batch jobs run on a schedule or a trigger and are integrated with data
> quality monitoring tools to quickly detect regressions in the
> pipeline–either due to local or upstream code or data issues

I'm curious what tooling they're using to ensure data quality, in particular
time series data.

------
manugarri
So I have seen this Michelangelo pop up in other articles.

The key question is? Is Uber going to open source it? If not, why bother
writing articles about the specifics of their platform?

~~~
sunsetMurk
PR & attracting talent? Though open sourcing would be great and do the same
thing, but more so.

------
thecity2
This doesn't seem to be an open source project, unless I'm missing something.
It's kind of a show-and-tell basically.

------
tomerweller
Describing your ML platform is nice. Open sourcing it would have been nicer.

~~~
jamesmishra
Former Uber engineer here.

I don't think you would find an incredible amount of use from an open-sourced
Michelangelo. The biggest advantage that Michelangelo has for Uber is that it
is easy to integrate into all of Uber's other tools.

Depending on what your machine learning needs are, you could get pretty far
with just Spark + MLLib, and wouldn't need any of the customization that
Michelangelo has on top.

~~~
gregoriol
Why make all that noise with a detailed blog post then? If it's a custom-fit
internal tool, then good for you, the rest of the world doesn't care. Each
company has internal tools and stuff.

~~~
ClashTheBunny
There is the sharing of ideas. Maybe they couldn't open source it, but were
given permission to publish about it. Google never opensourced some of their
greatest contributions, just the ideas behind them.

------
cyrux004
TLC anyone ?

~~~
curiousDog
What is TLC?

~~~
Larrikin
An autocorrect of tldr

~~~
_pmf_
I prefer to think he likes a hug.

------
pyrophane
So, what? Uber is now the new IBM Watson consultancy?

~~~
jamesmishra
Former Uber engineer here.

I don't think Uber is anything like IBM Watson, mostly because Uber uses
machine learning to solve business problems for its own products--not for
other companies' products.

Most large companies that use machine learning in production will have
something similar to Uber's Michelangelo. For example, Facebook has FBLearner
Flow[1]. Catherine Dong recently wrote a TechCrunch article describing this
broader industry trend[2].

[1]:
[https://code.facebook.com/posts/1072626246134461/introducing...](https://code.facebook.com/posts/1072626246134461/introducing-
fblearner-flow-facebook-s-ai-backbone/)

[2]: [https://techcrunch.com/2017/08/08/the-evolution-of-
machine-l...](https://techcrunch.com/2017/08/08/the-evolution-of-machine-
learning/)

