
Google Vizier: A Service for Black-Box Optimization - captn3m0
https://ai.google/research/pubs/pub46180
======
kevlar1818
Okay, this is awesome (and easy to miss with a just a cursory skim):

"5.3 Delicious Chocolate Chip Cookies

Vizier is also used to solve complex black–box optimization problems arising
from physical design or logistical problems. Here we present an example that
highlights some additional capabilities of the system: finding the most
delicious chocolate chip cookie recipe from a parameterized space of
recipes... We provided recipes to contractors responsible for providing
desserts for Google employees. The head chefs among the contractors were given
discretion to alter parameters if (and only if) they strongly believed it to
be necessary, but would carefully note what alterations were made. The cookies
were baked, and distributed to the cafes for taste–testing. Cafe goers tasted
the cookies and provided feedback via a survey. Survey results were aggregated
and the results were sent back to Vizier. The “machine learning cookies” were
provided about twice a week over several weeks...

The cookies improved significantly over time; later rounds were extremely
well-rated and, in the authors’ opinions, delicious."

~~~
meinstream
I worked for Google at the time they went through the cookie iterations in the
cafes. Personally I disliked all the cookies that ever came out of this
project, mainly because they contained spices, I heard the same thing from
many others.

After you got handed a cookie you were asked to fill out a feedback form on a
tablet, which almost always is annoying.

But more importantly, the process of optimizing the ingredients left out many
parts of what makes food enjoyable, temperature and texture for example.

I know this was meant to be a fun showcase of ML but to me this is still my
favourite example of explaining misuse of ML technology where a simpler
statistical model supported by expert opinion would have outperformed the
taste of the cookies. Eg, any cookie expert might know that the number of
people who like spicy cookies is only a small subset of all cookie lovers.

~~~
oh-kumudo
I think this is exactly the case how ML could go wrong. I would assume the
people who hate the cookies probably will not bother to submit a survey and
try again. The sample might be biased and not representative of the general
population, thus overfitting, performs badly on a boarder test set.

~~~
whazor
Our dystopian future where we will be forced fed cookies in order for accurate
optimization.

~~~
archgoon
There was an icecream advertisment with that as a theme.

[https://www.youtube.com/watch?v=j4IFNKYmLa8](https://www.youtube.com/watch?v=j4IFNKYmLa8)

------
btown
Whetlab, a startup founded by some former colleagues of mine, provided a
service just like this 4 years ago (in fact, it's referenced in the Vizier
paper as its open-source variant Spearmint), but unfortunately, it was
acquired and shut down by Twitter:
[https://venturebeat.com/2015/06/17/twitter-acquires-
machine-...](https://venturebeat.com/2015/06/17/twitter-acquires-machine-
learning-startup-whetlab-will-kill-the-service-on-july-15/)

Black-box optimization is a hugely important problem to solve, especially when
experiments require real wet-work (i.e. medicine, chemistry, etc.). Kudos to
Google for commercializing this - I expect it will see a lot of use in those
fields. But it's bittersweet to know that it's taken this long for this type
of application to be promoted like this.

~~~
indiesolver
Whetlab was the first startup in this domain. Then, followed by SigOpt
([https://sigopt.com/](https://sigopt.com/)) which is doing a great job at
popularizing the concept.

~~~
levesque
Yeah SigOpt clearly has taken the space left by Whetlab. They must not be too
pleased about this announcement, I mean generally the worst thing that can
happen to your startup is a tech giant launching in your space. Then again the
problem of hyperparameter/black-box optimization is so ubiquitous that there
should be enough space for them both.

~~~
btown
I wasn't familiar with SigOpt - this is really cool. I actually think SigOpt
will see this as great publicity telling people about the usefulness of the
concept - and whereas Google won't lift a finger for (insert hedge fund or
pharmaceutical giant here) beyond maintaining uptime, SigOpt can provide those
customers with customized advice about how to integrate, what pitfalls to
watch for, how to design experiments to maximally take advantage of their
technology.

And they're competitive with Spearmint (though not necessarily the closed-
source versions of it used at Whetlab), though Vizier remains to be seen:
[https://arxiv.org/pdf/1603.09441.pdf](https://arxiv.org/pdf/1603.09441.pdf)

~~~
Zephyr314
Thanks! I'm one of the co-founders of SigOpt (YC W15).

You hit the nail on the head. We've been trying to promote more sophisticated
optimization approaches since the company formed 4 years ago and are happy to
see firms like Google, Amazon, IBM, and SAS enter the space. We definitely
feel like the tide of education lifts all boats. Literally everyone doing
advanced modeling (ML, AI, simulation, etc) has this problem and we're happy
to be the enterprise solution to firms around the world like you mentioned. We
provide differentiated support and products from some of these methods via our
hosted ensemble of techniques behind a standardized, robust API.

We're active contributors to the field as well via our peer reviewed research
[1], sponsorships of academic conferences like NIPS, ICML, AISTATS, and free
academic programs [2]. We're super happy to see more people interested in the
field and are excited to see where it goes!

[1]: [https://sigopt.com/research](https://sigopt.com/research)

[2]: [https://sigopt.com/edu](https://sigopt.com/edu)

------
minimaxir
This product is publicly available as a part of the hyperparameter turning in
Cloud ML Engine (although I haven't played with it):
[https://cloud.google.com/ml-
engine/docs/tensorflow/hyperpara...](https://cloud.google.com/ml-
engine/docs/tensorflow/hyperparameter-tuning-overview)

~~~
riku_iki
Why do you think it is vizier, and not something developed specifically for
clous?..

~~~
minimaxir
If you click through the explanation in the documentation:

> Cloud Machine Learning Engine is a managed service that enables you to
> easily build machine learning models that work on any type of data, of any
> size. And one of its most powerful capabilities is HyperTune, which is
> hyperparameter tuning as a service using Google Vizier.

[https://cloud.google.com/blog/products/gcp/hyperparameter-
tu...](https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-cloud-
machine-learning-engine-using-bayesian-optimization)

------
Zephyr314
SigOpt (YC W15) is a related service that performs a superset of these
features as a SaaS API (I am one of the co-founders).

We've been solving this problem for customers around the world for the last 4
years and have extended a lot of the original research I started at Yelp with
MOE [1]. We employ an ensemble of optimization techniques and provide seamless
integration with any pipeline. I'm happy to answer any questions about the
product or technology.

We're completely free for academics [2] and publish research like the above at
ICML, NIPS, AISTATS regularly [3].

[1]: [https://github.com/yelp/MOE](https://github.com/yelp/MOE)

[2]: [https://sigopt.com/edu](https://sigopt.com/edu)

[3]: [https://sigopt.com/research](https://sigopt.com/research)

------
beejiu
It is referenced in the paper, but it's worth pointing out Yelp's MOE. This is
an open source (Apache 2.0) implementation for black-box optimization. It
doesn't look like it is well maintained, but it is interesting nonetheless.
[https://github.com/Yelp/MOE](https://github.com/Yelp/MOE)

~~~
Zephyr314
Hi, I'm one of the co-authors of MOE (it was a fork of my PhD thesis).

We haven't been actively maintaining the open source version of MOE because we
built it into a SaaS service 4 years ago as SigOpt (YC W15). Since then we've
also had some of the authors of BayesOpt and HyperOpt work with us.

Let me know if you'd like to give it a shot. It is also completely free for
academics [1]. If you'd like to see some of the extensions we've made to our
original approaches as we've built out our ensemble of optimization algorithms
check out our research page [2].

[1]: [https://sigopt.com/edu](https://sigopt.com/edu)

[2]: [https://sigopt.com/research](https://sigopt.com/research)

------
sevensor
Only up to 64 variables? Why such small problems? And why are their results
are averaged across all of the different test functions? I'd like to see the
performance difference between Rosenbrock and Rastrigin, thank you very much.
And they have a weird fixation on stopping rules, when in general your
stopping rule is how many evaluations you can afford. Was this written by
interns? There's no discussion of the retrogression by CMAES on small
problems? What parameters does their algorithm have and how sensitive is it to
its parameterization? What parameters did they use for the competing
algorithms? If I were reviewing this paper I'd have these and other pointed
questions for them. It's as if they did a cursory lit review and then decided
to reinvent the entire field. Their one contribution seems to be not allowing
you to run it on your own hardware.

~~~
ScoutOrgo
It is 64 hyper-parameters, not variables, which is a huge amount.

Stopping rules are also a requirement of tuning. Can't just have a while
True:. If an objective function doesn't improve after 100 (or any #), there
needs to be logic to stop the trials since this system is apparently serving
all of Alphabet.

~~~
indiesolver
I think that in the context that he meant, 64 variables = 64 hyperparameters
because it was about benchmark problems solved in black-box settings. In other
words, it is not about the common misconception of comparing #hyperparameters
and #variables on some ML model, where the latter can be in 10^7.

~~~
sevensor
You're correct -- I'm talking about how many variables control the behavior of
the black box.

------
abhgh
I had come across this library - [1] - a while ago that claims to be the open
source equivalent of Vizier. Does anyone know how well justified the claim is?

[1]
[https://github.com/tobegit3hub/advisor](https://github.com/tobegit3hub/advisor)

------
richardliaw
Very related open source project from the RISE Lab at UC Berkeley:
[http://ray.readthedocs.io/en/latest/tune.html](http://ray.readthedocs.io/en/latest/tune.html)

~~~
pinhead
Very cool project! Glad to see support for PBT and HyperBand. Also related to
[https://www.microsoft.com/en-
us/research/publication/hyperdr...](https://www.microsoft.com/en-
us/research/publication/hyperdrive-exploring-hyperparameters-pop-scheduling/)

------
indiesolver
A related service currently in beta:
[https://indiesolver.com/access](https://indiesolver.com/access)

~~~
malkia
This site was forbidden to look at from the company I'm working in?

~~~
indiesolver
Thanks for reporting! The site is hosted on Google Cloud Platform. If some of
GCP's IPs are banned in your company, then it might be the reason.

------
ddutta
Another open source tool for hyperparameter optimization inspired by Vizier
[https://github.com/kubeflow/katib](https://github.com/kubeflow/katib)

------
pm90
Totally calling the next GCP AI Service: VaaS (Vizier as a Service)

~~~
usefulcat
I know a guy who told me 8 years ago about an idea he had for a "black box"
optimization consulting service. You give him a pile of data, he uses ML to
optimize for whatever.

~~~
minimaxir
That's known nowadays as AutoML. Google recently released AutoML products for
image and text classification.
[https://cloud.google.com/automl/](https://cloud.google.com/automl/)

~~~
abhgh
FYI there is also a different "automl" suite of libraries maintained by Frank
Hutters lab at the University of Freiburg - [1]. His group has been leading a
lot of research in the area of hyperparameter tuning.

[1] [https://www.automl.org/](https://www.automl.org/)

