
Learning to Simulate - natanielruiz
https://towardsdatascience.com/learning-to-simulate-c53d8b393a56
======
lalaland1125
Interesting, but there is one major problem with this work in that they are
probably using RL when it's not necessary.

I highly suspect they could get quite good results simply with random search
as the parameter space is not that complicated.

They appear to report random search results in table 1, but I highly suspect
those numbers are incorrect because they report lower performance using random
search compared to random parameters, which doesn't make much sense. I see how
that could happen in extreme cases with severe overfitting, but that doesn't
seem that likely.

~~~
natanielruiz
Hi there! I'm one of the authors of the work. I can assure you the numbers are
not incorrect: random search did not achieve good results in the semantic
segmentation experiment. We had similar skepticism with reviewers at ICLR
initially, and our new experiments convinced them.

Random search is a good alternative and it did produce good results for the
car counting experiment. We do not claim that RL is the best way to solve the
problem at any point in our work. We just observe that random search did not
perform well in the segmentation experiment while RL did perform well in all
of our experiments. We think the RL formulation is a good one since it is
flexible (can be easily adapted to neural network policies with thousands of
weights for example).

Hope this helped!

~~~
lalaland1125
Hi Nataniel,

Thanks for the response and paper. My main question here is that it doesn't
seem to make sense that random search did significantly worse than random
parameters? It seems that random search should only consistently lose in cases
when the dev set performance is anti-correlated with the test set performance.
Why do you think you saw random parameters beating random search? (Is there a
typo in the table or something)?

~~~
natanielruiz
No problem! I'm here to help clarify any doubts.

I believe in this specific case, there was a local optimum that random search
(with the parameters we selected, which we actually tuned as well) was not
able to escape.

In most cases I think you would find random search doing better than random
parameters (unless we have the paradoxical situation you described), so your
intuition is correct in the general case. But in this case there is no typo on
the table!

Does this answer your question?

~~~
lalaland1125
How could you have a local optimum with random search? It's not doing any hill
climbing or anything so it shouldn't care at all about local optima. Yes, of
course you could get bad results if you specify the random search
distributions with too strong of a prior pointing in the wrong direction, but
we usually specify a weak prior (uniform/etc distributions) for doing random
search. What random distribution did you use for your random search?

Anyways, if you had a bad random search prior, then that should have equally
negatively impacted your random parameters as both should have been drawn from
the same distribution. If you used a different random distribution for random
search vs your "random parameters", why did you use a different distribution
and why were they so different?

~~~
natanielruiz
Maybe we're talking about different algorithms. Do you agree that we are
talking about random search as outlined in this Wikipedia page
([https://en.wikipedia.org/wiki/Random_search](https://en.wikipedia.org/wiki/Random_search))?

~~~
lalaland1125
Ah, that makes sense. By random search, I thought you were referring to random
search as defined in
[http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a....](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).
I believe that's the more common method for hyperparameter optimization cases
like this
([https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf),
[https://scikit-
learn.org/stable/modules/generated/sklearn.mo...](https://scikit-
learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)).

It has much better guarantees than the method you used and tends to work quite
well in practice. It's also probably what the reviewers meant by a random
search baseline.

~~~
natanielruiz
Our "random parameters" baseline is actually the random search that you refer
to, while the "random search" that we use, is the actual derivative-free
optimization technique of random search.

Reviewers asked for a comparison with this specific random search. You can go
look at the reviews on OpenReview.

------
amoreno32
Awesome!

------
ohduran
Hitting a paywall...can anyone assist?

~~~
natanielruiz
[https://towardsdatascience.com/learning-to-
simulate-c53d8b39...](https://towardsdatascience.com/learning-to-
simulate-c53d8b393a56?source=friends_link&sk=b52f70a73f431ae384ec2404e5548c3e)

------
MITpmt
Great!

