
Using Model Tuning to Beat Vegas - Zephyr314
http://blog.sigopt.com/post/136340340198/sigopt-for-ml-using-model-tuning-to-beat-vegas
======
mswen
Zephyer314 hello, interesting read. It is interesting to me that the lowest
cumulative level for the tuned model is approximately equal to the lowest
cumulative level for the simple model with no tuning. In fact if we had been
only looking at the results up until Dec. 12 or so we would have concluded
that the simpler model works better and barely beats the house and that maybe
the performance of the simple and tuned models are converging.

However, they now seem to be diverging with a clear advantage to the tuned
model. If you had been actually using this to bet you might have given up on
the tuned model around Dec. 7th when draw down was at its worst.

What will really be interesting is whether the performance over the rest of
the season continues to diverge in the current direction or if the performance
tends float up and down around the break even line.

Is one of the parameters a progress through the season index? Anyway thanks
for sharing the example.

~~~
Zephyr314
Thanks! Great question.

There is a huge variance in the cumulative gains as you pointed out. Because
of the 100 win/110 loss edge that Vegas has you need to win over 52.5% of the
time to register a profit. Even with a model that can win 52.5% + e of the
time you are effectively flipping a (biased) coin. Over the 131 games in the
holdout dataset this random walk will tend upward, but can vary widely
(especially at the start). As the season continues on this should start to
wash out (in the cumulative profit). We plan to write a followup post at the
end of the season to see how it performs. You could also fork the code [1] and
train/tune on '00-'13/'13-'14 and use the entire '14-'15 set as the holdout
(or a similar combination).

The features can be found here [2], we don't use individual player stats or
temporal stats, but it should be relatively easy to add if you wanted to try
it out.

[1]: [https://github.com/sigopt/sigopt-
examples](https://github.com/sigopt/sigopt-examples)

[2]: [https://github.com/sigopt/sigopt-
examples/blob/master/sigopt...](https://github.com/sigopt/sigopt-
examples/blob/master/sigopt-beats-vegas/predictor/features.py)

~~~
mswen
Thanks for the reply. I don't know if I have sufficient interest to actually
grab the code and try it out. However, I will try to remember to look for your
update post at the end of the season.

~~~
Zephyr314
If you decide to grab the code and have any questions please feel free to
reach out.

------
Zephyr314
Hello, I'm the author of this post and co-founder of SigOpt (YC W15). Let me
know if you have any questions about the post or what we do at SigOpt.

All of the code used in this post can be found at
[https://github.com/sigopt/sigopt-examples](https://github.com/sigopt/sigopt-
examples)

------
Oxydepth
Very in depth. Good read.

~~~
Zephyr314
Thanks! Let me know if you have any questions.

