Opening this on desktop.. I'm not reading any of this.
Glancing in more I see react-native mentioned, and expo+expo go mentioned, but like... it's been proven time and time again that Hacker News isn't your ideal feedback loop (ie: Dropbox, AirBNB).
Why make 4 paragraphs of text when you could just say:
- We obliterated X problem: (link)
- We demonstrate why that problem is significant here: (public link or private deck)
- We are growing at X->(timeframe or whatever impressive metric)
Reach out if interested. Thanks. [contact@info.com]
Thanks a lot for the feedback. This is a first pass implementation, but I agree that more thought should be put into the cutoff threshold, specifically for when there are only initially 2 (or maybe 3) variations.
I would suggest that you put together a quick monte-carlo simulation for any of the models you're experimenting with to see how well they perform when you actually know the true conversion rates. There's plenty of theoretical issues you can find with any method and the more complex what you're doing is the harder it can be to work it all out with pencil and paper. Likewise, because you're dealing with probabilistic solutions, real-world results can be deceptive (for example conversion rates may naturally fluctuation between weeks or months). I've found that testing with simulations is the best way to get a real sense of how whatever method you wish to employ will work.
I would ignore any non-baysian MAB posts out there. The formulation used by other approaches is one that considers an infinite number of repeated trials, which is basically an insane assumption. Epsilon greedy and UCB1 aren't optimal except with that assumption.
Except in that example the author is choosing E based on the observed difference between two means (which is the exactly the unknown you're trying to determine, so it makes no sense to use it as a constant in a formula), rather than the threshold for the minimum distance you care about.
If you're going the classical statistics route the entire point is that you need to determine your sample size before you peek at the data. In that post you would need to replace E with a threshold of difference that you care about, then calculate n before you start the test and not look at the results until you had reached n observations.
However with the Multi-armed bandit approach it does make sense to re-consider that since it's a continuous optimization problem and the overall average conversion rate would be higher.
That visualwebsiteoptimzer page is mathematically confused and the simulation is fatally flawed, I wish it'd disappear from the internet. Check the comments.
Interacting with the map should hide the top and make the only thing scrollable in the viewport the map (with the "Your Alias" bar fixed at the top). Scrolling down causes you to zoom out when your mouse is over the map, which is a really bad experience.
When you scroll down, maybe the "Your Alias" bar has the BitBar logo on the left, and a metro-picker dropdown on the right. This would make the usability a lot cleaner, because it's hard to zoom into a city if you accidentally click on a bar, which slides up the Drink Picker.
There should also be a "Cancel" on the Drink Picker, that doesn't require you to click on the map, because clicking on the map puts the body scrollTop to 0.
The map, business logo, and people icons just help illustrate the feature. Volume and other value related benefits are communicated elsewhere in the page content, but aren't really additive to the blog post.
I've just tried hackerfollow.com and found a bug with self posts: After entering a login phrase I see all of PG's comments and posts. Many of the posts are discussions on HN (eg http://news.ycombinator.com/item?id=1861577), which gets linked to http://hackerfollow.com/comments/1861577, which in turn shows an "500 Internal Server Error".
Glancing in more I see react-native mentioned, and expo+expo go mentioned, but like... it's been proven time and time again that Hacker News isn't your ideal feedback loop (ie: Dropbox, AirBNB).
Why make 4 paragraphs of text when you could just say:
- We obliterated X problem: (link)
- We demonstrate why that problem is significant here: (public link or private deck)
- We are growing at X->(timeframe or whatever impressive metric)
Reach out if interested. Thanks. [contact@info.com]