I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.
As for going with the winner: Yes, if the test result (39/28) is the only information you have and there are only two choices (go with winner / go with loser) then it makes sense to go with the winner.
I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.
So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2?
I'm open to correction here.