I'm a little unconvinced based only on the data/graph shown.
It appears that the test "Boosting All" got out to an early lead (sometimes statistically invalid if there are "forcing links" that the dev team is generating and I've even seen teams use those links to order/convert, without thinking through the implications of those actions on the data purity [they're honestly just testing to make sure the changes don't break the site, and naturally need to test all the way through; I don't think it's done to intentionally bias results])
I'd want to see the graph just from Jan 21-26, or to see the day-wise, rather than cumulative, graph/data. It looks to me like it's not winning anywhere near 15% on the right 2/3rds of the graph.
Doesn't effect this test at all. But I think the most common reason I will hit checkout on something I don't intend on buying at the time is because shipping cost or payment methods aren't listed elsewhere on a site.
It would be interesting to now do another A/B with just the extra email info, looking at the 2 examples the lack of info about why you want the email is the major thing that would put me off.
That's the first thing I thought of. I understand the urgency to try out new things, but testing each individual change would have been more interesting as data. Perhaps the email change was responsible for the bulk of the improvement. Perhaps the visa and security links were - it's going to be hard to say.
The variation does in fact tells why they are asking for email info. It says: "we will send the product download link to that email id". I am guess other eCommerce stores if they want to collect email ids can simply tell the users that they will send receipt at this email id.
It appears that the test "Boosting All" got out to an early lead (sometimes statistically invalid if there are "forcing links" that the dev team is generating and I've even seen teams use those links to order/convert, without thinking through the implications of those actions on the data purity [they're honestly just testing to make sure the changes don't break the site, and naturally need to test all the way through; I don't think it's done to intentionally bias results])
I'd want to see the graph just from Jan 21-26, or to see the day-wise, rather than cumulative, graph/data. It looks to me like it's not winning anywhere near 15% on the right 2/3rds of the graph.