That sounds like an odd way to do it - do your customers typically purchase on the same day they first hit your web site? If not, they likely came back several times and might have been bemused or even frustrated to find the phone number appearing and then disappearing. Or just seeing the phone number one of those days was enough to accrue all the benefit of having the phone number on any day.
Good point. A better way would be to randomly sort visitors to the site into either the A or B group, and make that sorting sticky over time (cookies aren't perfect, but are a reasonable approximation). No magically disappearing phone number!
This is a good point. I assume the testing measured how many sales were made during which days. A more thorough test would be 1 month intervals for say 6 months (even better would be for them to figure out how long the sales cycle typically takes from the 1st visit, double it and add a day, then alternate for that period).
I presume he means he puts all users who arrive on an even day into the control and all users who arrive on an odd day into the "show a phone number" bucket.
You then use cookies to track which user is in what bucket, because you have to know which treatment the user was in when they converted. As you said, that may or may not be on the same day they were treated.
That's still kind of weird, though. You should just randomly assign each user as they arrive to one group or the other.
If he's doing what you're suggesting that's bad methodology for all the reasons you said, and I wouldn't trust the results.
That sounds like an odd way to do it - do your customers typically purchase on the same day they first hit your web site? If not, they likely came back several times and might have been bemused or even frustrated to find the phone number appearing and then disappearing. Or just seeing the phone number one of those days was enough to accrue all the benefit of having the phone number on any day.