1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption.
Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).
More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.
2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.
Facebook is known to run countless online experiments – most of which are kept secret. Luckily for us we have the Internet Archive that allows us to pry into these possible design optimizations.
Sure. But the idea is to do a quick ice breaker project that can later turn into a larger one down the road. Hence in that case a $500 - $2000 structured project makes sense, I think.
I used to work in the printing industry. I thought that if I got a small order, like a set of business cards, it would be a great way to break the ice. It almost never worked.
Businesses are in business to make money. If you can provide value (I'm not saying you can't) and they see an ROI on their money, they'll pay. They want a solution to their problem AND they want to work with someone they like and trust.
Depends on the service. Sometimes the service explicitly is my time. Tutoring, for example, or job candidate interviews. I didn't mention this in my first post, but I think I would emphasize that you're explicitly buying time with me, i.e. we'd be on Skype together, working through something. Pair programming for hire!
The question I recently have asked myself was this: on these structured or productized pages, should we push the customer to a low friction gradual engagement lead generation action, or close the sale with a payment? I'm now trying out the former. Thoughts?
Which option has the most valuable result to the customer? Regardless of payment, I think delivering something that is going to really blow them away upfront is going to be the best thing... And you have alot of amazing work to do that :)
Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).
More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.
2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.