

Stop guessing. Use A/B testing to determine ideal price for your product - paraschopra
http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-price-testing/

======
paraschopra
In 2000, Amazon got caught doing price testing and the news was covered on all
major blogs. Here is a link: <http://news.cnet.com/2100-1017-245631.html>

~~~
mattmanser
What's your source on a/b testing price on identical products being illegal?

I've never heard that and none of the links provided in the comments so far
justify your claim. It is a pretty extreme assertion and you provided no links
justifying it either.

I can see this rapidly turning into an urban myth, whispered wild eyed at 3am
over ramen noodles as founders try and figure out a price for their product.

~~~
ekanes
I don't believe it's legal to charge different prices to different people _for
exactly the same_ product. Changing the price (for everyone) is fine (you
could even change the price 5 times a day), or adding a back scratcher and
bumping the price by $500 is fine, but doing an AB test where you flip the
price back and forth on the same thing isn't legal as far as I know. IANAL.

~~~
roel_v
How would that be so? How do you 'believe' a factual thing like this? I can
charge one guy 10$ for a pie in my bakery shop, and the next guy 100$ because
I don't like his face. There is a narrow set of classes on which it is not
allowed to discriminate but all else is fair game.

I'm not claiming to know this of all jurisdictions, there may be specific
circumstances in which this does not hold, but these are the exceptions.

~~~
ekanes
> There is a narrow set of classes on which it is not allowed to discriminate
> but all else is fair game.

You're thinking about laws around hiring. I can't charge you double to buy a
VCR in my store because I don't like your face.

~~~
roel_v
Why not?

------
michaelleland
"Pricing for physical goods is simple. For example, if you are manufacturing
staplers, all you need to do is to calculate cost of production and
distribution, slam 20% margin on it and there you have the price you can sell
your shiny stapler machines for."

Not true. In fact, a lot of Maker businesses get hung up here. The pricing of
physical goods depends on the market every bit as much as the pricing of
software or web services. In fact, large companies rely on that all the time--
case in point is P&Gs Swiffer Sweeper, that they charge much more than the
manufacturing cost for the sweeper and for the refills.

Many small business owners I've run across actually feel guilty if they charge
"too much" (say 100% markup) for their physical goods they are selling, even
though their competitors sell at that price. The price elasticity of demand
applies just as much for physical goods as it does for non-physical.

~~~
roel_v
To an extent I agree with the sentiment of your post, but the practicalities
are different imo, and "The pricing of physical goods depends on the market
every bit as much as the pricing of software or web services." I don't agree
with at all. Many non-software products and services (staples, pies, chimney
sweeping, plumbing, ...) are much more commodities than software. Efficient
markets, lemon problem etc. drive down the markup to a low (single to low-
double digit) percentage. Since cost is also much less variable (costs in
terms of resources, labor, capital investments etc. for a pie from one baker
is going to be very similar to those of another down the street) the end-price
is going to be not very flexible; and any eventual elasticity will just drive
down _consumption_ rather than _price_. The degree of availability of
substitution options is a major influence. That's why even major supermarket
chains operate with low single-digit profit margins, even on a product (food)
where there is obviously a constant and high demand.

Compare that to software where no two offerings are direct substitutions (the
degree varies, of course) and where one company can upset a market with new
technology very easily without his competitors necessarily being able to
duplicate (for skills reasons, legacy investments, ...). An incumbent can
enter the market with fixed costs that are 75 or 50 or 25 percent of the
established players. Such an incumbent can do two things: go deep on price to
get market share, or get fat off 500% profit margins. I don't know of many
such situations in physical goods businesses.

(of course the same could happen with physical goods, it's just much more
common in technology)

------
rabidsnail
Has anybody tried using hill climbing to arrive at an optimal price?

eg:

1\. Pick a pivot price

2\. Make N buckets with prices at fixed intervals from the pivot

3\. Find the bucket with the most profit, and use its price as the new pivot

4\. Repeat

~~~
khafra
Fascinating concept. Makes me want to start selling something in large
volumes, just so I could apply other optimization techniques like simulated
annealing or beam search to price points and other scalar variables.

------
btilly
A major complication that you do not address is how to do confidence testing
on the results of your pricing difference. There is an important difference
between "A did better than B but who knows why" and "We're confident that
there is a real difference." Figuring that out is not easy.

~~~
paraschopra
Yes, that's right I should have touched that point. I think the revenue for
each variation can be approximated as normal curve and then a hypothesis test
can be done to see if difference in mean is statistically significant.

~~~
btilly
That approach absolutely works. But fairly few people have enough of a math
background to figure out how to do it.

There are also complications around noise from large outliers. Though this
matters less with a fixed price service.

------
wensing
Keep in mind that this is secondary to finding the market that puts the
greatest value on your product.

~~~
paraschopra
Interesting. Can you clarify a bit more? An example would really help. Thanks.

~~~
wensing
Stormpulse has recently shifted from B2C to B2B. Pricing is so completely
different there's no A/B test you could use to determine that.

Say you were selling tickets to hear Lang Lang play the piano. You could try
selling those on the corner of 3rd and Main and get $20 for them because no
one you talk to (unless you get lucky) has heard of him (and you could waste a
lot of time convincing them that he's great). Or you could move your booth to
the more fashionable part of town and sell them in 5 minutes for $250 each.

~~~
wensing
This is why I think the golden goose of web optimization tools is going to be
the one that automagically segments a user base into population/profiles based
on their repeated behaviours and their origin IP (consumer/biz) (provide a
klout.com type of analysis on everyone that visits/registers). It will allow
startups to see their potential market segments and follow the most promising
ones.

~~~
revorad
Oooh there's this company that starts with an F and ends with a k. Ah but the
valuation's too high. Fuck!

------
acangiano
Pro tip: Have at least three pricing levels with directly comparable features.
It will increase the number of sales for the intermediary level.

~~~
jacquesm
I've heard that one before and I've seen sites that do it but I have yet to
find a place where there is actual evidence that it works, do you have a
reference for that?

We're debating trying this and I'd like a bit more evidence that it really
works and why it does.

~~~
tomkinstinch
Here are a few citations, not all directly relevant, from Dan Ariely's
_Predictably Irrational_ (itself a good read for any startup founder):

___________________________________

1\. Amos Tversky, "Features of Similarity," Psychological Review, Vol. 84
(1977).

2\. Amos Tversky and Daniel Kahneman, "The Framing of Decisions and the
Psychology of Choice," Science (1981).

3\. Joel Huber, John Payne, and Chris Puto, "Adding Asymetrically Dominated
Alternatives: Violations of Regularity and the Similarity Hypothesis," Journal
of Consumer Research (1982).

4\. Itamar Simonson, "Choice Based on Reasons: The Case of Attraction and
Compromise Effects," Journal of Consumer Research (1993).

5\. Amos Tversky and Itamar Simonson, "Context-Dependent Preferences,"
Management Science (1993).

6\. Dan Ariely and Tom Wallsten, "Seeking Subjective Dominance in
Multidimensional Space: An Explanation of the Asymmetric Dominance Effect,"
Organizational Behavior and Human Decision Processes (1995).

7\. Constantine Sedikides, Dan Ariely, and Nils Olsen, "Contextual and
Procedural Determinants of Partner Selection: On Asymmetric Dominance and
Prominence," Social Cognition (1999).

~~~
revorad
Robert Cialdini's work also has some examples -
<http://www.influenceatwork.com/>

------
podperson
The problem with doing this is that if your customers find out they're likely
to be pissed off. If this doesn't matter to you then feel free.

------
rcavezza
I don't like the first paragraph.

"For example, if you are manufacturing staplers, all you need to do is to
calculate cost of production and distribution, slam 20% margin on it and there
you have the price you can sell your shiny stapler machines for."

That's called Cost Based Pricing and is one of many options. That's a decision
that was made. You can argue that's the wrong way to price. If you make
something in the US and charge 20% on top of costs and have a competitor
making Staplers in China, you're out of business.

Boom. Roasted.

------
bnoland
If 99% of your value is in the base product and 1% is in the advanced
features, most people will chooses the cheapest plan available.

My customers don't care about ssl or extra storage so they'll always choose
the cheaper plan. But if the basic plan was +$10 what percentage would still
buy it?

~~~
jules
He is not saying you should offer a second basic plan next to the primary
basic plan. He is saying that you should AB-test two different basic plans,
only one of them is visible for any given customer.

------
rmc
Why is it illegal to offer different prices for the same service (as this
article says)?

~~~
paraschopra
Read this: "PRICE DISCRIMINATION AND THE LAW"
[http://www.imakenews.com/strategicpricing/e_article000649775...](http://www.imakenews.com/strategicpricing/e_article000649775.cfm?x=b11,0,w)

~~~
jat850
The article you linked specifies that the statute in question only applies to
goods, not services. Is that applicable here?

~~~
paraschopra
I am not a lawyer but I think it should apply to digital products. Perhaps not
consultancy services, if you are offering a product, it should apply.

------
bnoland
What if you do display different prices, but at the end of checkout you
surprise them and charge everyone the lowest price in your range.

~~~
jeremydavid
That is a really interesting idea.

The only issue I would see with this is that having the sudden "We lowered the
price!" message would influence conversion rates at the final step.

You would have to measure and compare the number of "add to baskets" and
"proceed to check outs" and not the revenue / conversions.

------
phylofx
Well, if one has to offer "completely" different products at different prices
for testing purposes (say product "x", "y", "z") like the author suggests, how
can you determine the ideal price for product "x" in the end?

------
lachyg
Good post, I think more people should be testing different values (I remember
reading something where someone kept doubling the price til they reached the
tipping point).

Cheers

~~~
paraschopra
Yes, price testing is very important at initial stages of launch. It becomes
progressively harder as your product gets established.

After your product is established, the only way to test price sensitivity is
to launch pro version.

~~~
ambirex
A pro version isn't the only way, you could launch:

* light version

* experiment with different sales and promo codes

* offer different rates for paying ahead a certain amount

------
stcredzero
Any best practices for A/B testing on the iOS App Store?

~~~
RobertKohr
I have done this with an iphone app. The ideal case for testing:

-Your app has little publicity and is generally found through keyword searching. This means that you have few repeat visitors to the app page

-Your daily sales data has little variation. You typically see this 2 weeks after release when it is no longer in the new apps list. Avoid holidays and other sources of variation for your test.

-Schedule pricing into the future on one week increments. Try not to have pricing go generally up or down, perhaps by using some randomization or shuffling of possible prices.

-Use the revenue generated during each of the weeks coupled with the pricing to identify your ideal price.

Using poor scientific measurements (I crossed holidays, and did a declining
price from a high point) I came up with $2.99 as an ideal price for my app.

Keep in mind that your sales rank influences your revenue, and if you think
your app can break the top 100 apps list, then price it at $.99, because once
you get into that list any losses prior due to underpricing an app are
nullified.

Too bad the app store doesn't primarily sort by revenue (aka value to the
consumer) and doesn't provide a continuous scroll past the top 100 which would
make such devaluations unnecessary.

