
Launch HN: Jika (YC S20) – Price A/B Testing for Shopify - kennandavison
Hi all! I&#x27;m Kennan and I recently founded Jika (<a href="https:&#x2F;&#x2F;jika.ai" rel="nofollow">https:&#x2F;&#x2F;jika.ai</a>). Jika helps Shopify sellers A&#x2F;B test their product pricing so they can make more money.<p>Let’s say you sell shoes on Shopify for $100 and want to test $110. You’d 1-click install the Jika Shopify App and go to a dashboard where you select and start a test for $110. We then handle the heavy lifting of showing price A to 50% of your visitors and price B to the other 50%. As the test is running, Jika keeps track of revenue, conversions, and visitors at each price point. You’d repeat this process as much as you can since the optimal price for a product is always changing.<p>Up until now, most people have priced off competitors, but that doesn&#x27;t cut it anymore with big companies getting more and more sophisticated. A prime example of this is Amazon accelerating their development of in-house brands (~640 in late 2019) and using their dataset to outprice other online sellers (2.5m price changes per day in 2013 and more today).<p>One recent test of a best selling product for a Shopify brand doing 1M+ monthly revenue resulted in a data significant ~10% revenue lift from a -7% price. This better price for the best selling product translates to a ~30k+ monthly revenue increase (~180k+ revenue over 6 months).<p>I chose to work on pricing because I love optimization. Past optimization includes dropping out of college after 1 year to be a growth (A&#x2F;B testing) engineer at Pinterest for 3 years and building an app for discounts + promotions at Hulu. In high school, I also optimized enough to be a top 200 player in League of Legends.<p>Would love to answer any questions or comments below!<p>PS. If you know anyone on Shopify, I&#x27;d love to help them nail down their pricing. :D
======
psadri
I remember people used to hate finding out they were shown a different price
from other people for the same product. Has this changed?

~~~
neonate
Could that problem be solved by changing the price at different times, rather
than for different people in the same time interval? I wonder if that's why
Amazon is making 2.5m price changes a day like the OP says above.

~~~
kennandavison
Unfortunately this wouldn't be a proper a/b test since it'd be comparing 1 day
of 1 price to another day of 1 price. Only by splitting traffic 50/50 do you
properly isolate the price variable.

~~~
neonate
Why can't the time slices be much shorter, say 15 minutes rather than a whole
day?

------
outloudvi
"Keep in mind that Amazon does dynamic pricing already so it's already
commonplace to not always get the same price."

(from [https://jika.ai/faq/](https://jika.ai/faq/) )

"Amazon also does something" doesn't mean that's right. As one of the biggest
cloud providers, AWS is widely recognized for its complicated pricing model.
Does it make a complicated price model good, welcomed, encouraged or well
reputed?

(Why I'm still standing to object this even when this thing seems to make
sense? Because I'm a customer.)

------
hn_throwaway_99
I recall seeing this blog post a while back arguing you should never A/B test
your pricing strategy (IIRC it was posted on HN, can't remember):
[https://www.priceintelligently.com/blog/bid/180676/why-
you-s...](https://www.priceintelligently.com/blog/bid/180676/why-you-should-
never-a-b-test-your-pricing-strategy) . I'm curious about your thoughts on the
points in this post.

~~~
kennandavison
Re point 1: in many cases people actually DO have the sample size if the
magnitude of the revenue increase is large enough (e.g. double price and
increase conversions - lots of stories about this).

Re point 2: in the end increased revenue from better pricing means you're
capturing more of the value you're delivering to customers. but yes, there's a
brand decision to be made here between 1 price that you always keep the same
vs. a changing price that is more optimal.

Re point 3: companies should always be testing their pricing since external
factors don't stay the same.

PS: this is also on the blog of a dynamic pricing company so they don't want
the people on the fence about handing over pricing control thinking about a/b
testing price haha.

------
cdipaolo
1\. How are you handling statistical significance questions / ensuring the A/B
test has enough statistical power to be meaningful?

2\. Same for seasonality.

Are you offloading much of the statistical concerns to the user or are you
hoping to answer these for them?

Have to say, I don't really understand the hate on this thread having worked
in this area. This seems like an awesome product that can really help sellers.

------
mtnGoat
Can you easily opt traffic out of testing based on traffic source? for
instance Google Shopping(Microsoft Shopping as well) is pretty strict about
the price in the ad page, matching the price on the product page.

do you plan to offer this product for other platforms? the number of Shopify
merchants with traffic and sales large enough to get good significance in a
timely manor is rather small, IME.

------
niqmk
Is your pricing [https://jika.ai/pricing/](https://jika.ai/pricing/) also A/B
testing? :D

------
dyeje
Do most Shopify stores have enough traffic for this to produce real results?
Seems like the vast majority of Shopify sites are small and are just going to
get random results due to a small sample size.

~~~
kennandavison
Depends on the magnitude of the revenue increase.

One thing that inspired me to build Jika were the stories of sellers doubling
their price and actually increasing their number of conversions. I haven't had
a customer like that yet, but I had one person who reduced their price 30% and
increased their conversions 5x+. In that case, it's pretty clear there's a
winner.

~~~
dustin
30% might work in software, but it's is a very deep discount for most physical
retailers.

If the product has a 50% margin (sells for $100, costs $50 to source) then
your retailer makes $100 for 5 orders instead of $50 for 1 order.

Except they're probably keeping their own inventory, so they now have to keep
$250 of inventory for that $100 in profit, rather than $100. This is
problematic because a lot of online retailers are limited by their working
capital.

This is a drastically simplified example, but the bottom line is that physical
retailers think very differently than software sellers.

Great idea for an app either way. I wish you the best of luck with it as you
learn the territory. For something similar in the Amazon space (a platform
that lends itself much less to split testing) check out splitly.com

------
pgodzin
Few questions:

1\. How do you determine who fits into the A cohort and who fits into the B
cohort? If I refresh could I fit into a different cohort and thus see a
different price? If I go Incognito?

2\. Instead of a 50/50 A/B test, can you multi-arm bandit a handful of
different price points and shift a higher proportion to the "best" one? And
thus be able to change over time as well?

~~~
kennandavison
1\. Using random id + localstorage, so yes, incognito could result in a
different price.

2\. I'd love to implement MAB, but focused on nailing core product for now.

~~~
kipply
How do you avoid customers changing their behaviours by observing pricing
changes? Or when they share it with a friend and go "cute shirt for 10
dollars" and their friend is like "wat it's 15 dollars"

~~~
cdipaolo
Could cohort based on geo-location instead of completely at random. Makes
analysis a bit more challenging though and doesn't completely solve the
problem of course.

------
wackget
Advertising prices like those in your example screenshots would be illegal in
the EU.

You can't imply that the price "was" $49 and is now $39 when it's also being
sold at $29 to another group of customers.

In fact, the whole thing is legally questionable according to EU price fixing
laws, not to mention ethically unsound.

~~~
srg0
Would advertising $49 as the base price and a time limited offer/discount for
a group of customers be acceptable?

------
desdiv
How do you deal with search engines? The vast majority of stores have the the
vast majority of their customers come in via search engines.

1\. Showing one page to the search engine spider and another to users is a
black hat SEO tactic known as bait & switch. Most search engines heavily
penalizes against this. How do you deal with your customers possibly losing
their search engine reputation that they've built up over the years?

2\. I don't know about other people, but when I see a price, X, on the search
engine result page, then get a different price, Y, in the actual store, I
immediately lose trust in the store. If the merchant is willing to bait &
switch me on the price, then I don't trust it not to bait & switch me on their
actual product or services.

------
davidajackson
This looks like a great infrastructure as a service company. Is there a way to
build an AB testing tool on Shopify that finds the right price by zeroing in
on it, rather than having the customer input several? Did you consider that
approach?

~~~
kennandavison
Yep, automatic testing will be coming soon! :)

------
newfeatureok
If this were to become popular what would stop someone from building something
to do the reverse of this, e.g. see if the item could be cheaper if you bought
it as another customer/from another device/computer/location?

~~~
asattarmd
That's probably a good problem to have! Nonetheless, customers are still
buying from the same store, so, it's a win-win. Breaking customer trust is
probably the only thing that's bad here.

~~~
kennandavison
Yeah, it's a tradeoff for sure. I would say discounts that don't apply to
everyone are a less organized way of a/b testing price. So a brand that does
discounts all the time on certain cohorts of their users is going to have an
easier time swapping over to using Jika vs. a brand that has 1 price, no
discounts ever, never have changed their price.

------
usaphp
I just want to point out that I like a way you describe what your product
does. Simple, clear examples and result. I like when people do that instead of
asking for your email just to see what your product is even about.

~~~
kennandavison
Thanks! Keeping things simple + self-explanatory is a big focus with Jika.

------
alphagrep12345
How does it compare to other A/B testing apps? Eg:
[https://apps.shopify.com/neat-ab](https://apps.shopify.com/neat-ab)

~~~
kennandavison
Neat A/B is a 100% daily traffic swap app. This approach isn't a valid a/b
test since it isn't properly isolating the variable (price).

~~~
neatab
Neat A/B Testing here! First off, congratulations on the launch. I would argue
showing different prices to simultaneous visitors is an ethical gray area -
not to say it's not something that shops do, or wish to do. But that's why we
choose to cycle pricing. In our experience, with proper consideration of cycle
timing, test length and traffic consistency, it's possible to get solid
results this way.

~~~
kennandavison
Hey there! Thanks for the congrats. I see your point, but external factors +
different cohorts visting throughout the test is a bit of a pickle.
Nonetheless, I wish you the best!

------
itsdavemartin
One observation: Pricing at $199, $499, $899 per month for a single feature
seems interesting considering Shopify itself charges $29 $79, $299 per month.

~~~
gustaf
Considering the value of Jika I'm surprised it's not higher price.

~~~
ketzo
Yeah, this is totally a "charge more" kinda scenario. A/B testing is, like,
THE first thing you do once your ecommerce business gets a little established,
right?

------
dontwaitesforme
Upsetting to see the negative reception to this. I know as a consumer this
sort of experimentation might leave a bad taste in your mouth, but on the flip
side I don't think you can blame a seller for trying to systematically find
the fairest price for their product in a market. Overall, I think this is a
cool idea with a simple but effective premise.

------
shafyy
Nothing against you personally, but I'm the only one who thinks it's weird
that a Shopify A/B testing app got into YC?

~~~
greenbay20
I'm incredibly disappointed this was the top comment. Instead of asking
whatever it is you want to know in an optimistic way, you just threw a random
comment that with decent chance just makes the OP feel worse.

For starters, a really simple question you could have asked that may update
your belief of how weird it is they got into YC: "How much money are they
making and how much are they growing per week?"

~~~
shafyy
I do think that an A/B testing platform can be a great business, and making
any business work is for sure an achievement.

I just had the impression that YC tries to back more "ambitious" or
"important" businesses, but may be I'm wrong about that.

~~~
kipply
ambition or "importance" don't always manifest in the early stages of a
startup (they probably usually don't). Shopify started as a snowboard selling
company and I'm sure that the founders here pitched YC a larger vision that
was appreciated.

------
gleb
Small suggestion: include how contribution margin changed in that example.
That’s the Northstar metric for a typical ecommerce company.

Including it would sound more credible, and would prevent a typical customer
from doing the math in their head of profit margin where this change becomes
profitable.

~~~
kennandavison
Thanks for the suggestion! Contribution margin/profit is another metric we're
going to track in the future for sure.

------
mgh2
Conversion rate optimization (CRO) specialist working on Shopify platform
here: Why can't I just duplicate a product page, change the price and do a
SPLIT instead of an A/B test? Some reviews will have to be ported/duplicated,
but it is totally doable.

~~~
kennandavison
It's better having 1 page from an SEO perspective.

------
kkt262
Why do you think you got accepted into YC? I'd love to check out your
application.

~~~
kennandavison
tldr; Pricing is painful. Most companies leave it to chance and there's a real
opportunity here to change that.

------
purple_ferret
Do privacy addons block this and if so, does your price tracking account for
it?

------
zatel
This seems like a great shopify app I will certainly be adding to my next
store. Do you have any go to optimization resources to share? (just general
stuff on how to think about optimizing things in ones own life)

~~~
aresant
Handful of good links on price elasticity testing

1 - [https://vwo.com/blog/ab-testing-price-testing/](https://vwo.com/blog/ab-
testing-price-testing/)

2 - [http://www.conversionvoodoo.com/blog/2012/12/why-price-
elast...](http://www.conversionvoodoo.com/blog/2012/12/why-price-elasticity-
testing-is-a-must-over-the-holidays/)

3 - [https://prisync.com/blog/price-testing-how-to-test-
different...](https://prisync.com/blog/price-testing-how-to-test-different-
prices-without-alienating-customers/)

~~~
kennandavison
Keep in mind price a/b testing is legal given that you aren't doing it off
protected classes (the VWO post says that price a/b testing isn't legal).

------
kipply
Is there an AI(specifically deep learning) element to optimally distribute the
A/B testing ratios or to suggest products/prices that may be good to A/B test?
(Referencing the .ai url)

~~~
kennandavison
Yep, automatic testing is definitely on the roadmap!

------
akbar-saliev
If this becomes popular what would stop Shopify from building this?

~~~
kennandavison
There's a lot of things Shopify should build, but yes, agreed, I am at their
mercy in many ways. Going to diversify in the near future.

------
pkiv
Congrats on the launch! Have you thought about exposing the API instead of
tightly coupling to Shopify? Similar to Split.io but for product pricing?

~~~
kennandavison
Yep, an API is definitely on the roadmap as well. :)

------
lukasfischer
This is evil... I would immediately stop using such a shop, if I know such
practices are used. Your solution is harming trust.

------
greenbay20
This is dope. As close as one can be to the money, niched down and in a
growing market. Good job! Keep us updated =D

------
itisit
Are there any good browser extensions you recommend that help detect a service
like yours?

------
gormanphil
Hi Keenan, want to share you product by Presenting at E-Commerce Day REMOTE on
oct 1? Shopify our lead sponsor, Rebecca Minkoff is keynote, 3,000 attendees
expected. You’d be a great fit...phil@ecommevents.com

------
jasoncui
Awesome product and love the landing page. Super simple and clear.

~~~
kennandavison
Thanks Jason!

------
break_the_bank
Congratulations on the launch! Curious why you went with venture funding over
indiehacking/bootstrapping?

Heard this podcast a while ago[0] where they discussed main dishes(who new
platforms) and side dishes(github apps / shopify apps). At the moment Jika
looks like a side dish, which seems well served by the bootstrapping
community.

Edit: Add link to podcast

[0][https://www.indiehackers.com/podcast/152-tyler-tringas-
and-j...](https://www.indiehackers.com/podcast/152-tyler-tringas-and-justin-
jackson)

~~~
kennandavison
Thanks for the congrats!

Yeah, Jika probably would've been a profitable side project, but one of the
goals here is to completely change pricing. It's super broken for most
companies right now and needs to be fixed asap.

------
thrownaway954
did anyone else do a double take and think the name was Jira?

------
lazydon
May be it's just me, but the first thing reading that name reminds me of is
the Zika virus. I wonder why would you choose a name for the startup that's so
close to something with horribly negative connotations. Especially, in today's
world scared to hell by viruses.

------
shafyy
> In high school, I also optimized enough to be a top 200 player in League of
> Legends.

Sorry but that's super obnoxious. No one in their right mind would use the
verb "optimize" in this context.

Feel free to downvote, but sometimes you need to call people on their bs.

~~~
ketzo
I play the game, so maybe I'm biased, but that's legitimately a huge
accomplishment. Use whatever verb you want I suppose, but that's literally top
0.001% in a game renowned for being incredibly difficult/competitive.

~~~
shafyy
I play esports myself and of course it's a great achievement. But the word
"optimize" just triggers me when used in this context.

