

A/B Testing You'll Actually Use: Optimizely (YC W10) Launches Out of Beta - dsiroker
http://www.optimizely.com/?utm_source=hn
It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public!<p>Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started.<p>We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback@optimizely.com and let us know what you think!
======
cperciva
You might want to adjust the pricing tiers. With $19/$79/$399 per month tiers,
I have to guess that most people will start by trying the lowest tier; but
that's limited to 2k visitors.

If you're doing an A/B test with 2k visitors and you've got an "A" conversion
rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order
to have statistical significance. That's a huge improvement, and I doubt many
people will be that lucky.

You'll probably be better off setting the limit for the lowest tier at 10k
(and adjusting the others upwards too); that should dramatically increase the
number of new users who find your service useful and stick around.

EDIT: Or another option would be to have a "free 10,000 visitors" trial rather
than a "free 30 days" trial.

~~~
dsiroker
All our plans start with a 30-day free trial so anyone can start with Silver
or Gold plan and at the end of the trial period switch plans.

We also offer 100% Money Back Satisfaction Guarantee so if at the end of any
month you aren't satisfied for any reason (including not reaching statistical
significance), we'd be happy to offer you a full refund for the month.

~~~
cperciva
_All our plans start with a 30-day free trial so anyone can start with Silver
or Gold plan and at the end of the trial period switch plans._

True, but that doesn't help people who don't get that much traffic within a 30
day period.

 _We also offer 100% Money Back Satisfaction Guarantee so if at the end of any
month you aren't satisfied for any reason (including not reaching statistical
significance), we'd be happy to offer you a full refund for the month._

Offering refunds is great, but it's even better if they're not needed. :-)

~~~
dsiroker
_Offering refunds is great, but it's even better if they're not needed. :-)_

Certainly agree! :)

~~~
catch23
Why not just collect their credit card, but don't bill them initially so that
the barrier to entry is low, then notify that they'll be charged at the end of
the month. If they're pleased with the service, they'll probably ignore the
email.

Requiring the user to remember to cancel or downgrade seems potentially
sneaky.

~~~
dsiroker
That's actually how it works right now. We ask for your credit card initially
but you don't get billed until 30 days later.

~~~
andrewf
If you intend to keep things this way, then don't call it a Money Back
Guarantee. Instead, be really clear that users have a trial, and billing is in
arrears. Something like:

    
    
      YOUR CREDIT CARD WILL NOT BE CHARGED!
      
      We collect credit card numbers during signup to
      prevent fraud. But we won't take a cent until your free
      thirty day trial is complete.
    
      The trial is obligation-free; just visit our easy-to-use
      cancellation page during the first thirty days and you'll
      never be charged.
    

Consumers know that guarantees and rebates are intentionally painful to
collect. They feel more confident and empowered when you don't have their
money yet.

One caveat: I've seen successful trials that are a week or two long. At thirty
days (and with users who aren't very committed), you might have an issue with
users forgetting about the forthcoming charge, and then getting angry when it
shows up on their bill next month.

~~~
patio11
You can alleviate a little of that with an email on day 25~27.

------
theospears
Disclaimer: I have watched the video but not tried the product, so some of
these observations may not apply to the the product itself.

Overall, this looks like a great product. I know a number of people who find
Google Website Optimizer complicated to use, and I would definitely recommend
this to them as a simpler option. I love how slick the browser interface to
edit pages is, and I think having the default 'engagement' metric so people
can see results without having to set up a goal page is a brilliant idea.

There are a few things you have done which I would consider doing differently.

1\. It looks like your mission is to make A/B testing really easy, but your
pricing page at the moment doesn't really reflect that. Number of visitors
tested is an easy metric, but one that it is hard for me to interpret without
lots of knowledge of A/B testing. How many tests does this mean I can run how
quickly?

I would also reconsider the additional features you offer in premium packages.
Cross-browser testing sounds complex and makes me worry that your site edits
will fail in IE6. I don't want to have to test it, I just want it to work.
With uptime monitoring, what does this have to do with A/B testing? Bigger
sites probably already have some form of monitoring already anyway, so it
looks like they are going to pay for something they don't need. I think your
core product is strong enough that you don't need to offer these.

2\. Showing me the percentage significance level appeals to my inner stats
nerd, but I suspect the sort of people I think will benefit most from
Optimizely will have difficulty interpreting this number. What level is 'ok'?
Having a rank out of 5 below doesn't really address this, is 4/5 ok or do I
need 5/5? Google deal with this very well with their bars which turn red or
green when they reach significance.

3\. The 'select container' option to expand the selection seems non-obvious,
and isn't how multi-select works in any other interface I've seen. Maybe allow
people to select multiple components and then take their deepest common
parent?

There are also some additional features I personally would like to see

1\. It would be great if you gave an estimate for how long until my experiment
will reach an appropriate significance level (obviously based on % change and
traffic seen so far).

2\. I would like to be able to choose my conversion action by clicking on a
form button or link in your page editor.

3\. It would be amazingly useful to have some automatic suggestions for how a
page could be changed - on many occasions I've seen people resist A/B testing
because the options are so wide and they don't know what to do. Doing this for
some simple suggestions sounds possible - e.g. making key links bigger and
moving them up the page. Doing anything more sophisticated could be a good
challenge though :-)

~~~
dsiroker
Great feedback!

On pricing page: I agree this isn't perfect. What metric would you like us to
segment our plans by ideally? We want to make this as simple as possible so
folks know how to budget for this and so they can easily know which plan makes
the most sense for them.

On additional features: Are there any other "value-add" services you think
bigger companies might want besides just more visitors?

On 'select container': we realize this is a bit confusing and we're working on
implementing a multi-select almost exactly as you described.

On choosing conversion action: right now we automatically track all reasonable
conversion events on a page (clicks, form submissions, subsequent pageviews,
custom events) and we want to make this even easier by allow you to explicitly
create a custom conversion event to track by specifying it when you are
editing the experiment.

On automatic suggestions for how a page could be changed: this is a hard one
and we hope to get there eventually. In the mean time we're going to try to do
a better job blogging about best practices and lessons we've learned working
with our customers.

Thanks again for all the feedback!

~~~
YaiEf
Another possibility for pricing would be to simply buy a one time number of
visitors for a test. I think that would be a much easier first time sell in my
organization.

One test, 30k visitors, x statistical significance - $100.

Perhaps with the option of only deleting variations on the fly that don't
work.

------
petekoomen
It was three months ago to the day that we announced our private beta. We were
thrilled to see thousands of people sign up for the beta. We've worked hard to
improve the product over the last three months based on their feedback and
we're now ready to come out of beta and launch to the public!

Our goal is to make it as easy as humanly possible for you to create and run
A/B tests on your site. All you need to do is enter your website URL and point
and click on what you want to change. Absolutely no coding or engineering
required. You don't even need to create an account to get started.

We're working hard to improve the product every day based on your feedback--
just reply to this comment or shoot us an email at feedback at optimizely dot
com and let us know what you think!

~~~
tnorthcutt
I actually got excited watching that video, and I don't even have something to
use it for right now. Looks fantastic. I also really like how you have the
plans setup to record stats from all visitors, but only unlock them if you buy
the appropriate plan. I assume that means you can start out with the small
plan, then upgrade and see stats that have already been recorded? If so, very
cool.

Based on the results in the promo video, looks like you should have your video
autoplay! ;-)

~~~
dsiroker
_I assume that means you can start out with the small plan, then upgrade and
see stats that have already been recorded? If so, very cool._

That's exactly right! Glad you like it.

You can also start at a Silver or Gold plan for your 30-day free trial and
downgrade at the end of your free trial if you like.

------
Ataraxy
What's the USP on Optimzely when compared to the already quite formidably
awesome Visual Website Optimizer?

Also, as someone that really wants to incorporate this tech into my larger
scale marketing efforts I am put off by the pricing models to both of the
above mentioned sources.

When only one of several ad campaigns are doing 100k visitors a day you can
see how that pricing model does me no good. Particularly when you take into
consideration that I never stop testing -something- on a campaigns landing
page.

Also would love an API so that something like this could be tightly integrated
into my custom conversion analytics solution.

Congratulations on your launch nonetheless.

~~~
dsiroker
Right now our biggest competitor is non-consumption. Businesses see the value
in A/B testing but just aren't doing it. The pain points we are trying to
solve are setup & implementation, easy-to-understand real-time results, and
resolving the marketing & IT impasse within organizations when it comes to
doing A/B testing: marketing wants to do it, IT doesn't have time to help. We
want to enable marketers to run A/B tests without having to rely on
engineering or IT.

Great feedback on the pricing model. I agree this isn't perfect. We do offer a
100% Money Back Satisfaction Guarantee so if at the end of any month you
aren't satisfied for any reason, we'd be happy to offer you a full refund for
the month.

~~~
mahmud
_Right now our biggest competitor is non-consumption_

You think like a winner, my man. You had your opportunity to rag at the
"competition" but you were wise enough to see the big picture. Cheers!

------
rumpelstiltskin
Anyone do a comparison of Optimizely and Visual Website Optimzer?

~~~
paraschopra
DISCLAIMER: I'm the founder of Visual Website Optimizer but I will try to give
a fair and honest comparision. Optimizely team, please feel free to correct me
if I am wrong in my understanding.

What VWO has that Optimizely doesn't

\- VWO has (JavaScript based) A/B and multivariate testing and split URL
(based on traffic redirect) on while Optimizely is strictly (JavaScript based)
A/B testing. You cannot do multivariate testing with it.

\- VWO has heatmap and clickmap reports while Optimizely doesn't

\- VWO has various advanced features such as visitor segmentation, cross-
domain tracking, testing behind login wall pages, test results notifications,
variation screen shot generation, etc.

\- VWO has team collaboration such as multiple-permission based logins and
subaccounts. Optimizely doesn't have that.

\- In VWO, for all plans, CEO equally shares the support work ;)

What Optimizely has that VWO doesn't

\- Changing layout of the page. Optimizely has a drag and drop feature in its
test designer which is useful if you want to test different layouts (say
sidebar to the right). To achieve the same in VWO, you need to enter some CSS
(say float:right v/s float:left) as there is no drag-drop support (yet)

\- Since Optimizely exclusively focuses on A/B testing (and not MVT), in its
test designer you can change several elements of the page (say headline, image
and text) at once to create a variation. In VWO (currently), you need to
change one element at once for A/B test and multiple elements for multivariate
test.

\- In optimizely, results update on the page in real time. In VWO, results are
also realtime but you have to refresh the report to see latest results (no
auto-refreshing)

I hope this was a fair comparision. I wish Optimizely team great luck for
their product. I hope A/B testing market is big enough for multiple players to
survive!

~~~
dsiroker
Great honest and fair comparison, Paras.

One benefit you forgot to mention is that your cheapest plan ($49) comes out
to $0.0049 per visitor. Optimizely's cheapest plan ($19) comes out to $0.0095
per visitor.

One thing I think we do well is allow you to try Optimizely and create your
experiment without needing to create an account or give us your email address.
The only time you need to create an account is when you want to save.

We also offer cross-browser testing and uptime monitoring and reporting.

One small correction: we do support testing behind login wall pages. You just
need to put the embed code on your page and the tool will load your site even
if it's behind a login wall or on your own local intranet behind a firewall.

Our biggest competitor is non-consumption. I'm glad there are other startups
out there helping to educate the market about the benefits of A/B testing. A
rising tide floats all boats!

------
ssharp
Can someone A/B test a web startup's name with words ending phonetically as
"ly" or "er" versus a more "normal" name?

Kidding aside, the product looks really great - congratulations on launching!

For a critique, I'll say that the intro video is very dry. A/B it with
something less Ben Stein?

One thing I'm always interested in is pricing. How did you go about
determining your various plans and rates?

~~~
dsiroker
Sorry you thought the video was dry. I'm not the best voice actor in the
world. We were thinking of putting a song from the Inception soundtrack in the
background or using a real voice actor from <http://www.voices.com/>

Do you think either would help?

Also, would love to share how we went about determining our various plans and
rates. We did a lot of customer development on this and it might be
interesting to other entrepreneurs. Let me know if you are interested and
we'll blog about it.

~~~
ssharp
I actually would be interested in seeing results of testing this video vs. a
voice actor and maybe even against a set of videos that were split up and
shorter.

~~~
dsiroker
Great idea! We're actually running an Optimizely experiment on the
<http://www.optimizely.com> homepage right now. I won't give away what we're
experimenting with but we'll blog about it once the experiment is over and
share what we learned.

~~~
photon_off
This post prompted me to have a look at optimizely's source. I won't give it
away either, but I'll certainly be surprised if there's an appreciable
difference between the 3 variations. In fact, if turns out there is, I'd be
much more inclined to use A/B testing in general!

Now, a coding question. I'm wondering: For the all_experiments_json property
of the main optimizely object, why is the value JSON-as-a-string, rather than
just JSON? You're forcing yourself to parse it from a string to native JSON
for no apparent reason. Just curious :)

~~~
dsiroker
Thanks for not giving it away. :)

As for your question, I think you're right that we could just write this as
straight JSON without wrapping it in quotes. What are the benefits of doing
this?

I tried just putting the all_experiments_json property in Firebug without
wrapping it in quotes and I get a 'SyntaxError: invalid label' error. Looks
like that is because the keys of the dicts are strings. Probably a way to
generate this JSON server-side that makes Javascript happy.

One potential downside of making it just straight JSON is that this is
generated by our Python template and if for whatever reason this isn't valid
JSON it would throw a javascript exception when the file is loaded.

Right now our backend that generates this JSON does this:

    
    
        dump = simplejson.dumps(experiments_dict)
        dump = dump.replace("\\", "\\\\")
        dump = dump.replace("'", "\\'")
    

The dump is the block of text that gets wrapped by '' in our template.

Thoughts on a better way to do this?

~~~
photon_off
First off, there aren't really any benefits, other than it being simpler and
more elegant. Right now you're going from Python Dictionary -> JSON-like-
string -> JSON. There should be a way to go Python Dictionary -> JSON. It just
seems like a code smell. Really though, if it ain't broke, don't worry about
it. I was just curious as to why there was this roundabout.

Removing the quotes won't fix the problem, because you've also altered the way
characters are escaped via the .replace() calls.

 _One potential downside of making it just straight JSON is that this is
generated by our Python template and if for whatever reason this isn't valid
JSON it would throw a javascript exception when the file is loaded._

I'm not familiar with Python, but there shouldn't be any reason why simplejson
would return invalid JSON. In the event that it does, your script would still
break, anyway. Granted, it won't be a javascript exception, but it would
certainly fail elsewhere.

Like I said before, it really doesn't matter because it works :)

------
paolomaffei
I don't understand people _using a CMS_ thinking that Google Website Optimizer
(GWO) for A/B tests is hard

I got myself a plugin for WordPress that lets you insert in any page the GWO
javascripts I created a variation page on WordPress, identified the original
one and the conversion page. On the site i created the test, started it and
it's just working.

But even if you're not using a CMS, copy/paste some code at the beginning and
end of a HTML page? how is that so hard?

Edit: clarified

~~~
patio11
Using variation pages causes SEO and site problems when the test ends unless
you manage your 301s properly, and the marginal load for the redirect to B
artificially biases test results. Also, I did a bit of an eye popper demo at a
company I just consulted with: deploying an AB test live on my site took two
minutes twenty six seconds while slowing my typing speed to explain every
step, while the company reports their experiments with GWO require 15 minutes
at a minimum. That gives the typical engineer 12 reasons not to AB test.

~~~
dsiroker
Hi Patrick, glad to hear that! Curious what you think about client-side A/B
testing products in general (Optimizely being just one of them) vs. server-
side products (like A/Bingo). Do you face pain points with A/Bingo that might
be solved with a client-side javascript based solution?

~~~
patio11
In general, I feel server side is _far_ superior for developers and client
side with a WYSIWYG UI is a fabulous option for other stakeholders. I could
talk on this subject for a while but my plane is about to take off. Suffice it
to say you can't match power, flexibility, and speed of server side with
client side testing. You can move away from just twiddling text to actually
redoing workflows or business logic.

I didn't write ABingo with a pain point included, but for Appointment Reminder
it will be helpful to do AB testing in non-Rails marketing pages, too. I
wouldn't append too much effort optimizing your solution for the needs of
people who have written AB test frameworks though.

------
SudarshanP
Really cool A/B Testing tool :)

Your variation editor looks really awesome too. How much effort does it take
create something like that? I mean just the HTML hover/manipulation etc. I am
planning to write a scraper related tool similar to <http://open.dapper.net/>

Did you encounter any serious cross browser issues? Or any other notable info
you would like to share with the HN community. Is there any open source basic
implementation that can be used as a starting point? Do you preprocess the
page on the server or is it all Javascript magic on the client side?

~~~
dsiroker
Our biggest challenge in building this tool was compatibility. Every webpage
is different and even seemingly innocuous things things like frame busters and
javascript redirects presented quite a challenge for us. We were fortunate to
have a great set of early beta customers that gave us feedback when things
didn't work and we took it one website at a time.

We'd love to share the lessons we learned along the way with the HN community
in a blog post. Let me know what you're most interested in and we'll share
what we've learned.

~~~
photon_off
How did you overcome frame busting and javascript redirects?

~~~
SudarshanP
<http://en.wikipedia.org/wiki/Framekiller> may be useful. But I am curious
about how it is done at Optimizely

~~~
photon_off
This is frame busting. I'm looking for anti frame busting. There's one way
which involves calling a page which returns a "nothing" header code when the
iframe tries to bust out. Though some sites like stackoverflow even take this
into account. I wonder if SO works with optimizely (I'm on iOS, can't test it
ATM).

Since optimizely has access to the document, there might be a way to wrap the
window.location object and prevent it from changing, before any experiment-
site js is loaded.

~~~
SudarshanP
[http://stackoverflow.com/questions/958997/frame-buster-
buste...](http://stackoverflow.com/questions/958997/frame-buster-buster-
buster-code-needed) This keeps getting funnier and funnier :)... that must be
the state of the art in recursive busting ;-)... And if someone is so obsessed
that his page must not be framed... I guess it is best to leave such a guy
alone...

------
moe
This looks extremely slick and well executed. Hats off, my feeling is you guys
will be quite wealthy in short order.

~~~
ericflo
I certainly hope so. Great product idea, well executed.

------
hammerdr
Suggestions:

In a couple of spots, the edit HTML dialog overlaid the items I was trying to
edit. I attempted and expected to be able to move the edit dialog around the
page and out of the way of what I was editing.

On my own site, I was particularly wanting to play with colors/themes. Not
sure what the best way to implement this is, but the only way I could do that
through this interface was with style tags.

Awesome product! I will likely use it in the near future.

~~~
dsiroker
This is a great suggestion. We try to automatically put the dialog in the
right place depending on the action you are trying to perform. For example, if
you use drag-and-drop and move something to the right-edge of the window, we
move the instructions dialog to the left-hand side of the element.

This certainly isn't perfect and we'll work to improve it. Would you mind
sending me the URL you were trying and some steps to repro the frustration you
had? Feel free to email us at feedback@optimizely.com

Thanks!

------
JonM
This looks brilliant, you just need to nail your pricing structure.. perhaps
consider per test or limit the number of tests and variations? Either way,
congratulations, some serious skills on show form your engineer(s)!

------
senko
Great stuff, the video made me want to jump right in and test stuff; too bad I
don't have any medium/big sites to test :)

For us folks with small-traffic websites (ie. your Bronze plan), an useful
addition would be a suggestion to people to not try too many variations at the
same time. (I don't have the math, but I assume less variations means less
datapoints until you get significant results)

Perhaps, as another commenter suggested, by estimating how long (in time or
visitors) it will take - and by showing this estimate _while_ adding
variations.

~~~
dsiroker
You are correct, the more variations, the longer the experiment will need to
run to reach statistical significance. Estimating how long it will take to
finish is a great suggestion and something we plan on doing.

------
davidnelson
nice idea. I tried to setup a new a/b test for a site and it became 'not easy'
when I wanted to track an adsense click as a goal.

I wrote a manual a/b test for the same site about 7 years ago, and got adsense
click goals working with 40 variations of the page, after about a week of
working on it after work at night after my regular job.

There is some jquery code you can add to track javascript events which would
probably track adsense clicks, but it's 'not easy', ie: I might as well not
pay for it.

Cool idea though! I would pay for it if I could track adsense clicks without
having to write code. Also, having an option to select multiple dom elements
at once would be nice. In addition, logging into the web site and editing the
html to add the code in the <head> tag would be nice too, especially since
it's kind of a hassle for me to open eclipse and re-deploy to app engine at
1am when I am tired.

haha, hope this helps.

~~~
dsiroker
Certainly something we are well aware of and trying to improve! We'd love to
automatically track ad clicks for you and provide the corresponding reporting
to let you know which variation gets you the most clicks.

Please email us at feedback@optimizely.com and let us know if you're
interested in this and I'd love to learn more about this use case.

~~~
photon_off
I think it'd be amazing if you could work smething out with the ppl at
mouseflow or thier ilk. They could track the clicks and mouse movements for
you. Maybe offer it as an add on for a test.

------
enjo
Do any of these A/B testing tools support offline conversions in any
meaningful way? For us a phone call is a conversion. I'd love to be able to
build the logic to support that once (give me an API call along with GET
parameters to the page that I can post back to you), and then be able to use
tools like this to actually try different tests.

~~~
jonknee
I imagine it would be quite difficult to match up the phone call with what
version of the website was being used. One idea: depending on how your
incoming calls work you could try using different extensions on each version
which would let you track it. A different phone number all together would work
too, but might be confusing and bad for SEO.

------
xal
It's a great product, we have been using it at Shopify with great success.
Fantastic news that it's available to everyone now!

------
revorad
Great job on the product and video - really easy and slick.

What's the use of real-time results for A/B testing? Most likely, it's only
going to lead to erroneous conclusions, as discussed in this thread -
<http://news.ycombinator.com/item?id=1778104>.

~~~
dsiroker
Great point. Jumping to pre-mature conclusions is certainly an easy mistake to
make.

During the Obama campaign we were frustrated by having to wait 24 - 48 hours
to get results for our experiments. We were lucky to be getting a ton of
traffic and hence be able to get statistically significant results in just a
few days.

Another big benefit of real-time results is helping our customers answer the
question "is this working correctly?" Seeing the data come in in real-time
helps reassure folks that the tool is working properly and collecting data.

~~~
revorad
_During the Obama campaign_

Whoaa why isn't there a crazy big testimonial screaming that on the homepage?

[EDIT: Sorry I misunderstood as you having used optimizely for the campaign,
but it sounds like you probably thought of the idea as a result of your
frustration.]

I can understand customers getting frustrated for having to wait. But, that's
probably what will make them jump to conclusions. I don't know how much you
want to get into advocating "best practices", but it would probably be good to
ask the user to set their minimum number of visitors beforehand.

The "is this working correctly?" question sounds like a very good practical
issue you picked up during your beta testing.

------
tsbaron
This website is great. I was able to jump in and mess around with my website's
home page within seconds after watching the video. I'm not a developer and
being able to help out with A/B testing will be a huge help and one less thing
our developers will have to deal with. Looking forward to using it!

------
Swoopey
This is great, just in time for my testing :), I'll definitely give it a try!
I would love for one of your pricing packages to include speaking directly
with an optimization consultant, or perhaps receiving a detailed report that
includes optimization recommendations.

------
dav-id
Why do all of these companies create cookie cutter names like 'Optimizely' is
' _ly' the next .com?

To me _ly says 'We have no track record, use with caution' I would feel more
comfortable using something with a name that isn't following some online
fashion.

------
bretthellman
Optimizely sounds like a great service but after entering my URL it took a
long time to load. It would be great if you could speed that up... If I hadn't
heard of the link from Hacker News, I would have closed the tab after a few
seconds.

~~~
dsiroker
Great feedback and certainly something we plan to improve. Thanks for bearing
with us!

------
brc
I get the impression that home page has been optimised to within an inch of
it's life and converts like crazy. A very impressive landing page, I had the
overwhelming urge to put in my Url and try it out. Well done.

~~~
prawn
I didn't believe it could possibly be that easy until I watched the video.
Subsequently, I think they should add text to that effect near the video.
After all the positive talk on HN, I was expecting something clever but was
still surprised by the implementation (single line to paste in!).

I doubt I'm the only one who doesn't always jump to watch demo videos (unless
referred by HN) or throw my URL into a strange form. For Ask HN's, I usually
just check out the design and try and get a rough idea of the concept, FWIW.

------
rmoriz
Will it be possible to define URLs to different, already existing versions (as
google does)?

Your online editing is awesome esp. for small things but sometimes one might
be able to render already existing pages…

------
zacharyc
This is a company of really smart and talented people. I have had limited
experience with the beta, but have been following the project with much
interest. I expect great things from Optimizely.

------
kevintwohy
Just gave it a spin and it's a really cool product - feels a bit buggy at the
moment but for the right application this would be a really valuable tool to
have. Great job, guys.

~~~
petekoomen
Hi Kevin, great to hear! I'm curious what felt buggy. Feel free to email us at
support@optimizely.com and give us a heads up. We'd love to look into it!

------
avichal
This is a really sweet product. Congrats on the launch!

------
reedlaw
I just went to dropbox.com and the play button got larger and larger each time
I hovered over it. Could this be a bug, or a weird optimization test?

~~~
petekoomen
Hi Reedlaw--I'm not able to reproduce this. I'm assuming you mean you created
an experiment on dropbox.com and that the play button somehow gets larger
every time you hover your mouse over it. What browser are you using? Thanks!

Pete

------
justinchen
Any chance this would work with dynamic content? i.e. a listings page that has
similar structure from page to page but different content?

~~~
dsiroker
Yes, we designed Optimizely with precisely this use case in mind. Here is the
documentation on how to set this up:
[http://support.optimizely.com/faqs/getting-
started/experimen...](http://support.optimizely.com/faqs/getting-
started/experiment-urls)

To clarify: If you want to run one experiment on a single dynamic page (not
multiple dynamic pages) you can do that as long as the structure of the page
does not change. The best way to find out is to just try it. Load one of these
dynamic pages in the tool and create your variations. Then put the embed code
on your site and click "Preview variation" from the variation drop-down menu
(you can get there by clicking the small upside down triangle next to the
variation title, e.g. "Variation #1"). If you have the embed code on your site
this will show you the page on your live site. You do NOT need to start the
experiment to do this.

Hope that answers your question

~~~
justinchen
Awesome, I'll give it a try!

------
Abid
Fantastic stuff. I was wondering if users are just shown the different pages
randomly or is there more granular control involved?

~~~
dsiroker
Great question! Visitors are bucketed randomly into each variation. Once they
have been bucketed they are cookied and will see the same variation over and
over again if they reload the page

You specify how much traffic you want to allocate to each bucket. You can do
this when editing an experiment and going to "Advanced Settings" ::
"Percentage of Traffic to Variations"

------
mcotton
This looks great. It looks like you are using google app engine, maybe a
future post about how well it is working out for you.

~~~
dsiroker
Great idea! I'll try to convince my co-founder to write a blog post about his
experience going from product manager for Google App Engine to coder using GAE
every day as a customer!

------
usagi7
Looks really exciting. Sounds like a great idea. I'm looking forward to
checking it out.

------
aqwan
Very impressive. I will definitely share this among my peers.

------
jyu
Does this work well on https pages?

~~~
dsiroker
Yes! https pages are a great place to start experimenting because they are so
close to the conversion event most businesses care about: making money.
Optimizely works seamlessly on https and http pages alike.

