
Launch HN: RankScience (YC W17) – Automated Split-Testing for SEO - ryanb
https://www.rankscience.com
======
ryanb
RankScience automates split-testing for SEO to grow organic search traffic for
businesses. 80% of clicks from Google go to organic results and yet most
companies don't know how to improve their SEO, or can't effectively measure
their efforts to do so. Because of the scale of SEO and the constant change of
both Google's ranking algorithm and your competitors' SEO campaigns, the only
way to succeed in the long-run is with software and continuous testing.

We've built a CDN that enables our software to provide tactical SEO execution
and run A/B testing experiments for SEO across millions of pages. Experiments
typically take 14-21 days for Google to index and react to changes, and we use
Bayesian Structural Time Series and Negative Binomial Regression models to
determine the statistical significance of our experiments.

Our software is 100% technical SEO, and doesn't do anything black-hat, spammy,
or anything related to link-building. One of our goals is to bring
transparency and shed light on what is largely considered a shady industry,
but is so important to so many companies' revenue and growth. In fact, If SEO
didn't have such a bad reputation, we think someone else would have built this
a long time ago.

SEO as an industry earned itself a stigma for being spammy: between buying
links, creating low-quality pages stuffed with keywords and text intended for
Google rather than humans, and the used car salesmen attitude that many SEOs
have, many people have been conditioned to dismiss SEO as an invalid or
illegitimate growth channel.

We're software engineers-turned-SEO's, who have previously consulted for
dozens of companies on SEO, from YC startups to Fortune 500 companies like
Pfizer. We previously shared our case study with HN, where we increased search
traffic to Coderwall with one A/B test:
[https://www.rankscience.com/coderwall-seo-split-
test](https://www.rankscience.com/coderwall-seo-split-test)

Ask us anything! We'd love to answer any questions you have about SEO, A/B
testing, and RankScience.

~~~
etler
Can you explain a bit more about the technical details of what the platform
can do?

Does it integrate with web master tools and google analytics?

Does it help identify page populations that are stable and similar enough to
be compared?

How are the different test versions sent to you guys?

Can we use our own CDN on top of your CDN?

What about dynamic pages that change over time?

How big of a sample size is needed to get statistically significant results?

~~~
dillonforrest
Howdy etler!

> Does it integrate with web master tools and google analytics?

Yep!

> Does it help identify page populations that are stable and similar enough to
> be compared?

Yep!

> How are the different test versions sent to you guys?

For the most part, we own the design of treatments. Some of our customers have
their own experiments, but that's a small minority of our customers. Happy to
talk more in depth about this if you're interested! dillon@rankscience.com

> Can we use our own CDN on top of your CDN?

Yes, we encourage this too!

> What about dynamic pages that change over time?

We address this on a case-by-case basis, but our software is designed to be as
un-invasive as possible in use cases that we do not expect.

> How big of a sample size is needed to get statistically significant results?

We have to consider several variables here. Sample size is a function of
Google's crawl rate on your site, the number of pages you have, and the amount
of traffic you receive.

~~~
etler
Thanks for the response! I know the sample size question is hard to answer
because of the number of variables involved, but the reason I ask is that the
CoderWall blog post example mentioned 20,000 pages, which is more than we have
so I was curious about what a rough rule of thumb range might be.

------
austenallred
A few questions about your product (I run SEO and digital marketing at LendUp
- YCW12). This is very, very cool.

* What types of optimizations does it do/do you test?

* I assume tests take a while to run, waiting for Google to re-index etc. Do you essentially monitor rank changes, assume that Google has re-indexed at that point, and use that as the data to optimize on? Or do you have some smart way to monitor for when Google re-indexes?

* I'm again assuming here - that a user will plug in a few variations of things to test, and let your software test them? Or are there automatic optimizations the software tries to make?

* How many tests can one do in a given time period, without confounding test & control variants? It seems like they would take a while to run?

* Is there a good way to control for non-technical/non-content-based changes (e.g. external, links)? For example we get hundreds of negative links pointed at us per week. Do we just hope/assume that's not the cause of rank changes?

~~~
dillonforrest
Hey there!

We test everything on-page. This usually includes CTR stuff, like titles and
meta descriptions and paragraph text that Google extracts for the meta
descript, headers, images, calls to action, even conversion rates.

Correct, we continually iterate based on feedback like rankings and clicks,
and we're working on smarter ways to monitor Google's crawler too!

Users are as hands-on or hands-off as they'd like. We usually own the
experiments, but some of our customers design their own too.

The number of tests depends on the number of pages and your traffic. Our
largest customers have full factorial experiments with thousands of concurrent
split tests every day.

------
jitbit
Guys, The idea looks great. Congrats on the launch.

2 SEO questions:

1) a CDN means "thousands of websites on one IP address". Google doesn't like
it _UNTIL_ it knows, the IP belongs to a well-known CDN like
cloundfront/cloudflare etc. Please comment?

2) An A/B test might look like "cloaking" to googles. How exactly do you run
it? I assume, not in parallel, but "variation A _then_ variation B" \-
correct? If yes, does it mean I have to leave the website UNTOUCHED for 21
days so the test results are not distorted by my other activities? (adding new
content, internal links etc)

~~~
ryanb
Thanks! Great questions

1) We actually work with CloudFlare/CloudFront etc if companies are already
using them. In that case, we sit in between CloudFlare and your origin
webserver. But if you don't already have a CDN in front of your website, our
SEO CDN goes in front of your origin webserver, and we use AWS so as far as
Google is concerned it looks like AWS. We also use different IPs for every
customer.

2) We don't do cloaking - we're essentially testing somewhat small tweaks
across groups of similar pages, and seeing if they have any impact on clicks
or rankings. It is preferred if the pages remain largely the same, but they
don't have to be totally untouched.

~~~
jitbit
Thanks for the answer.

PS. whoa, exciting (the 'in-between cloudflare and your server' bit)

------
sgslo
Looks great, congrats on the launch!

Two points of feedback:

(1) The idea of the product is clearly conveyed, but I'm confused on exactly
how it works. The landing page mentions that title tags, headlines, meta-tags,
etc get tweaked - exactly how is this done? Do I have to manually enter a
bunch of alternative text, or are you using a big fancy thesaurus to switch
out some key terms?

(2) How do you evaluate performance of the product? Solely through click
rates, or by search rankings? How often do google search results get updated?
In short, how do I know the product is working?

~~~
dillonforrest
Thanks sgslo!

(1) We use both humans and software to generate experiments. For customers,
it's completely automated.

(2) We look at all primary search metrics (clicks, impressions, CTR, and
rankings), with clicks being our main metric. Search results get updated at a
pace determined by Google's crawl rate, which varies per site depending on
multiple factors including domain authority. We use bayesian structural time
series and negative binomial regression models to measure impact and
statistical significance to power our data-driven SEO recommendations.

------
birken
As somebody who wrote an article about A/B testing title tags in 2011 before
it was cool [1], this is an awesome idea. I've talked about SEO with many
companies and coming up with the proper title tags and meta descriptions alone
is often worth so much traffic for such little effort (once you get past the
upfront cost of running the tests).

However, I think a critical aspect of SEO is thinking about an entire site
holistically. Not only because certain signals are site-wide, but because a
key aspect of SEO is deciding what pages of your website are "good" for SEO
and which ones aren't, and then focusing on making the "good" pages better and
not worrying about the "bad" pages. Good and bad in quotes because it is often
quite a bit of an art and not a science.

How does RankScience play into this? You've nailed the on-page stuff but is
there any world in which RankScience is able to talk about a site holistically
and recommend which types of pages and content seem to be working most
effectively (and maybe even suggesting pages that could be removed/de-
indexed)? Or do you leave that to SEO consultants and you just nail the hell
out of the on-page stuff.

1: [https://www.thumbtack.com/engineering/seo-tip-titles-
matter-...](https://www.thumbtack.com/engineering/seo-tip-titles-matter-
probably-more-than-you-think/)

~~~
dillonforrest
It's a true honor to speak with an SEO A/B testing OG! :)

> How does RankScience play into this?

That's a really great point. Right now, our software either focuses on only
pages which are good for SEO, or it runs less frequent experiments on the
"bad" pages. We've had a few success stories of turning "bad" pages into good
ones and having them become big revenue generators for our customers, but
that's not common as I'm sure you can imagine.

------
BickNowstrom
Though using A/B testing to improve user experience or conversion rates is
fine, I thought using A/B testing to reverse engineer the ranking algorithm
was against the guidelines. Has this been updated?

From
[https://support.google.com/webmasters/answer/7238431?hl=en](https://support.google.com/webmasters/answer/7238431?hl=en)

> Best practices for website testing with Google Search

> The amount of time required for a reliable test will vary depending on
> factors like your conversion rates, and how much traffic your website gets;
> a good testing tool should tell you when you’ve gathered enough data to draw
> a reliable conclusion. Once you’ve concluded the test, you should update
> your site with the desired content variation(s) and remove all elements of
> the test as soon as possible, such as alternate URLs or testing scripts and
> markup. If we discover a site running an experiment for an unnecessarily
> long time, we may interpret this as an attempt to deceive search engines and
> take action accordingly. This is especially true if you’re serving one
> content variant to a large percentage of your users.

Next to this, the advice is to use rel="canonical" to avoid duplicate issues
with Googlebot crawling your variations. When using rel="canonical" this
should not show you how a variation influences ranking.

> If you’re running an A/B test with multiple URLs, you can use the
> rel=“canonical” link attribute on all of your alternate URLs to indicate
> that the original URL is the preferred version. We recommend using
> rel=“canonical” rather than a noindex meta tag because it more closely
> matches your intent in this situation.

~~~
ryanb
This is link is related to Conversation Rate Optimization testing (like
Optimizely). We don't do A/B testing on single pages, or do cloaking or
anything of the sort, but we run experiments across groups of URLs, and then
we sum up the results and run our analysis.

Also, our goal is not to deceive Google in anyway - a lot of our tests are
related to increasing CTR (which is testing humans) and on-page times. (again
testing humans) Overall we're trying to make pages better according to Google
guidelines -- which leads to a better experience for users.

Some more explanation of how it works here: [https://www.rankscience.com/how-
it-works](https://www.rankscience.com/how-it-works)

~~~
BickNowstrom
My post was in response to the homepage copy:

> RankScience sits next to your website, making thousands of experiments to
> tweak your HTML in order to improve your page rankings.

This seems to me an attempt at reverse engineering the ranking algorithm. Is
my interpretation correct? And if so, is this allowed / in scope of the
guidelines?

~~~
ryanb
We're not reverse engineering Google's ranking algorithm, so no. : ) Good luck
to anyone trying to do that! (I'd recommend against it) Yes, you're allowed to
make changes to your pages.

~~~
BickNowstrom
Thanks! Though my (dated?) concern is probably not all that common, you could
think of adding something like "Split-testing to improve SEO is a perfectly
legitimate marketing technique" in your FAQ/technical copy.

~~~
dillonforrest
Thanks for your feedback! That's a pretty good idea!

------
7cupsoftea
Just want to chime in here and share that I'm a very happy customer. Ryan,
Dillon, and Chad are some super smart guys who deeply understand SEO. I use
RankScience for 7 Cups and Edvance360 and I've seen a huge ROI. They always
find time to meet with me and my other team members. The advice and feedback
they provide is worth the cost of the service alone. Highly, highly, highly
recommended!

~~~
dillonforrest
Thanks Glen!! :)

~~~
chadburgess
yaya, thanks!

------
killion
We've been using RankScience for months at Suiteness. They've given our
indexed pages a consistent boost every week.

~~~
dillonforrest
Thanks Kyle! :)

------
dillonforrest
Co-founder/CTO here. Please share any SEO questions!

~~~
saycheese
Why have you taken this approach to monetizing this SEO optimization method,
instead of other options, for example PPC arbitrage?

Why A/B testing and not a more complex statistical hypothesis testing method?

~~~
ryanb
The biggest reason is that there's so much more volume in organic search (80%
of all clicks) vs paid search (20%), and the ROI on SEO is so high. There's
also so many companies in the PPC space -- it's quite saturated. SEO is
appealing to us because we think it's overlooked due to stigma.

SEO and PPC are really two very different games overall, but I have a ton of
respect for people who are really good at PPC.

~~~
saycheese
Thanks, just to be clear, I mean you generate the clicks via SEO and selling
them on a PPC, lead, etc. basis.

~~~
ryanb
Oh, I misunderstood. Good question. A lot of people have suggested this. The
main reason we've avoided it is complexity, but it's something we might
revisit. I'd also be concerned about customers not having an easy way to
understand our pricing.

~~~
saycheese
>> "I'd also be concerned about customers not having an easy way to understand
our pricing."

That would likely to be best expressed relative to major PPC players like
Google.

For example, "clicks are 20% cheaper than Google for the same keywords."

Anyway, just nitpicking, awesome solution to a big problem. Good luck!

~~~
ryanb
I think this is a super compelling idea. : ) Thanks!

------
hartator
Congrats on the launch.

How do you check the effectiness of a SEO change? You check for Google
rankings or traffic?

~~~
ryanb
We look at Clicks, Rankings and Impressions. We use Google Analytics,
Webmaster Tools, and another third party data source.

More clicks is obviously the ultimate goal, but sometimes we also have
significant experiments where rankings or impressions going up are the key
metric

~~~
hartator
by curiosity, what's the thrid party data source?

~~~
ryanb
That's part of our secret sauce. :)

------
franciscassel
I consult with startups and tech companies (recently: SurveyMonkey) on SEO, so
this is super-interesting... especially the automated aspect, which is pretty
novel in this space.

What are some of the specific types of automated tests that you run?

~~~
ryanb
Here's a case study with a simple example of a test:
[https://www.rankscience.com/coderwall-seo-split-
test](https://www.rankscience.com/coderwall-seo-split-test)

* CTR on titles/meta descriptions * Any HTML changes * Design changes * Time on site

~~~
franciscassel
Thanks for the reply. Would you mind being more specific about what specific
aspects of the "automated" tests are "automated"?

E.g., in the CoderWall example, is it propagating of the title tag change to
all the pages in the test group that's "automated"?

At least to my ears, "automated" suggests that there are tests that are
selected and run completely without human intervention. (Which is hard to
imagine in the SEO space.) Is that in any way accurate?

~~~
dillonforrest
Correct! Our customers don't have to lift a finger, and our software
continually iterates on their SEO.

------
gukov
It seems like you integrate with the likes of Google (Webmaster Tools) and
Cloudflare in ways not done before, so my assumptions here are probably a bit
outdated.

Let's say you want to test a new title. You collect stats for a week, change
the title, collect stats for another week. The two datasets are then compared.
Am I close?

Does it mean that you can't AB test in parallel? If so, it's not optimal for
time sensitive stuff like breaking news.

~~~
ryanb
We don't run experiments on a single page, but instead run them across groups
of similar pages. The easiest way to think about this is running a test across
a "product" template on an e-commerce site. A site could have thousands of
product pages, but they're all on the same template. We would grab a set of a
few thousand product pages, and split them into control and variant groups,
execute the change on the variant group, and monitor how Google reacts to the
groups in aggregate over a period of time.

Depending on how many pages you have, we can run many tests in parallel.

~~~
gukov
Gotcha

Thanks

------
coolswan
Thanks for the AMA! What's the best way to deal with Google algorithm changes?
Also, at what point does site performance actually impact my SEO?

~~~
ryanb
The only way to deal with Google algorithm updates is through experimentation.
A data-driven approach is the only approach. : ) Staying abreast of what both
Google is saying publicly and what the SEO community is saying also helps (ie
Google announced they're cracking down on intrusive pop-ups for mobile sites)

Site performance is always relative to your competitors. Optimal server
response time is around 200ms, and that's what folks should strive for, but
I've seen sites have really slow pages and still get lots of traffic.

------
0bfus
You guys mention < 25ms as the performance hit of having you guys in front of
the client's application. Is there a more concrete SLA you guys provide? Do
you have a sense of what that hit might cost in page rank (there's some
theories out there that time to first byte is one of the features that search
engines reward)?

~~~
dillonforrest
Hey 0bfus, we're working on a fancy performance metrics dashboard and status
page to answer these questions! Also, we've done extensive research on
pageload speed's effect on page rank, and we found that <25ms of incremental
latency does not have any impact. We're working on a case study for the
latter!

------
ijafri
I am bit wary of the idea, despite the fair intentions, but Google in
particular tends to view anything remotely close to 'improving' search ranking
using 3rd party 'software', as a threat to their own Ego-rithms.

------
sixQuarks
I wish I could use a tool like this and know that it won't affect my rankings
negatively, but I feel that Google will eventually punish sites that do a lot
of A/B testing.

------
Mizza
How does this compare with the other offerings in the space?

~~~
dillonforrest
We focus on being automated! Our software never stops working to grow your
search traffic, even when you're asleep or if you don't want to think about
SEO at all.

------
gargarplex
What's the minimum amount of traffic to benefit from this offering. At what
point should a startup seriously consider working with you

~~~
ryanb
No real minimum - it's all case by case.

~~~
gargarplex
Would you consider donating some services to Zidisha (YC W14)? If yes, please
email me – zack@zackburt.com. Thanks.

------
jjyoung135
When you are just starting out with a website, what is the 20% of effort that
leads to 80% of the results in regard to SEO?

~~~
dillonforrest
Hey there! I know we just chatted on facebook about this, but just in case
anybody else is wondering: we're working on this post. Stay tuned!

------
ortekk
That looks great! How do you automate google's rank checking? Do you use
proxies/swarm of VPS's?

~~~
ryanb
We use Google Analytics, Google Search Console, and third party services to
supplement keyword rankings data -- we don't crawl Google SERPs. :) Plenty of
other companies do, though. Moz has great keyword ranking data.

------
omarchowdhury
How does your pricing work?

~~~
ryanb
It's based on how much traffic goes through our CDN. If you signup on our site
or send me a note directly ryan@rankscience.com I can give you more details

------
TomAnthony
Hey guys! Congratulations on the launch! Very exciting!

We've been seeing some great results with SEO testing, and have recently had
our biggest test result (in terms of revenue impact). Looking forward to
hearing more of what you guys are up to. :)

I think the fact that DistilledODN, RankScience, Etsy, and Pinterest have all
published SEO split-test results recently demonstrates the importance of this
type of data-driven approach to SEO!

Best of luck with everything!

Tom, Distilled (Disclaimer: I run the
[https://www.distilledodn.com/](https://www.distilledodn.com/) team)

~~~
ryanb
Thanks Tom. Would love to buy you guys a beer sometime. : )

