I count the name of one drug repeated over 100+ times.
I've been quite clear that there's nothing wrong with A/B testing. In fact, less than a month ago I tweeted that "A/B testing can be really helpful" and included a link to best practices: https://twitter.com/#!/mattcutts/statuses/191658511149711360
Looks like the subdomain you referenced did have a hack, but the main site didn't. I had done a Google fetchbot request to check before hand. Regardless, I can see how having hacked subdomains might trigger the warning.
Based on your comment, should we understand that split testing, where it actually redirects to a few different separate pages, is an okay form of testing that won't trigger the "this site was compromised" warning?
If you want low-level/detailed info, we discuss this topic on our support page at http://support.google.com/websiteoptimizer/bin/answer.py?hl=... . Hope that helps.
PS: Thanks again for your help on this.
- some pages on the domain were definitely hacked across multiple pages for weeks.
- we detected hacked pages on the site both manually and algorithmically based on our hacked site classifier.
- we sent a message via the webmaster console on May 5th to alert the site owner so they'd have a heads-up.
- it looks like this has nothing at all to do with A/B testing (which as I've said before is perfectly fine to do).
That's what I know with near 100% confidence.
Here's what I believe, but won't be able to confirm until tomorrow when I can talk to a few folks. I think the domain was hacked in multiple ways. I found the hacked pages on model1.webdesigncompany.net just from doing site: searches. And you're right that those are auxiliary pages, not related to the main part of the site. But I suspect the core site was also hacked, based on looking at the manual action that was submitted. I'll be happy to talk to the relevant webspam people to ask for more details.
One thing that rang alarm bells that you and readers should be aware about is to do with this statement:
If you are not randomly assigning visitors to a group and instead just putting odds and evens in separate groups when you run more than one test at once you are will synchronize those tests and generate incorrect results.
Also why did you segment the traffic during the test (ignoring traffic that didn't match you criteria) instead of doing it at the analysis stage? Is this a limitation of VWO?
I haven't used VWO rather I have rolled my solutions and I found segmenting traffic during analysis allowed for a great deal of discovery and indications for further testing. What if one of your test pages caused people who searched using a phrase that didn't indicate an intention to buy converted those users better?
Moving forward, I'll do the extra work to make VWO swap content on the same page and adjust the css appropriately.
Segmenting traffic after the test is something I wish VWO would allow me to do but that is currently a limitation of their system.
This not only has the advantage mentioned where you limit your regret for making a bad decision on too little evidence, but with some modifications to the algorithm (such as discounting old evidence from your inference) you can also deal with situations where the world in which you are running the different variations is itself changing over time, or where different types of users coming from different browsers may have different behaviors.
From the perspective of someone searching for your site, however, the results might be nearly identical: even if you have massive changes to the phrasing (having versions written in various regional dialects of English, for example), the resulting page will still largely be identical, possibly sentence-for-sentence (though not word-for-word). In this case, you will still want to get one of those variations indexed, and it might not matter which one.
As described, of course, this is both "cloaking" and not: the site is not treating GoogleBot differently than any other visiter on purpose, it is simply going to end up doing that because GoogleBot has different behavior on the site than a normal user, so it will learn and optimize (possibly somewhat randomly or uselessly) from this and end up treating that user differently than one coming from Internet Explorer (which may very well represent a different demographic of user).
It sadly does not seem like this advanced usage of testing is allowed by Google.
Hell, I myself am guilty of not bothering to type in URLs anymore (yes, just like those poor people who used to search "Facebook login" and one day got highly confused when an article about logging in to Facebook that happened to support Facebook Connect to leave comments ranked higher): I just search for things like "cdnetworks portal" and "1and1 client login"; I even, and I shit you not, do a Google search using the Chrome OmniBar for "google images", as I don't remember what the URL for that part of Google is.
[edit: As adgar correctly points out below, the following paragraph is a misunderstanding of the mechanism that was used by this website to implement this split-test algorithm. However, it seems fairly obvious to me that the mechanism used to implement the split test does not matter. Meanwhile, the reason I went in this direction is due to Matt Cutts specifically stating the trade-off in the below paragraph on the Google webmaster video series; it is not because I assumed something from this article's conclusions.]
It is /exceptionally/ irritating as their rules may have some (highly arguable) philosophical purity, but in the majority of cases leads to a /worse result/. For example, it would be /much more correct/ for sites like Hacker News to mark that the comment/title at the top of each page "is what search engines are allowed to see and index" and that the rest of the comments below it are "ancillary content that absolutely must not be indexed". Otherwise, when you search for a comment using Google, you find every single point along the tree that connects from that comment up to the root of the page, as the comment is present on all of them.
I found myself thinking about these issues a lot recently while working on writing some custom forum software, and even went and skimmed through every single Google web masters video that Matt Cutts put out, and the end result simply made me angry: I was finding myself purposely designing worse things so that they could be "indexed better", and when I'd look at what I was doing and go "this is nuts: is Google really that important?" I'd have to sigh and sadly remind myself "yes, it probably is".
Halfway through your rant, you clearly demonstrate that you did not bother to adequately understand the linked article before ranting about Google, throwing words around like "evil" and "despicable" without even knowing what you are ranting about, like so very many well-intentioned but woefully ignorant commenters in today's technorati.
FTA, emphasis mine:
You seem to have completely misunderstood what the author was saying. Naturally, since you are both misinformed and critical of Google, that makes you the most highly upvoted comment on today's Google Hacker News thread!
^ This is the reason why I went in that direction, by the way. This is a video from Matt Cutts, one of the many that I sat through watching during a massive five-hour- MattCutts-and-GoogleWebmasterHelp -a-thon that I force myself through a week or two ago. I am hotlinking to 3:23 into the video as this video answers multiple questions (as is common on the MattCutts channel, as opposed to on GoogleWebmasterHelp, where each question tends to get its own video).
In this question's answer, it is clearly stated that Google may very well consider the same page returning different loads of the page for purposes of A/B testing as "cloaking", and that webmasters should instead use client-side mechanisms to perform these tests. If tests /are/ done, it is claimed that the webmaster should only do so on areas of the site that are not being indexed (which may involve explicitly telling Google to stop indexing that part of your site).
Both Google's action and Google's inaction "make rules for the Internet". In an alternate universe where Google didn't implement this penalty, attacks using this kind of client-side redirection could easily be common and a serious problem. In that universe, alternate-saurik could come to Hacker News and complain that "Google is evil and despicable for not clamping down on these client-side hacks that are almost always used by attackers, not legitimate developers."
Google has to make choices. Those choices are going to feel like "rules for the Internet" for many people they influence. That's unavoidable. If you're going to criticize Google for this sort of issue, you need to focus on the details of those choices. In this case, your lack of attention to detail makes it clear you're not making a credible case.
I can even, if you demand, find other videos from Google (also from Matt Cutts) that show that the heavily-hedged suggestion in this video to serve different content from the server (even small snippets of text, such as titles or breadcrumbs) that is not based on IP address (and thereby might not be stable for GoogleBot) is also highly dangerous and can lead to Google believing that you are cloaking.
I therefore take issue with the fact that, once I misunderstood this specific article, that somehow means that all of the other research that I've done on this matter, and even the arguments that I state in my rant (where the specific paragraph that is currently under contention I am backing up with evidence outside of this single article, which is from someone none of us have ever heard of and for all we know could be lying about what Google did anyway) are now void.
I should probably make a new video or blog post about A/B testing, but I can try to summarize here. In the original video, I said:
- it's nice if you can A/B test on a part of your site that Google doesn't see. Then it's a moot point for us.
- failing that, it's helpful to do something server-side.
- don't do anything special for Googlebot.
The state of A/B testing has evolved quite a bit in the last six years though. If I were making a fresh video, I'd say: A/B testing is a perfectly fine thing to do. You can do A/B testing via either server-side or client-side technology. In both cases, don't do anything special or different for Googlebot. Treat Googlebot just like any other user and don't hard-code our user-agent or IP address. If you have any other questions, refer to http://support.google.com/websiteoptimizer/bin/answer.py?hl=... or http://www.youtube.com/watch?v=QHtnfOgp65Q to review best practices and avoid actions that search engines could potentially view as cloaking.