One wonders if there's an "advantage" you could gain from this.. If your site is getting requested by AVG but NOT a valid browser, you can assume someone did a search that you appeared on but they did NOT click on you. This metric could be useful for improving your CTR in regular search results, something totally impossible to measure before.
I'm actually building a web app with a small start up, and we just do not have the time to make our service ie6 compatible. We are going to let users know that and redirect them to a very plane jane site where you can still get the information but not very efficiently. However our market is mainly tech savvy people so it should work out.
Sorry, that widget just gives that little dropdown redirecting to their site? I meant that I was going to just provide an info page where the user can still get the information on my site, just not in the same flashy page as everyone else.
Fair enough, but sometimes tech savvy people use computers of tech unsavvy people (friends and relatives, internet cafés, corporations with reactionary IT departments...)
AVG is an antivirus. When users install it, they give it full privileges. A program like AVG Antivirus runs on many computers. Ofcourse this gives AVG enormous power to affect ad market, not only will CTRs go down, but also if AVG is corrupt, they can make auto clickers or have rival's ad income down by a great extent, or eat up adsense impressions of rivals.
I wonder why didn't anyone think before implementing this.
User-Agent information is inherently unreliable. I don't see why people are upset when a misguided practice that already provides false results with an unknown margin of error gets even worse. If they insist on using such a metric then it's their responsibility to assume the risks that it incurs.
Google / Yahoo do this as well to check up on sites to make sure whats served to their bots is the same as whats served to real users. Google has 3 or 4 different fake user agent strings they use for the googlebot
Google doing it isn't nearly the same as 10 million users doing it. In fact, this type of activity should be limited to search engines, not client-side apps.
I think the solution should be fairly simple. Since AVG is not a real browser, despite telling a web server that it is, it should be easy to craft a piece of Javascript and cookie on a webpage that would return a proper result in a real browser like IE, Firefox, Safari, or Opera. If a proper result is not returned, then the web server can just do a 302 meta redirect to AVG's website as mentioned earlier. If implemented widely on web servers (say as an apache module or PHP patch) this, in effect, would cause AVG to become it's very own DDoS attacker. I don't think they would want this, but it'll probably happen if they don't re-think the way they do things.
How would that work? The AVG robot probably only fetches the page's URL. By the time you serve up a JavaScript, you've already lost the ability to redirect the robot.
Also, once enough people start redirecting the AVG robot to the AVG website, they'll just update the robot to ignore those redirects. It's antivirus software so it gets updated very often.
I guess it could be done using some combination of first placing into every HTTP header: "Refresh: 2;URL=htpp://www.grisoft.com/", which would then allow a piece of Javascript on the web page up to 2 seconds to figure out if the visitor is a real browser (maybe even including some DOM checks too), and then acting accordingly (ie: sending the user to a page without the redirect header). If AVG's http fetcher doesn't execute the Javascript or fails to pass the browser test, then it gets redirected back to AVG's site because of the HTTP header.
Convoluted? Yes. Will it work? It should, unless AVG decides to implement a full Javascript engine and DOM stack in their http fetcher. Also, the initial re-direct doesn't have to go to Grisoft, it could go each time to a randomly selected site from a list of their competitors <evil grin>.
I think AVG is really playing with fire, it's really only a matter of time before things like this start popping up as defense mechanisms.
Presumably, AVG loads extra resources like images, CSS, iframes, <object>s etc. from your page. (if it didn't, it wouldn't be very good at finding malware, and be really easy to detect)
There aren't any. The User-Agent header has always lied, at least a little bit. What AVG wants from your site is the same thing that the users do: the content (in this case, so that they can scan it). If you insist on giving them a different page (like an error page) than a user would see, they're going to take steps to prevent you from doing that.
Remember that the desired outcome here is to see fewer AVG hits, not to "screen them out" just because they're not a human being (although they are a necessary condition for a human hit coming later). Try reducing the unique URL count on your site (basically: make it more RESTful and eliminate all the query strings on broadly identical content) so they have fewer URLs to crawl. Set your caching headers properly so that they don't have an excuse to recrawl you too often. That won't force them to hit you less, but it will "help them to help you" by reducing the total bandwidth. Remember they don't want to pull any more than they have to either.
If AVG didn't have their head up their asses, yes. Though that would make the problem for webmasters even harder. What really chews my liver is that they didn't even do it right.
