Does DuckDuckGo respect `robots.txt` files with onerous implicit restrictions like that? If I were them, I'd probably just write it so that if Googlebot is allowed then DuckDuckBot is too.
It's a dark pattern to limit access/force app downloads for access to content on mobile. I hate rooting for google, but I'll root for open web any day.
I'm totally open to being convinced, but I'm skeptical right now that they are in fact that shady. I'm more inclined to believe that it's a placebo effect from suspicious people suddenly paying more attention to ratings after a sales call, and also attributing any negative dip in rating to the phone call they recently had. There are also third-party groups who call claiming to be able to list your business higher or rate it better, and I'm suspicious that some people confound these callers as yelp representatives.
And linked from elsewhere in the thread: http://www.latimes.com/business/la-fi-yelp-ratings-20140905-... -- it just seems like with no scientific study backing the claims and case after case being lost, I think the onus of proof is on the people accusing yelp of these threats.
You ask what would proving it accomplish? It would allow everyone to actually know what’s going on instead of stringing together a bunch of conjecture and anecdotes. It would enable change.
These stories are always about "friends" or "relatives", but nobody ever provides the link to the evidence, despite that being the easiest thing in the world to provide.
One wonders how profitable it'd be for Yelp to call small, local businesses at scale for shakedown, considering what's on the line reputationally if one or two people know how to record calls.
Gather a paper trails. For starters, screenshots of the page + emails and communication with the company.
The reality is that its likely explainable through a combination of disgruntled business owners and overzealous/evil salespeople who make threats they can't follow through on, just to get sales.
(Though, some of the admin panels there have (or had, when I was there) a easy to use "Make user Yelp Elite" button, but I'm sure you'd get caught :)
I totally believe that there are some shady salespeople at Yelp that talk about... the implications. But I doubt they actually do sell improved ratings.
We do know. Just most people refuse to believe until it happens to them, so Yelp continues to get away with it.
It seems like it would be an easy experiment for somebody to do. Find a local restaurant with some poor ratings. If they want to cooperate, pay a Yelp salesperson to "fix" the problem and see if it goes away. If they deliver, then you have a bombshell article that could go viral.
I haven't evaluated it so it is possible that the study has serious flaws in its methodology.
Basically, they couldn't find evidence, but they can't disprove the hypothesis and they can only tell from the year 2012. Thus, they can't see if stores let their advertising expire before that year or see the reviews stores had before advertising if they started before that.
Side note, Yelp literally called my partner who ran PR for non profits art org and asked to be paid $$ or else competitors ads would run on their page. Literal words. One would pay for NOTHING except not having competitors getting promoted on your own pages.
Partner laughed and hung up.
Google Maps. It also asks questions sometimes like "Does this place only accept cash?" or "Is this place wheelchair accessible?"
I think recently they even added a feature that lets people ask their own questions which then get answered by people who recently visited the place in question.
There is some fun stuff in there, like this?!
Also appears they block archive.is outright (which ignores robots.txt).
If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.
(In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)
I'm not allowed to look in /documents/source/? Perfect. Let's start there.
They might also not be looking at the file, and just appending random words to the end of Yelp's biz URLs to scrape every business. Which, at some point, it might hit that URL by accident, since they are all words you'd find in other business URLs. Though this seems less likely.
Love Google and Foursquare.