Hacker News new | past | comments | ask | show | jobs | submit login
Yelp Claims Google Broke Promise Made to Regulators (wsj.com)
99 points by IBM on Sept 12, 2017 | hide | past | favorite | 70 comments

Don't know if Yelp does allow any search engines to scrape, but if they'd allow duckduckgo to scrape and help build some marginal competition against Google, it might give them more leverage overall.

Looking at their robots.txt, they allow Google but not DDG.

Huh... Looks like they have a whitelist that allows Bing, Google, and Yandex (among others), but not DuckDuckGo: https://yelp.com/robots.txt

Does DuckDuckGo respect `robots.txt` files with onerous implicit restrictions like that? If I were them, I'd probably just write it so that if Googlebot is allowed then DuckDuckBot is too.

Used to like Yelp as an alternative to Google owning everything, but they are destroying their brand using shady app-hijacking tactics.

It's a dark pattern to limit access/force app downloads for access to content on mobile. I hate rooting for google, but I'll root for open web any day.

Heh, you should hear their shady business tactics. It would be a shame if your business suddenly had 1 star on yelp.

I'm unconvinced. I have read a ton of anecdotes and 2nd-hand accounts about this, but I've never seen any experiment or any piece of hard proof that yelp or its representatives hold your rating over your head if you don't sign up for advertising. It seems like an easy scenario to test out.

I'm totally open to being convinced, but I'm skeptical right now that they are in fact that shady. I'm more inclined to believe that it's a placebo effect from suspicious people suddenly paying more attention to ratings after a sales call, and also attributing any negative dip in rating to the phone call they recently had. There are also third-party groups who call claiming to be able to list your business higher or rate it better, and I'm suspicious that some people confound these callers as yelp representatives.

And linked from elsewhere in the thread: http://www.latimes.com/business/la-fi-yelp-ratings-20140905-... -- it just seems like with no scientific study backing the claims and case after case being lost, I think the onus of proof is on the people accusing yelp of these threats.

I've seen it first hand. My friend is a veternarian who was working in a very expensive neighborhood in California. Every time he got a 5 star review, it wouldn't display on his Yelp page. A few 1 star reviews did appear though. Yelp would call him and tell him essentially the 5 stars would reappear if he advertised with them. He had no choice but to pay and they all appeared, driving his average back up. How would you propose to prove it? And what would proving accomplish since many folks already believe it?

You say you have seen it first hand? Isn’t that second hand knowledge your describing?

You ask what would proving it accomplish? It would allow everyone to actually know what’s going on instead of stringing together a bunch of conjecture and anecdotes. It would enable change.

It's trivially easy: link to the page. Put up or shut up. If what you're saying is true, it will be obvious.

These stories are always about "friends" or "relatives", but nobody ever provides the link to the evidence, despite that being the easiest thing in the world to provide.

Specifically, Internet Archive links that show the before/after situation, with reviews appearing that were made before "before" but only appear in "after".

Do you know for sure it was Yelp calling and not some third-party scammer? I have heard there are companies that pull this, and some might pretend to be Yelp but in fact they just create fake reviews and them delete/change them later. (I presume this only works if the target pays "Yelp" over the phone). I am not sure what to believe but like your parent xommentor I am skeptical.

Given the scale at which phone scammers operate these days, and the fact that many of the scammed business owners are likely not savvy to this, it's plausible.

One wonders how profitable it'd be for Yelp to call small, local businesses at scale for shakedown, considering what's on the line reputationally if one or two people know how to record calls.

It's illegal in a lot of states to record calls without notifying the other party.

>>> How would you propose to prove it? And what would proving accomplish since many folks already believe it?

Gather a paper trails. For starters, screenshots of the page + emails and communication with the company.

Additionally, you could literally ask anyone who worked at Yelp. I have. Sure not everyone there has seen every bit of code that powers their service, or every business process, but if they were systemically encouraging mobster-like business tactics, someone would have heard about it. I never did. I heard a lot of engineers and sales people laughing about the fact that people thought they did.

The reality is that its likely explainable through a combination of disgruntled business owners and overzealous/evil salespeople who make threats they can't follow through on, just to get sales.

(Though, some of the admin panels there have (or had, when I was there) a easy to use "Make user Yelp Elite" button, but I'm sure you'd get caught :)

Enough people have left Yelp that if there were black programs in place, we would know.

I totally believe that there are some shady salespeople at Yelp that talk about... the implications. But I doubt they actually do sell improved ratings.

>we would know

We do know. Just most people refuse to believe until it happens to them, so Yelp continues to get away with it.

Where are all the ex-employees and leakers exposing the shenanigans? I do believe they have salespeople that suggest paying to fix things, but I don't believe they can actually do that.

It seems like it would be an easy experiment for somebody to do. Find a local restaurant with some poor ratings. If they want to cooperate, pay a Yelp salesperson to "fix" the problem and see if it goes away. If they deliver, then you have a bombshell article that could go viral.

Here are some first hand accounts from the trailer for Billion Dollar Bully


This study done by a prominent university claims that the Yelp extortion allegations don't hold up.

I haven't evaluated it so it is possible that the study has serious flaws in its methodology.


Seems that they attempted to, but they can't definitively disprove it.

Basically, they couldn't find evidence, but they can't disprove the hypothesis and they can only tell from the year 2012. Thus, they can't see if stores let their advertising expire before that year or see the reviews stores had before advertising if they started before that.

Yes, all scientific studies have limits to their scope. That is tautological.

This is not about whether Yelp is good or bad. This is about Google and their abuse of power.

Hmm these comments really bother me - people seem to miss the point that Google is using its monopoly power to block out competition. This concept is very, very frightening and we should all be worried. This is no different than when railroads would block competition and promote their own products.

Doesn't Yelp have a reputation for much shadier business practices (i.e. extortion)? Maybe they're not a very sympathetic underdog in this fight.

Yea perhaps - I'm not really defending Yelp as much as I'm worried about Google. I think this extends beyond just Yelp into the wider market (hotels.com, tripadvisor, paid search results, etc).

Yelp is definitely not a sympathetic party, but that doesn't invalidate or weaken their claim. Particularly if, as they claim, Google is specifically violating an agreement with the FTC.

Yelp is indeed sht company and needs to be investigated for its own shakedown techniques. But this doesn't mean that Google isn't behaving in an anti-competitive way.

Side note, Yelp literally called my partner who ran PR for non profits art org and asked to be paid $$ or else competitors ads would run on their page. Literal words. One would pay for NOTHING except not having competitors getting promoted on your own pages.

Partner laughed and hung up.

I'm not a Yelp sympathizer, but this does pretty clearly look like a violation on Google's side. Building up local content is hard and expensive and to represent someone else's content as their own is pretty unethical. If Google wants to present Yelp's content, they need to come up with a deal or grow their own photography content the hard way.

FWIW, Google has been building their own content library of photos of local businesses. Android phones these days actually request users share photos they take with Google for inclusion into their Places content. Not sure which app/service is responsible for it, but I hear a lot of people talk about it.

> Not sure which app/service is responsible for it

Google Maps. It also asks questions sometimes like "Does this place only accept cash?" or "Is this place wheelchair accessible?"

I think recently they even added a feature that lets people ask their own questions which then get answered by people who recently visited the place in question.


There is some fun stuff in there, like this?!

        Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0
	Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-1

Supposedly a honeypot: https://news.ycombinator.com/item?id=6573338

Also appears they block archive.is outright (which ignores robots.txt).

So I can load that url on a high-traffic page and IP block all visitors to my page from yelp?

Archive.is is designed that way, though. It performs requests on behalf of a user, acting as a proxy and setting X-Forwarded-For with the user's IP address.

Can anyone explain how that honeypot works? i.e. how would anyone try to go to this page?

To catch and block scripts scraping Yelp in ways they don't like.

If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.

I see. Thanks, but then why so many?

To make it hard for people to write code to get around it.

Well, wouldn't that only be very effective if Yelp update this file every so often? But this line was added as early as 1425 days ago (https://news.ycombinator.com/item?id=6573338).

I assume that it's supposed to be regularly updated, yes.

I wonder if any of their pages has a hidden link to one of those entries in robots.txt

robots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because you have something good there. So, some bad crawlers might go to the robots.txt since it tells you where the good stuff is hidden.

(In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)

I think it's more likely that they have hidden links and references to these URLs scattered across their "normal" pages. A human would never click on them, but any bot that's just blindly following links would stumble across them, which alerts Yelp that it's likely a bot accessing the site.

Be careful w/ these urls - I visited one and it appears my IP is now blocked from accessing yelp.com.

What do you suppose would happen if we all visited these URLs from every public access point we use? (edit: and shared VPNs, Tor, corporate networks, etc.)

That's really interesting. They might be trying to blacklist scrapers that don't properly respect robots.txt files.

Why would a scraper who doesn't respect robots.txt be accessing that file?

If I was building an unruly scraper (which, in case our new overlords are listening, I would never do), I would read robots.txt so that I had a clue where the secret information that the company did not want me to read was located.

I'm not allowed to look in /documents/source/? Perfect. Let's start there.

They might be using it as a way to find specific pages which have content Yelp doesn't want you to scrape. The "evil" scenario.

They might also not be looking at the file, and just appending random words to the end of Yelp's biz URLs to scrape every business. Which, at some point, it might hit that URL by accident, since they are all words you'd find in other business URLs. Though this seems less likely.

I assume the GP was using their browser to access these links, though they still be blacklisted by Yelp?

Scrapers can be pretty shady, so they're isn't a good way to ensure that web traffic is coming from a legitimate human using a browser. To the server it's all just bits on the wire.

I just visited all of them: the IP was not blacklisted

Worked on my laptop, but when I tried the same thing on my phone, I was not blocked. (even when I request the desktop version of the site)

Because they whitelist the exit IP for the phone network, else one person could block access for millions.

And what a great loss that is :-)

They assume the human who created the robot is built to Asimov's law.

Removing Yelp from my phone. Never liked it anyways.

Love Google and Foursquare.

I am guessing that this is going to end up be a PR nightmare for Yelp once Google let's the hounds loose. They have been such an easy target in the past.

Can you elaborate?

Did you know the name Yelp comes from "Yellow Pages"?

Non-paywall link (twitter redirect): https://t.co/5sCeMykOWW

Yelp is a Rival to Google...? This claim surprises me.

Google is so big it's basically a rival to any company that sells computer software or services.

The only rival for business reviews, which is almost exclusively a sub-domain of map apps now. Yelp has the reviews in Apple maps.

Since when is Yelp a Google rival?

Since Google+ reviews show up in search results for things like restaurants.

Google bought Zagat

Yelp and Whiny seem to really rhyme together.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact