
Wait, Google Sent Me - jashkenas
https://chrome.google.com/webstore/detail/wait-google-sent-me/igloabeeeiagmdkhellmibpnlnjdmdbb
======
umurkontaci
I have actually made a bookmarklet that allows you to do the same without
installing an extension.

[https://umur.io/paywall-bypass-bookmarklet-come-from-
google/](https://umur.io/paywall-bypass-bookmarklet-come-from-google/)

Related HN discussion:
[https://news.ycombinator.com/item?id=9134118](https://news.ycombinator.com/item?id=9134118)

~~~
jashkenas
Very nice approach. I didn't know that Google had that auto-redirect feature.

But it's not working reliably for me. Frequently, when I click the
bookmarklet, it gets stuck on a blank page. Does it work every time for you?

~~~
frik
> I didn't know that Google had that auto-redirect feature.

It uses the functionality of the "I feel lucky" button. It will only work if
the site has been indexed by Google, but that's usually the case with news
pages that show such fake paywall messages.

Btw. please Google fix your frontpage, you serve an unformatted text message
(IE9-11):
[http://s7.postimg.org/52fu9wbdn/google_bug.png](http://s7.postimg.org/52fu9wbdn/google_bug.png)

~~~
pbhjpbhj
Is it Google or IE that's broken? Other browsers seem to be coping with it - I
know these big sites tend to serve really mangled markup/code however.

------
jfoster
It would be nice if Google could make some of the Chrome permissions a bit
more fine-grained.

This one needs to "read and change your data on all websites", but with a
better extensions architecture could be something much less scary like "read
and change your data on the current website whenever you click the extension's
icon".

~~~
woodcroft
Alternatively, it would be good if extensions could have a whitelist and/or
blacklist of sites that they are enabled on, with most extensions having a
default blacklist of all financial sites, banks, brokerages, email accounts,
etc. Perhaps also create a way for corporations to blacklist their intranets.
It makes me very nervous that if some extension owner sells out to the wrong
person, suddenly they could see my private information.

~~~
frozenport
Yeah and then you could extort people for money like the AD-Block guy!

~~~
fyrabanks
I haven't used AdBlock for a year or so now, but I'm fairly certain sure his
model was donationware. Did something change recently?

~~~
0942v8653
The AdBlock Plus guy gets paid by companies to put their site on the
"unobtrusive ads" list -- those ads don't get blocked for the majority of
AdBlock Plus users.

------
strommen
Attention, everybody who publishes on the web:

Selectively requiring authentication (based on referrer, user-agent, cookies,
etc.) is _not_ a supported feature of the internet. If you want to go ahead
and implement it anyway, that's fine - but it's pointless to get mad when it
doesn't really work the way you hoped it would.

~~~
wtallis
More generally, everyone who publishes on the web should understand that _user
agent_ is to be taken literally and at face value. The browser does what the
user tells it to do, which may not be what your server asks it to do. A good
browser will do things like lie to your server on behalf of the user when you
try to get it to act counter to the user's interests.

------
jashkenas
Source code:
[https://gist.github.com/jashkenas/9a4d9e648eb11aab394e](https://gist.github.com/jashkenas/9a4d9e648eb11aab394e)

~~~
dreen
Could you give more information on how do some websites react to a google
referer?

~~~
patio11
The design intent is almost certainly using Google's "first click free"
policy, which primarily applies to newspapers like the NYT or WSJ. Google back
in the day struck a deal with the print media: they would remain in Google's
index if and only if clicking from a Google search to their articles
circumvented their paywall on the first click. If they put only partial
content or a full paywall, they'd be delisted. It was not exactly a secret
that this would mean that Google would rank scrapers higher for NYT articles
than they'd rank the NYT.

Thus, when someone posts a paywalled link to HN, someone inevitably posts the
advice "Google the title and click through." (Other than, say, "If you're a
working professional, just buy a WSJ and NYT subscription.")

~~~
colinbartlett
I am a working professional but I have always resisted buying a subscription
because I don't want one single publication to become my de facto news source.

I would love if I could pay $20 to $30 per month to some aggregator that lets
me read articles from Al Jazeera, NY Times, WSJ, BBC, and a selection of other
diverse sources.

~~~
stevesearer
I like Circa News (take my money please!) because they aggregate news from a
variety of sources and present news items in small bit sized chunks listing
facts or statements. They also link to the appropriate sources. edit: and you
can follow news stories to get updates when new information is available.

~~~
aaronbrethorst
They're either shutting down or being acquired soon:
[http://fortune.com/2015/04/30/news-app-circa-
buyer/](http://fortune.com/2015/04/30/news-app-circa-buyer/)

~~~
malnourish
That's a damn shame. Their (android) app has unfortunately become
significantly less useful due to technical reasons lately. I always did wonder
how they would monetize.

------
habosa
Interesting. I posted a Chrome extension that did the same thing a while back
(WSJ specific) and HN unanimously told me I was in the wrong, so I took it
down.

[https://news.ycombinator.com/item?id=8109987](https://news.ycombinator.com/item?id=8109987)

This time the comments seem to be mostly positive.

~~~
eli
If it makes you feel better, I feel that both extensions are wrong. If you
enjoy reading the WSJ, you should pay for it. They do some really great
business reporting and it's not _that_ expensive.

~~~
maratd
> I feel that both extensions are wrong.

I feel that discriminating based on referrer is wrong. Or IP address. Or
geographic location. Or user agent.

More than that, discriminating based on those things is really flimsy.

A website should provide the same content regardless of what browser you're
using or where you're coming from.

~~~
eli
I'm not saying I think the WSJ's paywall is a good idea (I don't), but if they
want to try to make money to support their (largely very good) journalism that
way, I don't think it's very nice for someone else to publish a tool that
automatically subverts the paywall for countless other users.

~~~
bdcravens
This tool saves the 3 seconds it takes to highlight the article title, 'Search
Google for ...', and click the link in the search results.

~~~
eli
I don't follow. Would using the plugin be less acceptable if the paywall were
harder to get around?

~~~
esrauch
Yes, definitely.

------
donohoe
Disappointed in you Jeremy.

Its fair for many here to complain about some of the online advertising
practices of publishers.

However the viable alternative it to pivot to some form of subscription model.
Its a hard transition and many orgs are making risky changes to do better.

Extensions like these are a step backwards.

If you value the work you should support it.

See bio for usual disclaimers

~~~
patcon
Not a fan of the shaming...

If news orgs value their journalists and the future of the profession, they
should experiment with more daring models, rather than ones that simply
maximally enrich the org under the same old model

EDIT: imho a model that falls apart legally under a browser extension that
fits in a 40-line gist is nothing on which to base the livelihoods of a bunch
of people

~~~
donohoe
I agree - but to be fair you're assuming they haven't experimented. Micro-
payments, dumb 'capthcha' based models, memberships, events, "premier access".
All have been tried in various flavors. Some with limited success.

I'm not asking for you to dream up an alternative solution, but don't assume
this is a problem that has not been given a lot of thought by many
organizations.

This is partly why we are stuck with things like "native advertising" and
pretty shit IAB ad units.

~~~
mapgrep
Even people inside the NYT know the pricing for digital subscriptions is
excessive. They're high because circ department won an internal war and
inflated digital prices in a misguided attempt to protect print subs. The NYT
has experimented but is still clinging to the past.

Which is why the "All Digital Access" subscription costs more than Netflix,
Hulu, and HBO Now combined. Give me a break.

~~~
OrwellianChild
This raised an interesting question for me - how much money is saved on a per-
subscriber basis by sending the digital version vs. print (for the NYT)? Their
pricing certainly doesn't reflect it:

    
    
      Web + Smartphone:    $3.75/wk  $16.25/mo  $195/yr
      Web + Tablet:        $5.00/wk  $21.67/mo  $260/yr
      All Digital Access:  $8.75/wk  $37.92/mo  $455/yr
      7-Day Home Delivery: $8.90/wk  $38.57/mo  $463/yr
    

There are multiple levels of insanity here:

    
    
      Web + Smartphone & Web + Tablet are identical except
        for device type but Tablet is 33% more.
      Web + Smartphone plus Web + Tablet cost as much
        *together* as All Digital Access (and are feature
        equivalent).
      7-Day Home Delivery *includes* All Digital Access and
        costs about the same. Paper is free?
    

None of these pricing strategies appear to be designed to maximize readership,
broaden subscription basis, or grow brand penetration. They are structured to
capture the most value possible from customers with high switching costs or
price indifference. This is what a conservator in a dying market looks like.

~~~
untog
The reason that print+web packages don't cost significantly more than web-only
is because print advertising is still a lucrative business. NYT can a a
absolutely justify giving away the paper to subscribers on that basis.

~~~
OrwellianChild
Except, if that were true, why not just make it a free paper? Presumably ad
targeting... I understand the limitations of their existing revenue model. It
just results in some pretty consumer-unfriendly offerings, at the expense of
readership and penetration...

\---

EDIT: Went back to look at the numbers... They have tons of online eyeballs -
those eyeballs just don't make much money. They manage to _double_ their
digital revenue with the paywall. Even so, it's dwarfed by the print revenue.

This may shift as advertisers see the value of targeted online user groups...
I could totally imagine the NYT of the future being a collection of blogs
dealing in Tech, Biz, World News, Local News, etc. Ads on the Biz site would
be targeted differently than on the World News site. This might give
advertisers what they want in terms of a differentiated audience.

\---

From the 2014 Annual Report:

    
    
      Average Paper Circulation in U.S. (Int'l)
        648,900 (219,500) for weekday (Monday to Friday)
      1,185,400 (220,500) for Sunday
    
      Paid Digital Subscribers
      910,000 as of December 28, 2014
    
      Monthly (Annual) Digital Readership
      31MM (372MM) unique desktop/laptop visitors (U.S.)
      42MM (504MM) unique desktop/laptop visitors (WW)
      28MM (336MM) unique mobile device visitors (U.S.)
    

Off these eyeballs, they made:

    
    
      $667.5MM from print subscribers
      $169.3MM from digital subscribers
    
      $483.5MM from print advertising
      $178.8MM from digital advertising

------
DigitalSea
Great idea. This will probably be up there with the invaluable-ness of
Adblock. I have nothing against companies like the Wall Street Journal who
want to charge visitors to see content, but if you are going to paywall
content at least be consistent and charge everyone or don't charge at all. You
can't give it away to visitors from Google and expect people who weren't lucky
enough to come from Google to have to pay. It is unfair and discriminatory in
my opinion.

This kind of reminds me of the aged old trick that people used to use for
Google to get better SEO. Normal visitors would get one version of the site,
the Googlebot would get an over-SEO optimised version of the site stuffed with
keywords and junk (I am talking early 2000's here before Google was as
advanced as it is now) to give the impression of a different site.

Like I said, I think content publishers should be able to charge people for
their content, but you can't have two sets of rules for the same content for
the sake of a better SEO ranking and NOT expect people to exploit that
loophole. What WSJ and other content publishers are doing is a little
borderline dodgy.

I am pretty disappointed in the comments from some people here shaming the
author for creating this plugin. He isn't doing anything that we haven't
already been able to do for a while now. He isn't breaking into servers and
getting the content for free, he is exploiting an SEO trick WSJ and other
sites are using to avoid being penalised in Google. Completely legitimate if
you ask me. If WSJ wanted to truly protect their content behind a paywall,
they would have presented the wall to everyone regardless of their referrer.

~~~
Lorento
You can. It's a form of price discrimination. Like coupons in a supermarket.
People who really want to pay less will jump through the hoop. If you just
want it conveniently and don't care about the cost, you can subsidise those
others. In a way this is more fair - it means people pay closer to what the
money is worth for them.

------
javiercr
IMHO rendering totally different things depending on the http referrer should
be considered as cloaking. It's basically the same practice than rendering a
different thing if the User-Agent matches Googlebot.

I wonder what would happen to your site if you did this without being The New
York Times.

Kudos to Jeremy for this.

~~~
MichaelGG
Google actually encourages this and call it first click free.

------
joshtaku
I think it's funny that Jeremy made this considering his background.

~~~
k-mcgrady
I was curious so I googled him and saw he created Coffeescript, backbone.js
and underscore.js. Is there part of his background I'm missing that makes this
funny. Seriously I don't get it and it's bugging me :)

~~~
joshtaku
The backbone.js stuff was during NYT work if I recall.

~~~
jashkenas
That's not quite correct. Both Backbone.js and Underscore.js were extracted
from DocumentCloud — a nonprofit organization.

But I am currently a Graphics Editor at the NYT...

~~~
alextgordon
Not for much longer, I bet...

~~~
jashkenas
Ha ha... We'll find out soon, won't we.

------
ergest
This is gonna mess up some major web analytics stats about referrers :)

~~~
at-fates-hands
As someone who works with SEO, this was my first reaction. It's going to make
a lot of my analytics wonky and the numbers won't be very accurate.

I guess you have to assume such a small percentage of people will actually be
using this, so the numbers won't be too bad??

~~~
scrollaway
As someone who works with SEO, maybe it's time you learn your entire field of
work depends on the good will of both search engines and users to not put a
screwdriver in the ball you play with.

~~~
at-fates-hands
Actually, I do more development work than SEO, so referring to my "entire
field of work" is a bit. . .well. . .erroneous.

But hey, whatever makes you feel better pal.

------
cgag
I'd rather have a plugin that just ensured I never went to these sites at all.
Or a list to drop in my hosts file.

~~~
dredmorbius
While I don't have a list of paywall sites, I've been compiling one of
Weaponized Viral Clickbait sites generally and the Spartz network (see the New
Yorker's profile of the "Virologist" from a few months back) specifically.

This is in addition to a set of blocklists curated from the uMatrix Chrome
extension.

    
    
        0.0.0.0			knowd.com		# WVC
        0.0.0.0			www.greenmedinfo.com	# WVC
        0.0.0.0			www.heavy.com		# WVC
        0.0.0.0			freepatriot.org		# WVC
        0.0.0.0			www.uproxx.com		# WVC
        0.0.0.0			tcr.tynt.com		# WVC
        0.0.0.0			slightlyviral.com	# WVC
        0.0.0.0			pro.moneymappress.com	# WVC
        0.0.0.0			pro1.moneymappress.com	# WVC
        0.0.0.0			weknowmemes.com		# WVC
        0.0.0.0			www.shareable.net	# WVC
        0.0.0.0			www.thisblewmymind.com	# WVC
        0.0.0.0			www.tsu.co		# WVC
        0.0.0.0			www.upworthy.com	# WVC
        0.0.0.0			www.distractify.com	# WVC
        0.0.0.0			news.distractify.com	# WVC
        0.0.0.0			www.evwow.com		# WVC
        0.0.0.0			evwow.com		# WVC
        0.0.0.0			www.wimp.com		# WVC
        0.0.0.0			livelol.me		# WVC
        0.0.0.0			www.scoopwhoop.com	# WVC
        0.0.0.0			bustle.com		# WVC http://www.bustle.com/articles/62365-there-are-no-words-for-how-horrifying-this-unknown-creature-found-in-a-tuna-can-is
        0.0.0.0			www.bustle.com		# WVC
        0.0.0.0			www.tickld.com		# WVC
        0.0.0.0			www.zmescience.com 	# WVC
        0.0.0.0			www.addictinginfo.org 	# WVC
        0.0.0.0			www.viralnova.com	# WVC
        0.0.0.0			opter.co		# WVC
        0.0.0.0			www.sliptalk.com	# WVC https://plus.google.com/u/1/+IrreverentMonk/posts/cuNpUdErc7V
        0.0.0.0			sliptalk.com		# WVC
        # Spartz:  http://www.newyorker.com/magazine/2015/01/05/virologist
        0.0.0.0			www.brainwreck.com	# Spartz network
        0.0.0.0			www.dose.com		# Spartz network
        0.0.0.0			www.unfriendable.com	# Spartz network
        0.0.0.0			www.smartphowned.com	# Spartz network
        0.0.0.0			smartphowned.com	# Spartz network
        0.0.0.0			www.givesmehope.com	# Spartz network
        0.0.0.0			givesmehope.com		# Spartz network
        0.0.0.0			www.philosoraptors.com	# Spartz network
        0.0.0.0			philosoraptors.com	# Spartz network
        0.0.0.0			www.lolbrary.com        # Spartz network
        0.0.0.0			www.omgfacts.com        # Spartz network
        0.0.0.0			notsafeforwallet.net    # Spartz network
        0.0.0.0			sex.omgfacts.com        # Spartz network
        0.0.0.0			www.itsamememario.com   # Spartz network
        0.0.0.0			www.whenuseeit.com      # Spartz network
        0.0.0.0			memes.mugglenet.com     # Spartz network
        0.0.0.0			www.mugglenet.com       # Spartz network
        0.0.0.0			www.memeslanding.com    # Spartz network
        0.0.0.0			www.pokestache.com      # Spartz network
        0.0.0.0			www.ragestache.com      # Spartz network
        0.0.0.0			www.sixbillionsecrets.com # Spartz network
        0.0.0.0			smartphowned.hollywood.com # Spartz network
        0.0.0.0			www.hollywood.com       # Spartz network
        0.0.0.0			www.tasteofawesome.com  # Spartz network
        0.0.0.0			www.thatssotrue.com     # Spartz network
        0.0.0.0			unfriendable.hollywood.com # Spartz network
        0.0.0.0			love.givesmehope.com    # Spartz network

------
jrochkind1
I need a 'facebook sent me' version, lots of websites allow facebook
references but not neccesarily google.

I'm reduced to going to facebook and sending a message to myself with the url
so I can click on it.

~~~
cmstoken
Curious as well... Which sites?

~~~
jrochkind1
Looks like most of them ALSO allow a google user-agent, testing with this
plugin, so I'm good.

Mostly various newspaper sites.

------
bcg1
Righteous indignation abounds, a good indicator of achievement.

Nice hack jashkenas.

------
manymany
I think they took it down :( guess I'm gonna have to find out how to set it up
from source code:
[https://gist.github.com/jashkenas/9a4d9e648eb11aab394e](https://gist.github.com/jashkenas/9a4d9e648eb11aab394e)

------
dredmorbius
This seems like a feature which Privoxy could provide...

[http://www.privoxy.org/](http://www.privoxy.org/)

------
digi_owl
Been pulling something similar with Firefox and Modify Headers, only i claim i
come from Twitter...

------
jrochkind1
This extension seems to have been removed from the Chrome Web Store. Hmmmm.

------
udev
Good idea, but does not work with Financial Times.

~~~
pornel
Hi! I work for Financial Times.

We've started experimenting with First Click Free — soon you'll get one
article a day for free if you come from Twitter, Facebook, LinkedIn or Reddit.

We'd like to let Google index more of our content (which will require allowing
First Click Free from Google as well), but that depends on our other efforts,
such as launch of a redesigned desktop site (you should try the beta, it's
pretty cool) and technology changes behind mobile sites.

Of course we'll keep First Click Free if it leads to more readers subscribing
rather than just faking referrers.

~~~
asquabventured
Can you add Drudgereport to that list? I follow a lot of links from drudge and
when he links to a FT article and I click without realizing I end up at the
paywall splash screen and just immediately close the tab.

------
drzaiusapelord
So change useragent to googlebot?

~~~
toxicFork
No:
[https://news.ycombinator.com/item?id=9531943](https://news.ycombinator.com/item?id=9531943)

------
asquabventured
Jeremy,

Useful plugin but BLAH chrome! Any chance you will port this to Firefox?

------
jebronie
Google users are priviliged for a reason.

~~~
sp332
I thought Google users were only privileged because news sites want the full
text of the page to show up in searches, and Google punishes pages that show
different content to the web crawler than they show to searchers.

------
higherpurpose
Really? With a Google™ in the name, too? What do you think is going to happen
in the next two hours?

1) They'll take down your extension because you're violating their ToS and
their trademark

2) The extension will already piss off their "partners", but they can't really
_say_ that's the reason they're taking it down, so good thing you provided
them with option 1)!

I have nothing against the extension itself. I search for such headlines all
the time in Google, too. I just think it's reckless to taunt Google like this
and give them an excuse to shut it down on a silver platter.

~~~
icebraining
I don't see how he is violating the trademark (though that ™ should be a ®).

~~~
CoryG89
It seems like Google recommends using ™ for most of their trademarks. Their
website says that laws about using ® vary from country to country and there
are sometimes harsh penalties for using it with an unregistered trademark. It
seems at least OK to use ™ if you are unsure.

[http://www.google.com/permissions/trademark/our-
trademarks.h...](http://www.google.com/permissions/trademark/our-
trademarks.html)

[http://www.google.com/permissions/trademark/rules.html](http://www.google.com/permissions/trademark/rules.html)

