The obvious problem here is that there is an oversupply of news.
In an age of electronic worldwide distribution, do you need 938 outlets reprinting the exact same story? No. But that's what you see on Google News.
Oversupply means that weeding has to happen. Those outlets that survive will be those that provide unique information (either local news, infotainment, or big budget national/international news).
The newspaper as an institution is no more sacred than the Broadway play (which was destroyed by movies) or the family farm (which was automated and scaled up).
The only reason many people believe it is somehow different is that a significant fraction of what they read is written by newspaper writers (as opposed to, say, playwrights or farmers).
You have a good analogy, but the you take it a little too far. Broadway plays have hardly died. From what I hear, they are doing relatively well, and I have watched a number of them as they left broadway and toured into my city at the time. They have fallen somewhat in prominence though.
Similarly, family farms have lost their former prominence in society, but they are far from dead. My grandparents ran one until my grandfather retired due to health problems (recently deceased), and even then they leased their land to another family farm nearby.
There will be a winnowing in journalism, but this does not mean destruction for any of news, plays, or family farms.
2) Regarding plays (Broadway and elsewhere), they've died in the sense that they represent a much smaller fraction of the entertainment dollar relative to movies.
Percentage wise those have both died and you make valid points, but that is hardly death in a true sense. Family farms are still alive and well in many places in America, as are plays, they are just smaller percentages than in the past.
Google could simply remove them from their index for a week, see if a 25% decrease in traffic for a month will make them think twice.
I bet that they would change their tune by the the 3rd day and beg google to re-add them.
EDIT: Also as someone said further down, google could even charge for their indexing of the news, IMO they are providing a service to the news companies.
In theory this is akin to holding someone hostage, but isnt this what the media companies want? I doubt google would ever pay to license the links so the only other option would be to omit them...
> Should Google be able to do that without consequence?
What consequence should there be?
> I guess they don't because we would go over to Bing.
I'll bet that they wouldn't lose much search share and would lose even less search revenue if they ignored a huge fraction of the supposed "news sources". (1) News search isn't huge or lucrative. (2) Most news is "commodity", so it doesn't really matter where you get it from.
Cheap distribution (read internet) has put all the commidity news folks in competition with each other, driving prices down. A lot of journalists haven't figured out how that applies to them.
Note that Craigslist demonstrated that a huge fraction of newspaper subscribers/buyers weren't buying the paper for the news, but for the ads.
I think the real debate here is that value in the internet is moving relentlessly away from content producers towards content aggregators. Even a big newspaper site doesn't have the diversity that you find in an aggregator like Google News; likewise, no tech blog can compete with the diversity of news in Hacker News. I visit TechMeme and Hacker News far more than I visit any single blog, for example. TechMeme makes far more money than any professional tech blogger too.
Aggregators tend to build much more value to themselves quickly by riding on other people's content. Yet, aggregators are worth nothing without the underlying content. This irks many content producers, particularly the professional ones whose output accounts of the bulk of the traffic that aggregators end up sending. This trend of professionally produced content accounting for the bulk of the links is evident even in Hacker News.
If these trends continue, giant aggregators could end up controlling much of the content. Yahoo already produces a lot of content, and licenses content for Yahoo News (which is fitting considering Yahoo News has more traffic than any news site in the world). Google News, Digg etc. could follow.
This is the future I suspect Murdoch does not like, because it appears from his perspective to be third parties building value out of his content, without compensating him. Legally, I am not sure he has a claim - if there is a lawsuit on this, it will reach the Supreme Court, that's for sure.
Exactly. I don't think Murdoch is actually upset that Google drives people to his sites, what bugs him is that they go to Google first.
When you read a story in a newspaper, you almost automatically consume other content in that paper, including ads. Why pick up another paper to read more news when it's already in your hands? To a lesser extent the same is true for cable news, you watch a brief segment and you are bombarded with "coming up next" teasers and flashy graphics, pretty anchors, etc., all designed to keep you on that channel (watching ads).
The web is way less sticky than even television, sure you could flip channels but you lose context, and you might have to keep flipping to find something interesting, broadcast is a push medium. Contrast this with the web page: what is the easiest thing to do after reading the linked story? Search around the new site (that you may not be familiar with) or hit the back button and resume what you were doing: Pulling down stories you want to read, instead of waiting to see if something interesting comes up.
I think Murdoch gets the power of the aggregator, I don't think he understands that he has always pushed content to consumers. (Yes a lot of work goes into creating a brand that attracts these consumers to the paper/network but once they are there they have content pushed to them.) The web is a pull medium.
Billions were spent on the "portal wars" of the 90's, and everyone who participated lost to a page with a text box and 2 buttons.
You might be a fantastic writer, but without a good editor, and (in the case of one-offs like books) probably a good publisher, nobody will see it, or know it.
The analogy of editor/publisher to aggregators/search engines seems pretty fair. If nobody deems your content worth aggregating to HN, maybe it isn't? People who read HN trust it the same way I imagine people who have read the NY Times or National Geographic their whole life trust it.
When the web first started appearing as a news and editorial destination, it seems that people were quick to proclaim the death of curated content, because with infinite space and no need for physical resources, the constraints of binding your words together in 80 stapled pages of dead tree were history -- everything can just link to everything else, and when that fails, search! Turns out that the problem was never that a magazine could only be so many pages long, but that people only have so much time to be bothered with content they want to see.
Online the content and the editorial dept don't need to share a building -- in some cases its probably still better if they do, and in others its probably better that they're se separate as possible. I don't think there's any less value on content, or on editorial curation of what content gets in front of people. Sites like HN, Digg, Amazon and Hulu are popular destinations because they offer tailored and curated gateways to content people want. The curators have always been the bosses of the content creators, but now that they don't have to live under the same roof -- now that anyone can curate anyone else's content, as long as their curation/editorial skills are deemed good enough by the public to get them enough readers/viewers -- the balance of power has gone a bit wacky.
When Hulu's CEO called the TV networks "content providers" in that well-linked blog post a little whie back, few people batted an eye, but that was a huge slap in the face to those guys. If you're just the content provider, that means, on the org chart of the consumable content world, that Hulu is your boss -- they're your editor, your publisher, curating your content and choosing what lives and what dies. Hulu is supposed to be their dorky portal for cheapskate college students and bored secretaries! Why are they calling the shots now??
It's tiring to see people like Murdoch bellowing at top volume about their content being stolen when the content is the one thing people are still clamoring to consume by any means necessary. It's the editor's desk that's under attack and that's what's really freaking them out.
Yeah, I think such a move would hurt Murdoch and AP more than it would hurt Google. I understand contextual advertising, I suspect that Google makes far less than average on searches for news related terms, because people aren't looking to buy something, or solve a problem.
And while they are at it, they should cut off access to customers on ISPs that are pursuing an anti-net-neutrality agenda.
In both cases, I think Google would be better fit out for a war of attrition than their adversaries. On the other hand, Google probably doesn't want to hand Microsoft any opportunities right now by cutting off ISPs.
This is off on a tangent from the piece about Google and Rupert, but it's worth noting that in the general discussion of the demise of the traditional print newspaper business, few folks point out that the small community papers are in a better spot.
The real issue is that the national and world news has become a commodity that you can get from nearly anywhere. At the same time, the quality of the reporting has declined. Folks like Politico may represent the future of national reporting -- a leaner group of highly talented journalists getting interesting stories out. But why pay to read about some major national news story in paper XXX when you have 1,000 other news sources to read essentially the same content?
Conversely, take a look at small community papers. For example, my hometown paper is online at http://bristolri.com/. These guys are doing just fine (I know the publisher). You can't get their content anywhere else. And you know what? Average everyday people want to read local stories about the new school superintendent, the police report, the controversy over a local land development deal, the letters to the editor from their neighbors, etc. They buy the paper. Local advertisers still advertise. The paper is doing fine, and he's not alone -- lots of small papers are doing OK because their content has not been commoditized.
As for Rupert Murdoch, I think he's going in the wrong direction. If I were him, I'd worry less about distribution rights on my essentially worthless commodity national news content, and I would instead set my target on the big ratings agencies. The ratings agencies played a huge role in the credit crisis -- they basically gave bonds a gold star that were, in reality, crap. They should be vulnerable with some sort of new model. There are few companies positioned as well to destroy Moody's as News Corp / WSJ / Dow Jones with their depth of financial industry content / resources / etc. I don't have any idea of exactly what a rating agency 2.0 would look like, but it would seem like a much more productive and valuable place for Rupert to put his focus.
The other problem the big newspapers chains have is that many of them have substantial debut burdens, which is, again, something a lot of smaller locally owned papers didn't have the privilege or inclination to do.
I've seen this story repeated a number of times now. Murdoch seems to be obviously wrong to those of us that follow this stuff. I wonder however, if Murdoch is trying to convince the 99% of the public who don't read HN/Techcrunch et all that Google is doing something bad here when he doesn't believe it himself.
It seems obvious to me that Murdoch et al. want to stay in the Web and be paid rather than de-index and disappear. Given that, these "just use robots.txt" articles seem just as disingenuous as the original statements they're mocking.
You was downmoded, but I think you are right. From the article:
"go to Google News, or type a newsy topic like "Obama wins Nobel" into Google's search box. What do you get? Headlines and very brief teasers linking to news stories from news sites. If you click on them, you are taken to that news site, where you can read the story, which is surrounded by that site's ads. What, exactly, did Google steal in this scenario?"
What exactly did Google steal? Very exactly, Google steal "Headlines and very brief teasers". From the perspective of copyright law, yes, it can be classified as quoting, so they can't sue google, but those "headlines and teasers" do carry news value and in some cases took large amount of resources to produce. Now, Google took it and provided link as a payoff. Is that payoff adequate? Murdoch and many news producers don't think so.
Of course, teoretically you can use robots.txt. But in reality you can't, since your competition is also indexed, so you have to play by Google rules, even if you believe they didn't pay you enough for the value they are taking.
Say that I wasn't happy with the salary that Google is paying me. It's a good salary, but I want more. After all, my labor has already generated more revenue for the company than I am likely to make in my lifetime. I, of course, don't have to work with Google - I can quit at any time. But in reality I can't, since my competition is also willing to work for the same wages and I have to eat. So I have to play by Google's rules, even if I believe they don't pay me enough for the value I am making.
Do you have much sympathy for me?
Why, then, do you have much sympathy for Rupert Murdoch and his billion-dollar corporation?
It's an analogy, and like all analogies, it's imperfect. But it's perhaps more apt than you'd expect. You might argue that I could easily quit and get a job at, say, Yelp or Facebook. But News Corp could easily block Google and form a partnership with Yahoo or Microsoft - heck, there's rumors that they are doing just that. Or you could say that I should quit and form my own business - well, News Corp could easily go hire a crack team of 100 programmers to build their own indexing engine. Except they probably won't find the good ones, since everything indicates that Google is a better place for a computer scientist to work than News Corp.
You might also argue that I'm already paid for my labor. Well, News Corp is already paid for the their headlines and their brief teasers: they're paid when people visit their page and advertisers put up money for that privilege. They just want to be paid more. It's not all that different from me wanting a higher salary because I generate far more value to the company than I cost.
One difference is that you could work for thousands of software companies. In search there's Google and somewhere down the road Bing. Everyone else faces a near impossible road uphill. What if the only places you could work is Google or Microsoft? Do you think you would still get a fair treatment from either?
In many other industries, there really are only 2-3 major employers you could work for. My sister's choices (petroleum geologist) are basically limited to ExxonMobil, ChevronTexaco, ConocoPhillips, BP, and to a lesser extent Valero. If you're an auto assembly worker in Detroit, your choices are basically Ford, GM, and Chrysler.
My point isn't that you'll get fair treatment (you won't), it's that life's not fair. It sucks to be an employee, it sucks to be a small business owner, and it particularly sucks to be one of the above when all your counterparties are giant monopolies. But as a society, our response to most people in these positions is "Suck it up and deal - you knew what you were getting into when you took the job." News Corp has been on the winning side of the "We're the only game in town" bargain a lot - they just finally met someone who plays the game better than they do. I really dislike the idea of changing the rules of the game because one billionaire is suddenly losing, when we didn't change the game when millions of middle-class families were losing.
If you use robots.txt and shut out Google, your news source will fade into obscurity. This is obvious. The conclusion, however, is that by indexing their news sites Google is providing the news sites a service, so if anyone should be paying it's Murdoch.
'What exactly did Google steal? Very exactly, Google steal "Headlines and very brief teasers". From the perspective of copyright law, yes, it can be classified as quoting, so they can't sue google, but those "headlines and teasers" do carry news value and in some cases took large amount of resources to produce. Now, Google took it and provided link as a payoff. Is that payoff adequate? Murdoch and many news producers don't think so.'
This seems very much the same as walking past a newsstand, glancing at the headlines of assorted newspapers and magazines, perhaps flipping through one or two, to see what's happening. The stand owner is "giving away" the headlines, but how else are people going to know about the publications?
By similar logic, a newsstand owner should have to pay just to place a newspaper on display. Where has Murdoch been all those years while this robbery was occurring?
Your analogy is logical, but there's still a difference. Newsstand and newspapers agreed beforehand the terms of their cooperation, and while they don't pay just for displaying the paper, they do pay certain amount of money for each sold piece, as is was agreed. But with Google, nothing was agreed beforehand, Google just decided that it will display the part of the content and will provide link as a payoff.
But now I don't want to criticize Google, nor do I want to be Murdoch's advocate. I just notice that the newspaper critics don't look at the issue from publisher's perpective, thus making their articles a bit one-sided.
Google is indexing the full text of the stories, and using that text to help figure out when to show the headline and teaser. They might just be showing you a little bit of the story, but they are making use of the full-text of it in a big way. I think that's worth noting here.
What I don't get is why the newspapers don't shut Google down in their robots.txt and then offer up a "firehose"/API a la Twitter that anyone can get access to for a fee. Yelp, Delicious, Twitter, Flickr, these services all have APIs that can be used to incorporate their content into other services, on their terms. I know the guardian and the new york times are working on APIs, but they should offer up a full-text firehose version that a company like Google could pay a fair price for.
> What I don't get is why the newspapers don't shut Google down in their robots.txt...
That's because their real motives have nothing to do with what they are saying.
You can't assume that they want what they say they want because, of course, they are completely capable of making it happens -- as you say, with robots.txt. It's trivial and I can guarantee you that they're aware of that fact.
What they are probably angling for is some kind of legal settlement, either a lump sum or royalties for when their search results are shown, or Google uses their content to compute rank, or some kind of special treatment, a special relationship with Google.
To really remove themselves from Google would be shooting themselves in the foot, and doubtless they know it.
Or, like you said, they want Google to pay for a firehose. But why would Google pay for a firehose when the content is right there on their sites, which Google indexes anyway?
People, you've got to be more multi-layered in your thinking. I know nerds are known to be more literal than most, but you can't go through the world thinking that other people are like you. You have to compare people's actions, with what they say, and figure out what the gestalt means. And you have to question a person's motives for saying a thing.
Speaking, writing and publicizing are not just low-bandwidth means of transferring pure information.
Newspapers should stop them from indexing and offer up this API instead, at a price.
Look, a web site is a web site.
Facebook doesn't let Google in to index all of it's data, this created an opportunity to eventually sell access to index it.
Newspapers could do the same, stop them from indexing their sites and sell them access to this firehose api, just like _every_ other data-driven startup.
No one flinches when Yelp offers and API with specific usage conditions. Newspapers should do the exact same thing.
I think that you can do this under "fair use", but "fair use" doesn't exist in all countries in the world.
And google has to obey to local laws and local copyrights and not just to US law.
The google book case is just another thing that's going to backfire at google. The ignorant attitude of google making foreign books (books written by authors where it is clearly illegal by the law do scan in the books and make those freely searchable on the net) available to all US citizens, even though the authors don't want their books to be available trough google, will only create more hate against google.
I believe that Google does obey local laws. I've certainly had to jump through hoops because of inane restrictions that the Chinese government puts on web content. A bunch of the stuff I've worked on simply isn't available in other countries because of various laws and restrictions that have to be cleared before we can launch.
I personally know an author who is simply delighted that google is making his (out of print) book available. I think he will even get payment if someone wants the whole book.
"From the perspective of copyright law, yes, it can be classified as quoting"
Then aren't we done. Fair use and all that. It doesn't matter what value Murdoch places to them. In fact, why should it matter? Should every teaser be worth $1,000,000? If you let the marketplace decide then what happens? Probably what we have right now, because at least one news producer will think that Google is providing a great service.
"but those 'headlines and teasers' do carry news value"
Newspaper vending boxes have a nice glass window in front so you can read the headlines and choose whether or not to buy the paper. How is what Google does any different at all?
I can't believe Google ever paid the AP in the first place. Seems like a bad deal from Google's perspective. They could have easily done without the AP's content.
AP doesn't compete with Google, they don't run ads. The newspapers do - if the newspapers were to all disappear but AP survived that might be good for an online source of news and ads?
As many people have said before, you can get reprinted news from everywhere, people will never pay for that kind of news as there will never be a situation where someone doesn't find value distributing in with the advertising model.
Investigative journalism and niche long form articles are the way to go, you create an interesting enough website for long form in a niche and people interested in that area will pay (well at least I could justify it).
Because he doesn't want his ad salespeople to have to admit to potential advertisers that "well, yes, in fact, we are not findable on google and other search engines."
In an age of electronic worldwide distribution, do you need 938 outlets reprinting the exact same story? No. But that's what you see on Google News.
Oversupply means that weeding has to happen. Those outlets that survive will be those that provide unique information (either local news, infotainment, or big budget national/international news).
The newspaper as an institution is no more sacred than the Broadway play (which was destroyed by movies) or the family farm (which was automated and scaled up).
The only reason many people believe it is somehow different is that a significant fraction of what they read is written by newspaper writers (as opposed to, say, playwrights or farmers).