Because the publishers are batshit crazy on this one.
Google has responded in a blog post to last week's "Hamburg Declaration" by newspaper and magazine publishers, which asked the European Union government to force online news aggregators to pay for content created by the publishers.
As this blog has noted obsessively, publishers on this side of the pond have similar views.
"We want a fair share of the revenues," Mathias Döpfner, CEO of the newspaper giant Axel Springer AG, said in a statement, "which are already being generated through the commercial exploitation of our content by others, as well as the development of a market for paid content in the digital world."
Google's response is simple: If you don't want Google News to aggregate your content, it won't.
"The truth is that news publishers, like all other content owners, are in complete control when it comes not only to what content they make available on the web, but also who can access it and at what price," writes Google executive Josh Cohen. After a two-sentence explanation of how webmasters can keep Google from indexing their pages, Cohen adds, "In short, if you don't want to show up in Google search results, it doesn't require more than one or two lines of code."
It doesn't seem to matter how many times Google or anyone else says this. The publishers don't want to hear it. And the reason they don't want to hear it is because they want Google to give them money. It's hard to picture Google losing this argument if logic is allowed anywhere in the building.
Google is saying: We're sending you customers, and we'll stop doing that if you want us to. I'll give the newspaper and magazine publishing industry enough credit to believe that its webmasters understand you can tell Google not to link, so the publishers are saying by their actions that they don't want Google to stop sending those customers. But they want Google to send some money along with the customers.
Google sending them readers is either good or bad. If it's good, great. If it's bad, Google's willing to stop. Why should Google also have to give the publishers money?
Speaking of money, it can make almost anything happen in politics, so let's say the publishers get their way and Google is forced to share revenue with them. Wouldn't Google simply stop indexing the publishers' sites on its own? The publishers would then go down the drain unless they could create their own Google, and who wants to bet on them to succeed at that?
Newspaper and magazine publishers have to be the first industry in history that's rebelled against another business sending them customers. Imagine a sandwich shop owner asking the busy train station next door to pay him to come in and give away free samples promoting the shop. "I can't make any money selling sandwiches because everyone's just eating the free samples," the sandwich guy says.
And the railroad thinks: So sell them something else, dummy.

Salon.com
Comments
...Wouldn't Google simply stop indexing the publishers' sites on its own? ...
well, i'm guessing if Google did that, then the publishers would have the audacity to sue Google for unfairly "not indexing" them!
I've been in the newspaper business since 1989. I was on the internet before most newspapers were, before there was even a World Wide Web. I always thought that as soon as publishers started to realize that they were losing money because they had content available for free they would just pull it back.
Some have and some haven't. The NYT has gone back and forth. The Wall Street Journal has always had a pay wall. Neither are doing particularly well.
That's because it's not about the price charged for the content; it's about selling the audience.
Newspapers are in trouble because they forgot how to sell audiences to advertisers.
When I worked as a newspaper circulation executive my goal was to make enough revenue to cover variable costs of printing and distribution, in essence making it free to distribute. Advertising sales had to cover all of the overhead - salaries, benefits, building, maintenance, presses, trucks, news gathering, etc.
This is not unlike TV, radio or even the internet. Radio and TV always gave away content because once the studio was built, the contract with the talent was signed and the transmitter was in place, there was very little cost to deliver the content.
However that content was sponsored so all of those other fixed costs could be paid with enough left over for profit, pension plans and a decent Christmas party.
This is what newspapers need to relearn. They can block Google or any other search engine from indexing their content so they can charge for it, but that's just not going to generate enough money to run the rest of the operation. Newspapers need to engage audiences and sell those audiences to people that want to reach them. Until they start doing that again they will never be profitable. People are just not going to pay enough for content to make up for all the losses in advertising dollars newspapers have seen.
Hiding from Google isn't going to help. Finding new ways to delight and amaze audiences, and proving to advertisers that your audience is delighted, will.
I have a Rhapsody account, and for $13 a month I can listen to all the music I want, selecting from millions of tracks. I don't have to go to hundreds of places around the web in order to listen to music. In other words, Rhapsody is basically a "music aggregator," but all the music is on one place and you have to have a paid account to listen to it. You can send links to other people so they can hear individual songs, but but you have to have an account to be able to do that.
Why don't newspapers follow that example -- shut down all of their web sites, and create a news aggregation service that basically has every newspaper in the country in it, all available for a monthly fee. People could then pay $X per month in order to read every newspaper in the country, as much and as many as they want. Subscribers could basically design their own virtual "newspapers" by selecting what they wanted to read. Revenue would come from subscriptions and advertising, and would be split among the newspapers according to some formula.
But this raises a question: when content can be distributed electronically, what is a "newspaper?" With electronic distribution you don't need newsprint, printing, trucks, etc. All you need are reporters and maybe a little office space. In other words, in an electronic era, it seems to me that much of what we consider to be the "newspaper" is simply obsolete non-value-added infrastructure. Eliminate or bypass the infrastructure, and the costs of providing the content go way down.
A large news aggregation service could also be used by individual journalists. If King Kaufman could publish his own stuff on such a service and get income from it based on the popularity of his articles, why would he need Salon.Com?
That's how I see it, anyway.
In the print model, advertisers pay based on the number of eyeballs, not on the number that acts on the ad. It seems to me that advertisers are getting free exposure every time their flash ad pops up on any screen regardless of clicks. Well, actually not, because advertisers are being charged a small fortune by Google while publishers get pennies for clicks. Something's wrong here.
But isn't that exactly what Google is doing? Where's the distinction?
I am a news consumer. I click through on my laptop and my CrackBerry. These guys don't want me to read their stuff and see their ads? Fine. I'll read elsewhere.
That is what these idiot publishers do not get and refuse to get. Stick an ad in front of my face enough, and I will click through to that, too.
But as I said, I might be wrong.
Do a google search right now for the word newspaper. The ads I got were
1.
San Francisco News & More
All of Your Top Local News, Coming
Events and Insider Tips from Locals
Examiner.comSan Francisco, CA
2.
Local Jobs
Found: 321 jobs in your area
Make $15 -$100 / hour with bonuses!
localjobsavailable.net/
The Examiner is a local paper here. If I click on there, they would pay google for my click. If they were on the front page of google news they would pay nothing for the click and google is the bad guy?
Google isn't a train station, it's a food court. Google gives out free samples for the best restaurants, for free. It's the newspaper's fault if their sandwiches taste like shit and no one wants to click through, not google's.
If the problem is people read headlines and nothing else, why don't we go cover up all the headlines on the physical newspaper boxes. Those boxes give a few 1/2 articles and headlines away for free. Usually those boxes are set right next to each other so you can choose by the writing and headlines which paper you want. Since we shouldn't be allowed to read headlines without paying for them - black out the front of those boxes. Not rushing out to do that? Why not?
All google is doing is giving more exposure to smaller newspapers and regional papers most of the country would never see. Boo hoo?
What are they complaining about? No wonder they are failing. They aren't getting the clicks to justify their advertising and they want to charge google to bring in eyes to justify their advertising. Who is running these places?
Use it. Why fight fires with gasoline?
But isn't that exactly what Google is doing? Where's the distinction?
How is that exactly what Google is doing? Google is collecting information that's publicly available, and disseminating it. Then it's selling advertising. It's not an industry. It's one company.
Yes, but Google has a stronghold on online advertising. When I talk about "single price" I'm taking about Google's adrates for advertisers. The company has control of both aspects of the web: the sale of advertising and the control of information dissemination. It seems obvious to me that this does not work in anyone else's favor -- the public's, the media, ergo the public again -- except for the company.
The irony is that newspapers used to have these same monopolies only within limited geographic regions. TV still does. These monopolies are easily manipulated when advertising dollars are involved. Example: every election held in which advertising is purchased. TV sells the ads at high costs, necessitating the raising of funds by politicians, and the person who raises most typically wins.
Recently several media outlets asked, "Can Google Change Election Outcomes?" -- (http://politics.theatlantic.com/2009/06/can_google_change_election_outcomes.php) which is hilarious because TV's been doing it for ages.
The relationship between advertising and distribution is not as diverse as many seem to think.
There have been many articles reporting on The Department of Justice's exploration of Google for antitrust violations, specifically as it relates to deals with Apple and book publishing. I suspect that as that case develops, the investigation will widen. You can 'google' (now it's a verb!) the company with antitrust and you will find dozens of stories on a wide array of ways in which the company is accused of having a monopoly.
Google does not have a lock on internet advertising, they are, like Coke and or Pepsi just the biggest elephant in the room.
Search Yahoo for socks. You will get a blue background area at the top that is clearly a paid landing paid, followed by search results. On the right column you will see a listing of other ads for socks which is probably some sort of click through pay model.
Alta Vista (yes, they still exist) does the same, premium landing pad at top, other ads on the side
Lycos (once again, yes, they still exist) - premium landing pad top and bottom of search results and column of ads on right hand side
ask.com - shows Amazon products at the top of the page, must have a exclusive deal with them.
cuil - ads on right hand side, premiums at bottom of page.
Anyway, point being Google happens to be the best search engine or the one with the most name recognition butthey are not the only player out there. Do they have an unbalanced effect on ad rates, sure, Coke as the same effect on sugar prices. I don't want to canonize Google them since I am sure they are as seedy as any other monolith corporation but as far as this debate goes calling them out for being the evil that is the demise of newspapers doesn't work. Especially considering that the demise of newspapers began long before the word google was a verb.
If they were creative and savvy they would find ways to work within Google's structure to increase their own revenues.
EC
But is that Google's fault? It seems to me if the notion of "markets" works anywhere it works on the internet. Google built a better product and people flocked to it. It's not the same as say back in the nineties when Microsoft was sticking Internet Explorer in their OS package and then making it almost impossible for users to disable/uninstall it and install a rival program. Google is where they are at because they did something right, and they will fall hard and mighty when they no longer give the people what they want.
But I think we're digressing. This is about whether newspapers should be getting a bit of Google's pie because Google offers a search service (free one at that) that will list a story from a newspaper with a link back (free again) to that newspaper if the story is relevant to the users search items.
To flip this around how about Google telling the Euro papers that if they want links back to their sites from the stories that come up as search results its going to cost them? Why should Google provide a free gateway to the London Times? Shouldn't they get a piece of their pie?