Why Scraping Will Continue and Will Only Get Worse

I wanted to write this post last week but never got the time. And it has actually worked out well because today Brian Stelter at the New York Times wrote an article about scraping. He (and many others) call it "excerpting" which is fine when you take a paragraph from an article. When you take a good percentage of an article and place it on your site, it’s scraping. I wrote about scraping and profit last December and offer additional input after a post I read several weeks ago on Alley Insider. In fact, it’s the same post that Stelter discusses in the Times today.

Let’s start by saying that there are two types of scrapers: those who scrape full rss feeds and those who scrape just enough to keep you on their site and keep the conversation there as well. In both cases the goal is to profit by keeping the reader interacting chatting on the scraper site and will rarely send any traffic back to the source. I’ve thought about this a lot and for a long time thought the full-feed scraper was the "bad" one and the other was "ok" but I’ve changed my mind. Both suck.

Alley Insider is one of the worst scrapers I’ve seen to-date in the "conversation" category. It seems you can’t go a day without some post being scraped. Frankly as I’ve said before I don’t get it. They have talented writers and have a bank of venture capital so why not write original content all the time?

The story discussed in the Times is one where editor Henry Blodget basically lifted nearly 400 words of an article by Peggy Noonan. And what’s on their site today is actually cut back! Note that Blodget added zero original content to the post. At the bottom Blodget notes, "We thank Dow Jones in advance for allowing us to bring it to you." What I wonder is why not quote one tiny bit, add some original analysis and link to the source?

Last week I had lunch with an NYC entrepreneur and we discussed the Peggy Noonan post at length. He said that the reason Blodget does this is because the WSJ will never email him and ask for the post to be removed. The entrepreneur went on saying that assume that the WSJ did ask for the post to be removed. Blodget would post the request on Alley Insider noting that it’s a great example of old media, yadda yadda. The reasoning makes sense.

Blodget has authored a post today about their excerpting policy. Here’s his policy, "We excerpt others the way we hope others will excerpt us." No, seriously that’s what he said. What Blodget is saying is that they will scrape/excerpt as much as they please.

The interesting thing is that Alley Insider added a link post a few times a week. This is a good thing and should continue and replace 100% of the scraping.

Why scraping will continue and will get worse this year

Scraping will continue because it’s easy and drives revenue, SEO credits and pageviews. And it requires no work. Here’s an example of a Digg frontpage entry by Digg top user Muhammad Saleem. In this case, he submitted the absolutely 100% scraped story from Alley Insider instead of the source. How many pageviews and users came to the story via Digg and never visited the source?

The goal should always be to excerpt the bare minimum and send the user to the source. It’s ok to include a small piece of another’s writing when needed to make a point, but whole posts shouldn’t be made of someone else’s work.

And understand, this is not just an issue for Alley Insider. I’ve written before about many other scrapers including so-called aggregators like Socialmedian. In SocialMedian’s case, they too have no reason to scrape so much of the content of a story. Unfortunately I’ve never received a direct answer to why they scrape so much content on stories that are shared on their service.

I will never understand why writers feel the need to scrape other’s hard work. You spend hours or days researching a story only to have someone scrape it in a minute and grab all the associated goodies. All the scraping does is hurt the overall blogging/media industry so let’s stop it now.