[New article]: Creating a generic Site-To-RSS tool (site scraping)

My blog has moved.
You can view this post at the following address:
http://www.osherove.com/blog/2003/9/28/new-article-creating-a-generic-site-to-rss-tool-site-scrapin.html
Published Sunday, September 28, 2003 11:48 PM by RoyOsherove

Comments

Sunday, September 28, 2003 8:15 PM by Scott Watermasysk

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Hi Roy,

1. DotnetWire already has a feed: http://dotnetwire.com/newsfeed/rss/

2. I would not recommend hosting a feed the scrapes someone else's site. If a site (such as DotNetWire) wanted to provide its content via RSS, they would. Many of these sites only revenue stream is through page views. Hosting something like this is effectively stealing from them. For your own personal use, I see no harm.

-Scott
Sunday, September 28, 2003 8:26 PM by Roy Osherove

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Scott: Oops! Ok. I won't host it. But it's still a great excercise! :)
Sunday, September 28, 2003 8:43 PM by Scott Watermasysk

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Absolutely. For internal/personal use I think it is a great idea. Companies that want to make use of RSS, but don't have the time to create the feeds can probably make a lot of use out of it.

yole used to have a feature in Syndirella which did this for you.
Sunday, September 28, 2003 9:34 PM by Randy H.

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Roy- don't be down. This is a great idea and good effort, glad to see the work you did here.
Monday, September 29, 2003 9:22 AM by SBC

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Roy.. good stuff.. you the man..
Monday, September 29, 2003 11:22 AM by Drew Robbins

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Great article. I need something like this on the backend of PDCBloggers.net to track blogs that do not have RSS feeds.
Tuesday, September 30, 2003 2:52 AM by TheCshark

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Tip: use ToArray instead of GetBuffer, ToArray works when the MemoryStream is closed.
Tuesday, September 30, 2003 9:46 AM by Roy Osherove

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Thanks for the tip TheCshark!
Friday, October 31, 2003 6:48 AM by Krisztian Gyuris

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

I tried your sitefeeder site and it is really nice :)
I had a small addition when I clicked on a feed and then tried to look at the help menu, it did not showed up in full only the first item is visible. If I am not watching the one of the feeds the Help menu works ok.

Cris
Monday, April 12, 2004 6:16 AM by Robin Good

# re: [New article]: Creating a generic Site-To-RSS tool (site scraping)

Though I highly respect and honour your ethical approach to RSS scraping, let me mention that RSS scraping can ALSO greatly increase a site reach and visibility, including the consequent traffic, market value and even credibility it may gain in cyberspace.

I would think that it is a fair bet approaching RSS scraping by always first contacting the site webmaster/owner and explaining the reasons and interests for doing so. Obviously if you are scraping a site for personal reasons (to keep trac of its changes), you are certainly going out of your way to do something that can be easily done much effectively with dedicated tools, and without a need to have an RSS feed.
(Free and paid service/tools like Copernic Tracker, Watchthatpage offer a simple and very effective way to track any web site for changes of any kind).

Therefore is your intention is syndicating through RSS, please do not restrain yourself on this front, while informing the site owner of your intentions and goals.

Having more re-usable RSS-based content out there can only be a beneficial thing, as the more we are able to convert Internet content to it, the more we will able to brea free of mainstream corporate controlled media and to start really appreciating the value of a growing number of people who will select, aggregate, filter, and combine the best news and content out there in specific categories of interest.

This is the emerging role of the "newsmaster", and the work you have done here in providing an extra route for such professional to increase its abilities, can only be greeted with sincere kudos and thanks from those who can see the above and who are willing to start investing in this direction early in the game.

Many thanks Roy, and keep up the good work!