Public available data - FREE screenscraping or pay for API

Been working on a semi-commercial pet project of mine, for which I need a data feed.

A decent enough subset of this data feed is publicly available from this content provider's main website. However, the full dataset (though I won't need all that) is available through an HTTP GET XML API... For a flat fee of over 500 dollars per year.

What would you do? 1) Roll it yourself in about 20 lines of .NET code (using HttpWebRequest & Regex's) and scrape it; 2) Pay for the API...?

Needless to say, I went for 1)...even for just the fun.

1 Comment

  • Mischa,



    I'd say the same. And yes, when there's a redesign, your regex's break, but that can easily detected.



    Also - the mechanism I'm using will store this data in an XML file and when it cannot retrieve the HTTP update properly (ie. an exception occurred), it will use the last successful feed data, and notify me of a problem.

Comments have been disabled for this content.