Yahoo! Pipes - Rewire The Web

Recently I've discovered a fantastic free service provided by Yahoo! which makes it possible to convert any XML data feed into a JSON data feed. http://pipes.yahoo.com/pipes/ It is easy to create a pipe that provides a JSON data feed with a callback function for web services that don't actually implement that functionality in their API.

This allows me to completely eliminate my xml2json generic handler in my mashups. Now I can create web applications that combine data from multiple public data sources without creating anything on the server. I've cut ASP.NET completely out of the picture! I was not very happy with that local web service requirement for my help files. Technically, I'm now relying on Yahoo servers to provide helper services that my local server used to handle.

As an example, I created a pipe for the LiveVideo featured videos data feed. LiveVideo is a popular vlogging site based on ASP.NET. I can tell that they use generic handlers to provide their XML data feeds by the ASHX file extension. But they don't provide their data in the JSON format that I prefer.

The screenshot below shows how I wired up a few modules to create a new data feed. I used a URL Builder module to provide my developer id string. The Fetch Data module is slightly tricky because you must provide the path to the item list that you are interested in. This is wired to the Pipe Output module. Click the Refresh link to ensure that you are getting the expected list of data items. Other modules allow you to rearrange the data or filter it based on regular expresssions. However you don't need any of them to transform XML to JSON.

Next click the "Run Pipe..." link at the top of the pipe editor. The "More options" drop down list includes the option "Get as JSON".

This just gives you the link to your pipe with an extra query string value for the data format. You can also add the callback function:

http://pipes.yahoo.com/pipes/pipe.run?_id=6JqtThPQ3BG7I1ewoeNLYQ&_render=json&_callback=ws_results

 You use this web address in your client web page as the source for external JavaScript:

Now you just need a callback function that is passed the object you use to reference the data:

function ws_results(obj) {
    for (prop in obj.value.items[0]) {
        alert(prop);
    }

Yahoo! Pipes might provide a simple solution for minor data transformation tasks that would otherwise require considerable developer effort or major investments in other technology. It might also be useful if you wanted to offload some processing from your server without taxing the client. This appears to be an example of cloud computing.