Robert Robbins

Recently I've discovered a fantastic free service provided by Yahoo! which makes it possible to convert any XML data feed into a JSON data feed. http://pipes.yahoo.com/pipes/ It is easy to create a pipe that provides a JSON data feed with a callback function for web services that don't actually implement that functionality in their API.

This allows me to completely eliminate my xml2json generic handler in my mashups. Now I can create web applications that combine data from multiple public data sources without creating anything on the server. I've cut ASP.NET completely out of the picture! I was not very happy with that local web service requirement for my help files. Technically, I'm now relying on Yahoo servers to provide helper services that my local server used to handle.

As an example, I created a pipe for the LiveVideo featured videos data feed. LiveVideo is a popular vlogging site based on ASP.NET. I can tell that they use generic handlers to provide their XML data feeds by the ASHX file extension. But they don't provide their data in the JSON format that I prefer.

The screenshot below shows how I wired up a few modules to create a new data feed. I used a URL Builder module to provide my developer id string. The Fetch Data module is slightly tricky because you must provide the path to the item list that you are interested in. This is wired to the Pipe Output module. Click the Refresh link to ensure that you are getting the expected list of data items. Other modules allow you to rearrange the data or filter it based on regular expresssions. However you don't need any of them to transform XML to JSON.

Next click the "Run Pipe..." link at the top of the pipe editor. The "More options" drop down list includes the option "Get as JSON".

This just gives you the link to your pipe with an extra query string value for the data format. You can also add the callback function:

http://pipes.yahoo.com/pipes/pipe.run?_id=6JqtThPQ3BG7I1ewoeNLYQ&_render=json&_callback=ws_results

 You use this web address in your client web page as the source for external JavaScript:

Now you just need a callback function that is passed the object you use to reference the data:

function ws_results(obj) {
    for (prop in obj.value.items[0]) {
        alert(prop);
    }

Yahoo! Pipes might provide a simple solution for minor data transformation tasks that would otherwise require considerable developer effort or major investments in other technology. It might also be useful if you wanted to offload some processing from your server without taxing the client. This appears to be an example of cloud computing.

The microblogging site http://www.pownce.com/ has opened to the general public so I decided to give their API a try. But the main point of this blog post is to dispute the assertion that JSON is "easy for humans to read and write". It is not easy to read! JSON is a mess of curly braces, brackets, and quoted strings that is even more difficult to read than XML. I had considerable trouble working with the JSON data that Pownce provides because it seems to be nonstandard and requires the use of array indices which aren't usually needed. At first I tried to determine the data's object model by loading the XML version into XML Spy and finding the XPaths for a value. This was only slightly useful. Then I found a JSON Viewer on CodePlex which proved to be extremely helpful.

 As you can see in the screen capture below, the JSON Viewer clearly indicates that to reference a profile image you need to use the following syntax:

obj.notes[0].note[11].sender[5].profile_photo_urls[0].tiny_photo_url

http://pownce.pbwiki.com/API+Documentation1-1 Pownce provides a JSON data feed with a callback function so there was no need for my XML2JSON generic handler.

// this function must appear before the remote script block, requires jQuery and jquery.flydom
function ws_results(obj) {
//    alert(obj.notes.length);
//    for (prop in obj.notes[0].note[0]) {
//		alert(prop);
//	}
     $('#response').createAppend(
            'table', { width: '500px', className: 'tableData', id: 'Notes' }, [
                'tr', { }, [
                    'th', { align: 'center' }, 'Pownce Public Note List'
                           ]
               ,'tr', { }, [
                'td', { align: 'left', id: 'info', style: 'font-size: 10pt;' }
                           ]             
            ]
        );
        // loop through items
        for (var i = 0; i < obj.notes.length; i++) {
            // create table row
            $('#Notes').createAppend(
                'tr', { }, [
                'td', { align: 'left', id: 'note' + i, style: 'font-size: 10pt' } 
                          ] 
            );
              $('#note' + i).createAppend('b', { }, 'ID: ');
              $('#note' + i).append(obj.notes[i].note[0].id);
              $('#note' + i).createAppend('br', { });
              $('#note' + i).createAppend('b', { }, 'Permalink: ');
              $('#note' + i).createAppend('a', { href: obj.notes[i].note[1].permalink }, obj.notes[i].note[1].permalink);
              $('#note' + i).createAppend('br', { });
              $('#note' + i).createAppend('b', { }, 'Type: ');
              $('#note' + i).append(obj.notes[i].note[2].type);
              $('#note' + i).createAppend('br', { });
              if (obj.notes[i].note[2].type == "link") {
                  $('#note' + i).createAppend('b', { }, 'Link: ');
                  $('#note' + i).createAppend('a', { href: obj.notes[i].note[3].link[0].url }, obj.notes[i].note[3].link[0].url);
                  $('#note' + i).createAppend('br', { });
              }
              $('#note' + i).createAppend('b', { }, 'Body: ');
              if (obj.notes[i].note[2].type == "message") {
                  $('#note' + i).append(obj.notes[i].note[3].body)
              }
              else {
                  $('#note' + i).append(obj.notes[i].note[4].body);  
              }
              $('#note' + i).createAppend('br', { });
              $('#note' + i).createAppend('b', { }, 'Timestamp: ');
              if (obj.notes[i].note[2].type == "message") {
                  $('#note' + i).append(obj.notes[i].note[4].timestamp);
              }
              else {
                  $('#note' + i).append(obj.notes[i].note[5].timestamp);
              }              
              $('#note' + i).createAppend('br', { });
              $('#note' + i).createAppend('b', { }, 'Display Since: ');
              if (obj.notes[i].note[2].type == "message") {
                  $('#note' + i).append(obj.notes[i].note[6].display_since);
              }
              else {
                  $('#note' + i).append(obj.notes[i].note[7].display_since);
              }               
              $('#note' + i).createAppend('br', { });
              $('#note' + i).createAppend('b', { }, 'Sender: ');
              if (typeof(obj.notes[i].note[10].sender) == "undefined") {
				  $('#note' + i).append(obj.notes[i].note[11].sender[0].username);             
              }
              else {
                  $('#note' + i).append(obj.notes[i].note[10].sender[0].username);               
              }
              $('#note' + i).createAppend('br', { });
              $('#note' + i).createAppend('b', { }, 'Photo: ');
              if (typeof(obj.notes[i].note[10].sender) == "undefined") {
                  $('#note' + i).createAppend('img', { src: obj.notes[i].note[11].sender[5].profile_photo_urls[3].medium_photo_url, align: 'left', hspace: '5'}, null);            
              }
              else {
                  $('#note' + i).createAppend('img', { src: obj.notes[i].note[10].sender[5].profile_photo_urls[3].medium_photo_url, align: 'left', hspace: '5'}, null);              
              }
              $('#note' + i).createAppend('br', { });              
        }
}

After I wrote my last blog post I did a search for ASP.NET design sites and found an old set of design templates on MSDN. http://msdn2.microsoft.com/en-us/asp.net/aa336602.aspx This appears to be a special section of MSDN that Microsoft created for designers who will be working with ASP.NET 2.0. I also found an old blog post from July 2005 which seems to be about this Dev Center for designers: http://blogs.msdn.com/bgold/archive/2005/07/15/439299.aspx Unfortunately I get the impression that this project didn't attract much interest or continue beyond its initial effort. For instance, there is a dead link for Building ASP.NET 2.0 Web Sites Using Web Standards. However, there are several design templates for you to download and examine. This resource seems to be all but forgotten. It was not mentioned in the Professional ASP.NET 2.0 Design: CSS, Themes, and Master Pages book I read or linked to on the official Starter Kit web page so I thought I'd draw attention to it here.

I haven't really searched for ASP.NET web design ideas and styles but I certainly have not come across anything through my casual reading of online articles. You can find plenty of material on the technical aspects of themes, ASP.NET 2.0 CSS Friendly Control Adapters, master pages, and skins but nobody applies any actual design to their examples. I've been researching many web design sites but their focus is entirely on CSS. Web designers don't seem to concern themselves with anything except the CSS while developers totally avoid applying the eye candy. This leaves being visually creative with ASP.NET an uncharted territory.

One of the things I would like to get from participation in the developer community is some feedback on my design efforts. I would also appreciate any tips and tricks that I could use. Ideally, we could use a community site that brings developers and designers together but I don't see that happening.

Here is an example of my typical web site design for code experiments. It does not get any plainer than this. All I use is a header tag and a horizontal line tag.

And here is my latest effort at a more inspiring design for my experiments. I borrowed the page layout from Microsoft Expression Web. The layout is based entirely on CSS and does not use any tables. However the original design was very boring and boxy so I added some rounded corners to the masthead and the main content area. I tried various methods of applying rounded corners but the method that worked best was something I actually found through the ASP.NET Weblogs. Check out Josh Stodola's blog post on how to create fluid-width rounded corners using divs. I like how the corners are partially transparent PNG graphics that take on the color of the background without requiring me to create new images. Although you cannot see it in this image, there is a striped background that is specified in the CSS. You can generate background stripes using an online tool at: http://www.stripemania.com/ There is also a shaded border around the center content which is applied dynamically through JavaScript. I found that at http://www.ruzee.com/blog/

My recent design effort was to improve the look of my tables. I found a great web site, CSS Table Gallery with a gallery of table styles you can use. The design that really caught my eye was UV Green which seems to make the rows pop visually. I created a GridView using the Northwind products table and applied this table style to the GridView without setting any of the web control styles. Since the page layout does not use a table I am free to style tables as I want. This page is using a "Green Theme" with a different background stripe and color scheme. You can see the shaded border better in this image.

I also have a blue theme so I needed to create a blue version of the UV Green table style. To find a harmonious color for the table rows I used an online color scheme generator http://www.colorschemegenerator.com/index-black.htm

 Another table style that I liked was Media Groove and it was easy to replace my previous table design but I did not bother to create a theme with a similar color scheme.

The ASP.NET logo with the reflection was created in Adobe Illustrator. You can find a detailed tutorial at http://www.bittbox.com/illustrator/tutorial-web-20-logo-reflection-in-vector-format-with-illustrator/

My next project will be to try different button designs with web controls. Most of my experiments have involved CSS because that is all you will find on web design sites.

When you get really good at design you can rename your business to "web development boutique". It sounds more frilly. 

Today I explored some alternative JavaScript coding methods to make it easier to dynamically create DOM elements. I used jQuery and the jQuery Plugin, FlyDOM, to work with the Amazon web service API. I bought the book Amazon.com Mashups by Francis Shanahan from the Computer Books Direct club which is closing and selling books cheap. I have not read the book yet but there is a chapter on converting Amazon data directly into JSON using XSLT (hmm, interesting) and it actually covers several other APIs and the technology behind mashups. There was some mention of Amazon wish lists which attracted my attention because I use wish lists to save books that I come across which I may want to buy some day. In other words, this is personal data that I am actually interested in.

My goal was to find a better way to generate dynamic content. Using the DOM functions createElement(), createTextNode(), and appendChild() was getting to be a tedious chore. When you throw in attributes that can be set with the setAttribute() function, you wind up with 3 or 5 lines for every HTML tag. Even a bold tag requires a lot more code than it is worth. Fortunately, jQuery with the FlyDOM plugin can reduce this code to a single line to create a single tag. 

Here are some things to note about the code:

1. Using a jQuery plugin is just a matter of pulling in an additional file of JavaScript.
2. You need to sign up for an Amazon Access Key and there is a small fee for using their web services.
3. The query string in the web address for the web service had a lot of ampersands which I replaced with the | character. I'm still using my xml2json generic handler.
4. FlyDOM allows you to create a table using JSON syntax. This isn't very easy but it is good practice.
5. No sample code was provided for dynamically creating table rows in a loop. I imagine you could do it better by treating it as an array expressed in JSON.
6. Troubleshooting dynamically generated DOM elements is difficult because viewing the source doesn't show you anything. This is where Nikhil Kothari's Web Development Helper proves to be invaluable because it has a DOM Inspector that reveals the dynamically generated elements.

 The table below the DIV was dynamically generated and will not appear in the HTML source view.

I've been working a lot with JavaScript lately so I decided to check out all the JavaScript libraries that are available. Many of these libraries have useful utility methods and modules for JSON and AJAX. There are six libraries that I intend to explore; the Microsoft AJAX Client Library (I need to see what was added in 3.5), the Prototype JavaScript Framework, the Dojo Toolkit, MochiKit, and the Yahoo! User Interface Library (YUI), and jQuery.

What prompted my interest in JavaScript libraries were some simple client scripting problems that proved to be unexpectedly time consuming. For example, I needed to format a ISO 8601 date string "2008-01-11T02:34:42.177Z" into a more readable date format. There are many different date formats being used by RSS feeds, web application APIs, web services, etc. I was hoping to find a library with conversion methods for all the standard date formats but none of these 6 libraries could convert that particular date string into a JavaScript Date object.

Another frequent bit of JavaScript that I need to write may be simplied by using jQuery. A compiled help file has problems getting a reference to background image files that are specified in CSS. To work around that problem I need to write some JavaScript to find the element based on the class name or id and set its style.background property. jQuery has a method to get all the elements with a particular class name and set a property for them all with just one statement and no loops.

Most of the JavaScript I'm writing is for my project documentation using compiled help files or help collections. It is possible to use these JavaScript libraries within help files. You just add the javascript file as a project item and it gets compiled in with all the web pages, images, and style sheets. So far I have only baked in jQuery, Prototype, and the Microsoft AJAX Client Library because there was just one file to include. I can even use two libraries within one page as long as the namespaces do not clash. Most of the disadvantages to using so many JavaScript libraries do not apply to help files. There are no browser compatibility issues because a help file always uses Internet Explorer. There are no server side considerations. The JavaScript libraries do not need to be downloaded.

I'm starting to use Microsoft Visual Web Developer 2008 Express Edition to code my JavaScript experiments. I was using Ultra-Edit or Microsoft Expression Web but Visual Web Developer 2008 gives me some Intellisense for the JavaScript libraries. Since I am only concerned with Internet Explorer compatibility I use the Web Developer Helper for script debugging rather than Firebug, unless I need additional clues to solve an error.

I've been using StumbleUpon to find interesting web sites about web development and yesterday I found a site that will generate a poster of your CSS.  It creates an image that resembles a class diagram. I think it is a great way to document your style sheets. One of my style sheets produced such a massive and confusing diagram that I think I'll show it to my client so I'll look like a genius.

Tonight I finally solved my problems with Visual Studio. I had both the Standard version and the Professional version installed. It was the Standard version that was trying to repair itself. I uninstalled the service pack for Visual Studio 2005 Standard version and I have not seen the Windows Installer start up since.

Today I learned how to create a tag cloud using ASP.NET 2.0. I may try to create tag clouds for YouTube vloggers using the Google API for YouTube. I'm really involved in the YouTube community because it is the most effective social networking site. Anyone who thinks of YouTube as just a video sharing site is missing its true significance. Many vloggers now feel unappreciated by the company that runs the web site. Some users have even threatened to create their own web sites. I fear they'll be posting ridiculous projects on RentACoder to clone YouTube for $500. However it did get me to thinking that it would be interesting to work on a social networking site. But if you want to keep on top of new developments in social networking then you should investigate social broadcasting. .NET Rocks should consider creating a show on one of the many social broadcasting sites. It creates a very strong sense of community and a lot of the audience do work in the IT industry.

I was working on converting an ASP.NET 1.1 web site to ASP.NET 2.0 today but I will probably regret it because I have been having problems with Visual Studio 2005 and Microsoft Visual Web Developer 2008 Express Edition. Ever since I have installed Microsoft Visual Web Developer 2008 Express Edition I've been finding the msiexec.exe (Windows Installer) process running and bogging down my PC. This occurs every now and then when I'm using either Visual Studio 2005 or the Microsoft Visual Web Developer 2008 Express Edition. Below is a screenshot of the culprit in Process Explorer where you can see it hogging the CPU.

I've tried running Visual Studio in safe mode but that does not solve the problem. Then I tried re-installing a few things like the Slickedit Gadgets. Today I completely reinstalled Microsoft Visual Web Developer 2008 Express Edition. I am not having this problem at work. It only affects my home computer. As far as I can figure it out, Visual Studio is trying to repair something. It does not matter what project I am working on. I can't get any work done without Process Explorer open so I can kill the msiexec.exe process.

Last night I finally finished the improvements to my xml2json generic handler which were required to develop a Stickam widget. Since I am sending the URL of the feed as a query string value to the generic handler it was necessary to deal with ampersands in the feed's URL without screwing up the query string. So I added a line to replace the | character with the & character. If I have a feed with an ampersand in the URL I'll just send it with the | character instead. Trace listeners don't seem to work in generic handlers so I added code to write the feed URL and the JSON string to a log file for debugging purposes. The Stickam players in the XML were causing invalid JSON syntax because they were not getting null values. I am not interested in that information so I used RegexBuddy to come up with the regular expression to replace them with empty strings. I came across the infamous "invalid label name" error so I added parentheses around my JSON string. The last thing I did was add XML comments for the method that actually converts the XML to JSON because this code should be replace when a more reliable algorithm becomes available:

To practice my web design skills I added rounded corners and a diagonal stripe background to the widget. This was purely a coding exercise because I've already developed a Yahoo! Widget for the same purpose. So far I have used this xml2json generic handler for the following; to get my dynamic IP address, to get the weather from the National Weather Service, and to get postage rates from USPS (all using web services that return XML). I have confirmed that this will also work in a compiled HTML file so you can include widgets in your documentation provided you can run the xml2json generic handler on a web server.

I have pre-ordered a new book, Professional Web Widgets with CSS, DOM, JSON and Ajax which promises to cover the topic in depth. I hope it has extensive coverage of JavaScript Object Notation because my other books on AJAX tend to skim over JSON.

Yesterday I found a useful tool for checking the syntax of JavaScript Object Notation strings: http://www.raboof.com/Projects/JsonChecker/ I needed that because I'm trying to create a widget for Stickam that will work in my compiled help file. I'm using my xml2json generic handler to convert XML to JSON and apparently it generated some invalid JSON. This tool helped me to narrow down the location of the syntax error.

I need to make some improvements to my xml2json generic handler. First it needs to surround the JSON string with parentheses to prevent the infamous invalid label error. Second it needs to output the JSON string to a text file so I can debug errors caused by invalid syntax (trace listeners don't seem to work in generic handlers). And third it needs to use regular expressions to replace bad JSON syntax with an empty string. I should probably build in the JSON Checker code and return an error message about bad syntax.

I have come across the JavaScriptSerializer class in the System.Web.Script.Serialization namespace but it cannot serialize XML or DataSets.

Working with JavaScript Object Notation is currently quite painful. I think programmers need better tools for working with JSON. We need tools to check the syntax and visualize the data or objects it represents because JSON strings are very cryptic.