Erwin's Blog

Developing with .NET

November 2008 - Posts

How to detect search engine crawlers?


Today I was looking for a solution how to detect when a client is a search engine crawler, you can create a fancy solution for this, but in the .NET framework their is already a solution to detect search engine crawler. The property Request.Browser.Crawler. If you use this property you always get false even if the site is visited by a search engine crawler, that's because it's not configured in a default installation of .NET.

ASP.NET uses the <browsercaps> section in machine.config or web.config to determine the client browser is a crawler or not. In the default installation the crawler filter information is all blank,  that's why you'd always get false. To fix this problem, you should add the search engine crawler filters in the <browsercaps> and add this section to your web.config. Like this:

  1. <configuration>
  2.  
  3. <!-- ...... -->
  4.  
  5.   <system.web>
  6.  
  7.     <browserCaps>
  8.  
  9.       <filter>
  10.  
  11.        <!-- Google Crawler -->
  12.         <case match="Googlebot">
  13.           browser=Googlebot
  14.           crawler=true
  15.         </case>
  16.  
  17.         <!-- Yahoo Crawler -->
  18.         <case match="http\:\/\/help.yahoo.com\/help\/us\/ysearch\/slurp">
  19.           browser=YahooCrawler
  20.           crawler=true
  21.         </case>
  22.        
  23.         <!-- MSN Crawler -->
  24.         <case match="msnbot">
  25.           browser=msnbot
  26.           crawler=true
  27.         </case>
  28.        
  29.         <!-- check Alta Vista (Mercator) -->
  30.         <case match="Mercator">
  31.           browser=AltaVista
  32.           crawler=true
  33.         </case>
  34.  
  35.         <!-- check Slurp (Yahoo uses this as well) -->
  36.         <case match="Slurp">
  37.           browser=Slurp
  38.           crawler=true
  39.         </case>
  40.        
  41.         <!-- Baidu Crawler -->
  42.         <case match="Baiduspider">
  43.           browser=Baiduspider
  44.           crawler=true
  45.         </case>
  46.  
  47.         <!-- check Excite -->
  48.         <case match="ArchitextSpider">
  49.           browser=Excite
  50.           crawler=true
  51.         </case>
  52.  
  53.         <!-- Lycos -->
  54.         <case match="Lycos_Spider">
  55.           browser=Lycos
  56.           crawler=true
  57.         </case>
  58.  
  59.         <!-- Ask Jeeves -->
  60.         <case match="Ask Jeeves">
  61.           browser=AskJeaves
  62.           crawler=true
  63.         </case>
  64.  
  65.         <!-- IBM Research Web Crawler -->
  66.         <case match="http\:\/\/www\.almaden.ibm.com\/cs\/crawler">
  67.           browser=IBMResearchWebCrawler
  68.           crawler=true
  69.         </case>
  70.  
  71.       </filter>
  72.  
  73.     </browserCaps>
  74.  
  75.   </system.web>
  76.  
  77.  </configuration>


Tip: you can find more crawler info in your IIS logs ([Windows Folder]\system32\LogFiles)

From


New ASP.NET Charting Control

Microsoft has released a new cool charting control. It comes with documentations, samples and a lot of features, features like:

  • All supported chart types.
  • Data series, chart areas, axes, legends, labels, titles, and more.
  • Data Binding
  • Data manipulation, such as copying, splitting, merging, alignment, grouping, sorting, searching, filtering, and more.
  • Statistical formulas and financial formulas.
  • Advanced chart appearance, such as 3D, anti-aliasing, lighting, perspective, and more.
  • Chart rendering.
  • Events and Customizations.
  • Interactivity and AJAX.

Below some pictures of sample charts:

 More information:

Posted: Nov 27 2008, 09:12 AM by erwin21 | with no comments
Filed under: ,
Parse QueryString with the HttpUtility

If you have a querystring from for example the Uri class (Uri.Query) and you want to parse it so you can get al the params and values, instead of doing al lot of string manipulation you can use a very handy utility class, the HttpUtility. The HttpUtility class has a method ParseQueryString, this method parse a query string into a namevaluecollection. Like this:

  1. string querystring = "?var1=test&var2=test2"
  2.  
  3. NameValueCollection nvColl = HttpUtility.ParseQueryString(querystring);
  4.  
  5. string var1 = nvColl["var1"];
  6. string var2 = nvColl["var2"];

 The string "var1" contains "test" and the "var2" contains "test2".

Posted: Nov 21 2008, 12:45 PM by erwin21 | with 4 comment(s)
Filed under: ,
VS2008 SP1 Hotfix to Support for intellisense Doc Files

A few week I mentioned how to get intellisense for jQuery. Now the visual studio web tools team released a hotfix so the intellisense files will be found automatically if they have the same name as the orginal js file and the suffix "-vsdoc.js" more info can be found on the Visual Studio Web Tools team blog

Parse a html string with ASP.NET controls


Today I searched for a solution to parse a html string with ASP.NET controls on runtime. A string like for example this:

  1. <table>
  2.     <tr>
  3.         <td>Name</td>
  4.     </tr>
  5.     <tr>
  6.         <td><asp:TextBox id="name" runat="server" /></td>
  7.     </tr>
  8. </table>

Now I want to parse this string a runtime into a Control object, this can be done with the ParseControl method from the TemplateControl class, the ParseControl methods can be accessed from the Page instance, like this for example:

  1. string html = "<table><tr><td>Name</td></tr><tr><td><asp:TextBox id=\"name\" runat=\"server\" /></td></tr>b</table>"
  2.  
  3. Control ctrl = Page.ParseControl(html);
  4.  
  5. somePlaceHolder.Controls.Add(ctrl);

After the html has been parsed the control can be added to for example a placeholder.
Posted: Nov 06 2008, 08:51 PM by erwin21 | with 8 comment(s)
Filed under: ,
Visual Studio Window Layout

Today I was playing around with my windows in Visual Studio to get them in place exactly how want it,, but how do I save my window layout settings? I found it on Sara Ford's weblog, simply close Visual Studio and the window layout will be saved.

When you mess-up your windows your can reset them by pressing "Window" -> "Reset Window Layout".

Posted: Nov 05 2008, 11:27 AM by erwin21 | with no comments
Filed under:
More Posts