Scott Forsyth's Blog

Postings on IIS, ASP.NET, SQL Server, Webfarms and general system admin.

Cloud Resources

IIS Resources

Reading a memory.dmp or other .dmp file

While the dreaded Blue Screen of Death (BSOD) occurs less frequently with newer versions of Windows than it did in years past, there are still times when the BSOD reveals itself. 

I just ran into four BSOD’s on two Windows Server 2012 machines and I had the ‘opportunity’ to analyze a memory.dmp file today, so I thought I would post quick instructions on how to get a handy summary of the memory dump.

I’ve had this ”I Found a Fix” debugging page bookmarked for years and I’ve used it many times, so I need to give full credit to ifoundafix for their helpful steps.  The only change I have below is to include updated paths.

It’s possible to debug remotely, and you may have requirements to do that.  My quick instructions here are for local debugging.  The debugging tools are very stable and if you install just what you need then they are small and a quick install, so running this on a production machine is generally safe, but you must make that decision for your particular environment.

This can be accomplished with 7 easy steps:

Step 1. Obtain and install the debugging tools.  The links do change over time, but the following link is currently an exhaustive page which includes Windows Server 2012 and Windows 8 Consumer debugger tools, Windows 7, Vista, XP and Windows Server 2003.

Debugging Tools Windows

All you need to install is the “Install Debugging Tools for Windows as a Standalone Component (from Windows SDK)” and during the install only select "Debugging Tools for Windows".  Everything else is used for more advanced troubleshooting or development, and isn’t needed here.  Today I followed the link to “Install Debugging Tools for Windows as a Standalone Component (from Windows SDK)” although for a different OS you may need to follow a different link.

Step 2. From an elevated command prompt navigate to the debugging folder. For me with the latest tools on Windows Server 2012 it was at C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\.  You can specify the path during the install.

Step 3. Type the following:

kd –z C:\Windows\memory.dmp (or the path to your .dmp file)

Step 4. Type the following:

.logopen c:\debuglog.txt

Step 5. Type the following:

.sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols

Step 6. Type the following:

.reload;!analyze -v;r;kv;lmnt;.logclose;q

Step 7. Review the results by opening c:\debuglog.txt in your favorite text editor.  Searching for PROCESS_NAME: will show which process had the fault.  You can use the process name and other information from the dump to find clues and find answers in a web search.  Usually the fault is with a hardware drivers of some sort, but there are many things that can cause crashes so the actual analyzing of the dump may take some research.

Often times a driver update will fix the issue.  If the summary information doesn’t offer enough information then you’ll need to dig further into the debugging tools or open a CSS case with Microsoft.  The steps above will provide you with a summary mostly-human-readable report from the dump.  There is much more information available in the memory dump although it gets exponentially more difficult to track down the details the further you get into windows debugging.

Hopefully these quick steps are helpful for you as you troubleshoot the unwelcome BSOD.

Last Day With OrcsWeb

It’s hard to believe that it’s been 10 years since my first day at OrcsWeb. Today is my last official day, but I’ll still be close by. I have a number of ties here, including being a customer through Vaasnet.

So much has changed in this time. Ten years ago I began working for OrcsWeb from Canada. Nine years ago I moved my family down here to North Carolina and assumed the role of Director of Technology.  I was able to be a part of the company as it grew in staff, servers, customers, and reputation. I feel honored to be a part of OrcsWeb during these exciting years.

During my time at OrcsWeb I have been given opportunities to attend conferences, meet and become friends with top technical experts in the field, write articles, co-author two books, and speak at conferences and code camps. It was through OrcsWeb that I was given opportunities to be active in the community, to become a Microsoft MVP and an ASPInsider.

I’m grateful to Brad and Karla Kingsley who have always treated me like more than an employee. They have always encouraged me to grow and to pursue my dreams.

I’m thankful to Jeff Graves who has been accommodating to my evolving schedule and less than full time availability.  And in terms of technical smarts, Jeff tops the list!  And I’m also thankful of the rest of the team at OrcsWeb who are experts in the field, and with whom it’s always been a privilege to work.

Moving forward, I have two main focuses. I’ll be able to spend more time on Vaasnet (a company I co-founded with Jeff Widmer) to see the company position itself further in the market and to strengthen both the product and the brand.  Additionally, I’m working in a part time basis with Dynamicweb an established CMS and eCommerce company in Europe who is just moving into the US market. Dynamicweb has a strong product already and I’m excited to work with the leadership team in the US. Expect to see more of Dynamicweb in the coming months and years.

I just want to reiterate a big thanks to OrcsWeb for helping write such an important chapter in my life. And it’s with excitement that I look forward to the next chapter of my life.

Introducing Testing Domain - localtest.me

Save this URL, memorize it, write it on a sticky note, tweet it, tell your colleagues about it! 

localtest.me (http://localtest.me)

and

*.localtest.me (http://something.localtest.me)

If you do any testing on your local system you’ve probably created hosts file entries (c:\windows\system32\drivers\etc\hosts) for different testing domains and had them point back to 127.0.0.1.  This works great but it requires just a bit of extra effort.

This localtest.me trick is so obvious, so simple, and yet so powerful.  I wouldn’t be surprised if there are other domain names like this out there, but I haven’t run across them yet so I just ordered the domain name localtest.me which I’ll keep available for the internet community to use.

Here’s how it works. The entire domain name localtest.me—and all wildcard entries—point to 127.0.0.1.  So without any changes to your host file you can immediate start testing with a local URL.

Examples:

http://localtest.me
http://newyork.localtest.me
http://mysite.localtest.me
http://redirecttest.localtest.me
http://sub1.sub2.sub3.localtest.me

You name it, just use any *.localtest.me URL that you dream up and it will work for testing on your local system.

This was inspired by a trick that Imar Spaanjaars introduced me to. He created a loopback wildcard URL with his company domain name.  I took this one step further and ordered a domain name just for this purpose.

I would have liked to order localhost.com or localhost.me but those domain names were taken. So to help you remember, just remember that it’s ‘localtest’ and not ‘localhost’, and it’s ‘.me’ rather than ‘.com’.

I can’t track usage since the domain name resolves to 127.0.0.1 and never passes through my servers, so this is just a public tool which I’ll give to the community. I hope it gets used. And, since I can’t really use the domain name to explain itself, please spread the word and tell others about it.

Some examples on how to use it would include:

  • Creating websites on your dev machine.  site1.localtest.me, site2.localtest.me, site3.localtest.me.
  • Great for URL Rewrite (IIS) or mod_rewrite (Apache) testing: redirect.localtest.me, failuretest.localtest.me, subdomain.localtest.me, city1.localtest.me.
  • Any testing on your local system where a friendly URL would be useful.

I hope you enjoy!

Google and Geo-location, CNDs, DNS Load Balancing-Week 50

You can find this week’s video here.

This week answers two Q&A questions from viewers. DNS Load Balancing and then some discussion and a walkthrough using Application Request Routing (ARR) for a Content Delivery Network (CDN).

There’s a growing movement towards Content Delivery Networks (CDN); fronting web farms and geographically dispersing websites. This week I continue with Q&A’s from viewers, taking questions on DNS Load Balancing and CDNs.

Question 1:

I would love to see some clever DNS load balancing (not sure what capability windows is offering). Flesik

Question 2a:

I would love to see an end-2-end CDN redundant network setup (dns balancing, ARR nodes, parent notes etc). Flesik

Question 2b:

I would be interested in seeing a series on building an ecdn or ecn using ARR and what the best practices would be to scale it out geographically.

It seems ARR is sold that way... really not sold but talked about. I have tried to put my theory out and try it but i just don't know the best way to route my clients to their designated locations. Can you help out with an awesome weblog maybe? Adam

 

The following URL is the one I mentioned in the video: http://learn.iis.net/page.aspx/649/deploying-application-request-routing-in-cdn

In this week’s video we look at DNS load balancing and geo-location issues that Google faces by using DNS to determine a user’s location. We also take a look at using Microsoft Application Request Routing (ARR) to create a CDN.

This is week 50 of a 52 week series for the web pro. You can view past and future weeks here: http://dotnetslackers.com/projects/LearnIIS7/

You can find this week’s video here.

What’s new in IIS8, Perf, Indexing Service-Week 49

You can find this week’s video here.

This week I'm taking Q&A from viewers, starting with what's new in IIS8, a question on enable32BitAppOnWin64, performance settings for asp.net, the ARR Helper, and Indexing Services.

This week we look at five topics.

Pre-topic:
We take a look at the new features in IIS8. Last week Internet Information Services (IIS) 8 Beta was released to the public. This week's video touches on the upcoming features in the next version of IIS. Here’s a link to the blog post which was mentioned in the video
Question 1:

In a number of places (http://learn.iis.net/page.aspx/201/32-bit-mode-worker-processes/, http://channel9.msdn.com/Events/MIX/MIX08/T06), I've saw that enable32BitAppOnWin64 is recommended for performance reasons. I'm guessing it has to do with memory usage... but I never could find detailed explanation on why this is recommended (even Microsoft books are vague on this topic - they just say - do it, but provide no reason why it should be done). Do you have any insight into this? (Predrag Tomasevic)

Question 2:

Do you have any recommendations on modifying aspnet.config and machine.config to deliver better performance when it comes to "high number of concurrent connections"? I've implemented recommendations for modifying machine.config from this article (http://www.codeproject.com/KB/aspnet/10ASPNetPerformance.aspx - ASP.NET Process Configuration Optimization section)... but I would gladly listen to more recommendations if you have them. (Predrag Tomasevic)

Question 3:

Could you share more of your experience with ARR Helper? I'm specifically interested in configuring ARR Helper (for example - how to only accept only X-Forwards-For from certain IPs (proxies you trust)). (Predrag Tomasevic)

Question 4:

What is the replacement for indexing service to use in coding web search pages on a Windows 2008R2 server? (Susan Williams)

Here’s the link that was mentioned: http://technet.microsoft.com/en-us/library/ee692804.aspx

This is now week 49 of a 52 week series for the web pro. You can view past and future weeks here: http://dotnetslackers.com/projects/LearnIIS7/

You can find this week’s video here.

What’s New in IIS 8

With the beta release of Windows Server 8 today, Internet Information server (IIS) 8 is available to the public for testing and even production workload testing.  Many system administrators have been anxious to kick the tires and to find out which features are coming.

I’ll include a high level overview of what we will see in the upcoming version of IIS.  The focus with this release of IIS 8 is on the large scale hoster.  There are substantial performance improvements to handle thousands of sites on a single server farm—with ease.  Everything that I mention below is available for download and usage today.

Forgive me if there are typos.  I’m writing this while at the MVP Summit in Seattle while trying to listen to another session at the same time.  Thanks to the IIS team who gave detailed demos on this yesterday and gave me permission to talk about this.

Real CPU Throttling

Previous versions of IIS have CPU throttling but it doesn’t do what most of us want.  When a site reaches the CPU threshold the site is turned off for a period of time before it is allowed to run again.  This protects the other sites on the server but it isn’t a welcome action for the site in question since the site breaks rather than just slowing down. 

Finally in version IIS 8 there are kernel level changes to support real CPU Throttling.  Now there are two new actions for sites that reach the CPU threshold.  They are Throttle and Throttle under load.  If you used WSRM to achieve this in the past, you no longer need to do so, and the functionality is improved over what is available with WSRM.

The throttle feature will keep the CPU for a particular worker process at the level specified.  Throttling isn’t applied to just the primary worker process, but it also includes all child processes, if they happen to exist.

The Throttle under load feature will allow a site to use all possible CPU if it’s available while throttling the worker process if the server is under load.

The throttling is based on the user and not specifically on the application pool. This means that if you use dedicated users on more than one app pool then it throttles for all of app pools sharing the same user identity. Note that the application pool identity user is unique so if you use the app pool identity user—which is common—then each app pool will be throttled individually.

This is a welcome new feature and is nicely implemented.

SSL Scalability

Unless you deal with large scale site hosting with many SSL certificates you may not have realized that there is room for improvement in this area. 

Previous versions of IIS have limited secure site density.  Each SSL site requires its own IP address and after adding a few SSL sites, startup performance becomes slow and the memory demand is high.  Every certificate is loaded into memory on the first visit to an SSL site which creates a large memory footprint and a long delay on the first load. 

In IIS 8 the SSL certificate count is easily scalable to thousands of secure sites per machine with almost instantaneous first-loads.  Only the certificate that is needed is loaded and it will unload after a configurable idle period.  Additionally, enumerating or loading huge numbers of certificates is substantially improved.

SNI / SSL Host Header Support

Using host headers and a shared IP address with SSL certificate has always been problematic.  IIS 8 now offers Server Name Indication (SNI) support which allows many SSL sites to share the same IP.  SNI is a fairly new feature (within the last few years) which allows host headers to work with SSL. It does this by carrying the target host name in the TLS handshake rather than the encrypted part of the packet.

IIS 8 makes SNI support a first class citizen in the site bindings.

Note that SNI doesn't work on all browsers. For example, Internet Explorer in Windows XP does not support SNI.  Read more about that from Eric Law's blog post. Over 85% of browsers is use today support SNI, but since it's not 100%, it will not work universally. However, like the adoption issue with host headers in the '90s, it will a fully supported before we know it. More details with a list of browsers can be found here: http://en.wikipedia.org/wiki/Server_Name_Indication

This sets the stage for sharing IP addresses which is extra important as ipv4 IPs become more valuable and consolidation of IPs becomes the trend. 

SSL Manageability - Central Certificate Store (CCS)

In IIS 7 managing SSL is labor intensive, particularly for server farms.  All certificate must be imported on every machine in the farm.  When setting up new servers you must account for time needed to import certificates when scaling out, and even on small server farms.  In previous versions keeping certificates in sync between servers is difficult to manage and often requires manual steps.

In IIS8 there is a new Central Certificate Store (CCS).  Central Certificate Store allows storing certificates on a central file share instead of each machine.  You can point the servers to a single network share, or use replication like DFS-R to sync the folders between machines.

Renewal and syncing is as simple as xcopying pfx files to the location that you specify when enabling CCS on the web server.  Enabling CCS is straight forward too.  It works very similar to enabling Shared Configuration.

CCS compliments the SNI functionality to support sites with multiple certs and a single IP.

The mapping of bindings to certificates uses a bit of magic … by convention rather than configuration. This is important for extremely large lists of certificates. Now you don't need to select them from a huge list. The value of the host header needs to match the name of the cert. Your CCS folder will have many .pfx files with names that match the domain name.  Basically the name of the .pfx files in the certificate store is the primary key.

If you use a wildcard cert then it needs to be named _.domain.com.pfx.

As you would assume, there is support for Multiple Domain Certificates (Unified Communications Certificate [UCC]). If you use multiple domain certificates using the subjectAltName feature of the certificate then you just create multiple copies of the pfx, one for each subjectAltName.

Note that you can use the old method which binds to by certificate identifier and it works the same as it did in the past.

Furthermore there is a neat feature for the central repository that allows grouping by expiration date, which groups by "Today / This Week / Next Week / Next Month / Later" which is handy for seeing which certificates are ready to expire.

With these changes to the certificates, it makes for a powerful solution for large scale webfarm hosting with multiple tenants.

Dynamic IP Restrictions

Information about this is already available on the web, but it's moving along and getting closer for the final release.

FTP Logon Restriction

Yay. A new FTP IP Restrictions module is coming! This is similar in concept to Dynamic IP Restrictions for HTTP. One of the key differences is that it does gray listing rather than black listing. When someone is blocked, they are only blocked for the sample period (e.g. 30 seconds). This is nice because it's enough to thwart or slow brute force and common name password attacks, but legit invalid attempts can continue to attempt to log in without waiting for long periods of time.

What's extra nice about having this feature is that you can set it slightly more sensitive than your domain username lockout policy so that brute force attacks don't cause your username to be locked out from too many invalid attempts. The FTP IP Restrictions can throttle the hack attempts without locking out your domain users.

Application Initialization Module

Previously known as the application warm-up module which was pulled for a time, now it's ready in full force as Application Initialization Module.

This allows spinning up sites and pages before traffic arrives and handling of requests in a friendly way while the application first loads. It's not uncommon for a site to take a minute or longer on the first load (yes SharePoint admins, we feel your pain).  This allows you to protect the end user from being the person that triggers this.

It's possible to set a warm-up page at the server level as a single setting, or you can use powerful URL Rewrite rules for more flexibility.

You can also ensure that your load balancer’s health test page doesn’t serve up a valid response until the site is fully initialized according to your preferences.  Then the load balancer will bring a node into rotation only after the entire warm-up has completed.

Configuration Scale

The IIS configuration files (e.g. applicationHost.config) can handle very large files with ease now.  There are substantial performance improvements in the upcoming version. Only administrators with large numbers of sites on the same server or server farm (think thousands) would have noticed before, but for large scale performance the new changes are paving the way for huge scale.

Web Sockets

It’s important to include Web Sockets in this list too.  Apart from some brief information I really haven’t looked into Web Sockets in detail yet so I’ll just include a great link from Paul Batum on it.  Web Sockets does require Windows 8 or later on the server side. 

All in all these are welcome changes.  While previous versions of IIS already did a great job of handling massive amounts of traffic, IIS 8 now can handle thousands (or tens of thousands) of sites and their extensive configurations on a single server farm.  With HTTP and FTP logon restrictions, CPU throttling, the Application Initialization Module, and large scale SSL and configuration improvements, IIS 8 brings a number of welcome improvements.

IIS FTP Troubleshooting-Week 48

You can find this week’s video here.

This lesson covers ways to troubleshoot IIS FTP. When it works, it works well, but if you run into issues getting an FTP account working it can sometimes be difficult to resolve. This video will help you understand some helpful tricks and it will walk you through ways to isolate and resolve the issue.

Over the last five weeks we’ve been looking at IIS FTP. See the list below to jump to a specific FTP topic.  This week we explore some troubleshooting techniques and review the following FTP connectivity stack.

  • DNS Resolution/Network Connectivity
  • Firewall Access (Passive/Active / Secure?)
  • IIS Bindings
  • Authentication
  • Authorization
  • Isolation Mode / File paths
  • NTFS Permissions

There were two external resources which I referenced. They are:

This is now week 48 of a 52 week series for the web pro and it is the final of a 5-week mini-series on IIS FTP. The five weeks include:

You can find this week’s video here.

Posted: Feb 21 2012, 10:58 AM by OWScott | with no comments
Filed under: , , ,
FTP Firewall Settings, Active vs. Passive, and FTPS Explicit vs. Implicit-Week 47

You can find this week’s video here.

Have you ever wondered what FTP Active mode or Passive mode means? Do you have a good understanding of the FTP data channel or control channel? It can be difficult to fully understand FTP, which firewall ports to enable, and how to navigate the two communication channels. This lesson will hopefully clear up these questions and more.

This week’s video lesson takes a deep dive into FTP Active vs. Passive modes. As part of this you’ll get a chance to see the various modes in action, see what the traffic looks like in Wireshark, see exact firewall rules, learn about stateful FTP, find out about Explicit FTPS and Implicit FTPS, and learn about the FTP data channel and control channels.

This week's video lesson is the 4th of a 5-week mini-series on IIS FTP. The five weeks include:

  • Week 1: IIS FTP Basics
  • Week 2: IIS FTP and IIS Manager Users
  • Week 3: IIS FTP and User Isolation
  • Week 4: IIS FTP Firewall settings, Active vs. Passive
  • Week 5: IIS FTP Troubleshooting plus FTP Host Headers

This is now week 47 of a 52 week series for the web pro, and the 4th of a 5 week mini-series on IIS FTP. You can view past and future weeks here: http://dotnetslackers.com/projects/LearnIIS7/

You can find this week’s video here.

Flush IIS HTTP and FTP Logs to Disk

Today I wanted to find a way to flush the IIS FTP logs on-demand.  The logs for IIS FTP flush to disk every 6 minutes, and the HTTP logs every 1 minute (or 64kb).  This can make troubleshooting difficult when you don’t receive immediate access to the latest log data.

After looking everywhere I could think of, from search engine searches to perusing through the IIS schema files, I figured I had better go to the source and ask Robert McMurray.

Sure enough, Robert had the answer and even wrote a blog post in response to my question with code examples for four scripting/programming languages (C#, VB.NET, JavaScript, VbScript).

There is not a netsh or appcmd solution though, so the scripting or programming options are the way to do it.  Actually, you can also flush the logs by restarting the Microsoft FTP Service (ftpsvc) but, as you would assume, it will impact currently active FTP sessions.

This blog post serves three purposes. 

  1. It’s a reference pointing to Robert’s examples
  2. I’ll include how to do the same for the HTTP logs
  3. I’ll provide a PowerShell example which I based on Robert’s examples

1. The reference is mentioned above already, but to give me something useful to write in this paragraph, I’ll include it again. Programmatically Flushing FTP Logs.

2. For HTTP there is a method to flush the logs using netsh.

netsh http flush logbuffer

This will immediately flush the HTTP logs for all sites.

3. The FTP logs can be done from PowerShell too.  Here’s a script which is the PowerShell equivalent of Robert’s examples.  Just update $siteName, or pass it as a parameter to the script.

Param($siteName = "Default Web Site") 
 
#Get MWA ServerManager
[System.Reflection.Assembly]::LoadFrom( "C:\windows\system32\inetsrv\Microsoft.Web.Administration.dll" ) | Out-Null
$serverManager = new-object Microsoft.Web.Administration.ServerManager 
 
$config = $serverManager.GetApplicationHostConfiguration()
 
#Get Sites Collection
$sitesSection = $config.GetSection("system.applicationHost/sites")
$sitesCollection = $sitesSection.GetCollection() 
 
#Find Site
foreach ($item in $sitesCollection){ 
 
    if ($item.Attributes.Item("Name").Value -eq $siteName){
        $site = $item
    }
}
#Validation
if ($site -eq $null) { 
    Write-Host "Site '$siteName' not found"
    return
}
 
#Flush the Logs
$ftpServer = $site.ChildElements.Item("ftpServer")
 
if (!($ftpServer.ChildElements.Count)){
    Write-Host "Site '$siteName' does not have FTP bindings set"
    return
}
 
$ftpServer.Methods.Item("FlushLog").CreateInstance().Execute()

I hope one of these programming/scripting options come in handy for times when you want immediate access to the latest FTP log data.

IIS FTP User Isolation-Week 46

You can find this week’s video here.

I’ve been looking forward to releasing this week’s video.  IIS FTP User isolation is an interesting topic because it offers a lot of power and flexibility but it’s not very intuitive because of how it’s managed.

This week we walk through the five isolation modes to gain a full understanding of the IIS FTP method of configuration for user isolation.

IIS FTP is a powerful application, but some of the flexibility is hidden through a unique convention based method of management. It’s easy to miss the fact that IIS FTP allows the ability to have multiple users who can be directed to different folders and be fully isolated from each other. For example, you can have a designer1 who has access to the whole site while designer2 has access to just project1 only, while—if you set it up correctly—you can feel confident that designer2 can’t gain more access than they are allowed.

IIS FTP requires understanding a few core principles to manage it effectively and to ensure that you don’t overlook key security settings that would allow users to gain more access than they should. IIS FTP 7.5 offers five different isolation modes, each of which targets a different situation.

This is now week 46 of a 52 week series for the web pro, and the 3rd of a 5 week mini-series on IIS FTP. You can view past and future weeks here: http://dotnetslackers.com/projects/LearnIIS7/

Also, if you’re reading this early enough, I’m taking questions for the last couple weeks of the series.  Read more about it here.

You can find this week’s video here.

More Posts « Previous page - Next page »