Logging ASP.NET Application Shutdown Events

Someone on a listserv recently asked whether there was a way to figure out why and when ASP.NET restarts application domains.  Specifically, he was looking for the exact cause of what was triggering them on his application in a production shared hosted environment (was it a web.config file change, a global.asax change, an app_code directory change, a directory delete change, max-num-compilations reached quota, \bin directory change, etc). 

 

Thomas on my team has a cool code-snippet that he wrote that uses some nifty private reflection tricks to capture and log this information.  It is pretty easy to re-use and add into any application, and can be used to log the information anywhere you want (the below code use the NT Event Log to save it – but you could just as easily send it to a database or via an email to an admin).  The code works with both ASP.NET V1.1 and ASP.NET V2.0.

 

Simply add the System.Reflection and System.Diagnostics namespaces to your Global.asax class/file, and then add the Application_End event with this code (note: you can also download a .zip file containing the code from here):

 

public void Application_End() {

    

    HttpRuntime runtime = (HttpRuntime) typeof(System.Web.HttpRuntime).InvokeMember("_theRuntime",

                                                                                    BindingFlags.NonPublic

                                                                                    | BindingFlags.Static

                                                                                    | BindingFlags.GetField,

                                                                                    null,

                                                                                    null,

                                                                                    null);

   

    if (runtime == null)

        return;

   

    string shutDownMessage = (string) runtime.GetType().InvokeMember("_shutDownMessage",

                                                                     BindingFlags.NonPublic

                                                                     | BindingFlags.Instance

                                                                     | BindingFlags.GetField,

                                                                     null,

                                                                     runtime,

                                                                     null);

   

    string shutDownStack = (string) runtime.GetType().InvokeMember("_shutDownStack",

                                                                   BindingFlags.NonPublic

                                                                   | BindingFlags.Instance

                                                                   | BindingFlags.GetField,

                                                                   null,

                                                                   runtime,

                                                                   null);

   

    if (!EventLog.SourceExists(".NET Runtime")) {

        EventLog.CreateEventSource(".NET Runtime", "Application");

    }

   

    EventLog log = new EventLog();

    log.Source = ".NET Runtime";

    log.WriteEntry(String.Format("\r\n\r\n_shutDownMessage={0}\r\n\r\n_shutDownStack={1}",

                                 shutDownMessage,

                                 shutDownStack),

                                 EventLogEntryType.Error);

}

 

I tried this out using a simple web-site using ASP.NET 2.0 and the built-in VS Web-Server (aka Cassini).  When I changed the web.config file in my running application, the following was logged to my "Application" event viewer:

_shutDownMessage=CONFIG change

HostingEnvironment caused shutdown

CONFIG change

CONFIG change

_shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

at System.Environment.get_StackTrace()

at System.Web.HttpRuntime.ShutdownAppDomain()

at System.Web.Hosting.HostingEnvironment.ShutdownThisAppDomainOnce()

at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state)

at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state).

 

When I updated my Global.asax file with some code change, the following was logged:

_shutDownMessage=Change in GLOBAL.ASAX

Change in GLOBAL.ASAX

Change in GLOBAL.ASAX

HostingEnvironment caused shutdown

_shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

at System.Environment.get_StackTrace()

at System.Web.HttpRuntime.ShutdownAppDomain()

at System.Web.HttpApplicationFactory.OnAppFileChange(Object sender, FileChangeEvent e)

at System.Web.DirectoryMonitor.FireNotifications()

at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)

at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state)

at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state).

 

And when I changed the contents of my \bin directory I got:

_shutDownMessage=Change Notification for critical directories.

bin dir change or directory rename

HostingEnvironment caused shutdown

Directory rename change notification for 'E:\Unload'.

Unload dir change or directory rename

_shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

at System.Environment.get_StackTrace()

at System.Web.HttpRuntime.ShutdownAppDomain()

at System.Web.Hosting.HostingEnvironment.ShutdownThisAppDomainOnce()

at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state)

at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading.

Hopefully this is a useful trick that you can re-use in your own applications to get better visibility into what is going on with them.  If you are using ASP.NET 2.0, then you should definitely also investigate the new ASP.NET 2.0 Health Monitoring feature-set.  This provides a rich eventing architecture for instrumenting your code, as well as raising notifications of issues to admins when they occur within your application.  K. Scott Allen has written a good overview article of how this new feature-set works, and links to the MSDN documentation.

 

Hope this helps,

 

Scott

39 Comments

  • Great trick. Thanks



    Pawel

  • Thanks, Scott. This is very useful. I did not know I could access private members via reflection -- I had assumed that only public and protected members were available. This opens a lot of doors that I deemed closed :-).

  • This is cool stuff, Scott! I didn't know about it. Thanks!

  • This is awesome! Thanks a lot for sharing it. I always have a sneaking suspicion that some app pools are thrashing and I don't know about it.

  • I tried this at my hosted site and received no data in shutDownMessage or shutDownStack. Worked locally though. Must be a permission issue.



    Thanks for sharing. Cool ideal.



    Cheers,

    Mark

  • Hi Mark,



    You do need permissions to make private reflection calls (which I believe can be turned off via code-access security) -- so that might be an issue depending upon how your security is setup.



    Hope this helps,



    Scott

  • I like this! I going to get this implemented right away. Good lookin' out.

  • Hi, awesome trick!

    It made a tip of the day on my developers community website. Simply great!

  • Hey Scott, I tried implementing the code and when I ran IISreset from the comand the application_End did not fire...



    however Changing the web.config did work

    Is this expected behavior?



    Is there anyway to trap this information?



    Thanks

  • Useful stuff! Thanks for sharing :)



    Do you know if it would give meaningful feedback (pointing clearly to the cause) if the worker process in IIS6 is being recycled due to anti-virus software "touching" the worker processes exe file?



    Just something I experienced at a clients site causing user session information to be lost regularly. Was driving me crazy trying to figure out the cause! :P

  • You are not alone in being hit by anti-virus software touching files on disk! I've seen this with several people now.



    I haven't actually seen a case where anti-virus software modifies the actual w3wp.exe process file though. Usually the cause I've seen is with them modifying web.config or machine.config files (they update the timestamp). The above trick will definitely log that information and help you pinpoint it as the cause.



    Hope this helps,



    Scott

  • Hi,     My web app (ASPNET 2.0) was reseting when I delete a sub-folder     ("TestFolder").     When my web app runs over aspnet 1.1 it doesn=B4t reset.     Are there some documentation about this?

  • Hi Mahdi,

    This post goes into more detail about the application shutdown semantics of deleting a directory in ASP.NET 2.0: http://blogs.msdn.com/toddca/archive/2005/12/01/499144.aspx

    Hope this helps,

    Scott

  • It is really nice articale....and helping all the developers.

  • Hi Scott,

    I'm curious as to how this is different from including a healthMonitoring section in your web.config and adding a rule for "Application Lifetime Events"? It will automatically log all app domain restarts with a Reasons enumeration.

    Thanks,
    N.

  • Hi Nariman,

    Health Monitoring is very useful and something you want to enable. Unfortunately, though, I believe it doesn't always log the application shutdown reason (it will in the future -- but doesn't with the released V2.0 bits).

    The above code will capture the reason though for you.

    Hope this helps,

    Scott

  • Thanks a million for this pointer.

    I'm trying to translate this to VB and cannot get a successful reference to the HttpRuntime object. Keep getting NullReferenceException (which doesn't even appear unless i wrap the statement in a try/catch).

    I'm not terribly fluent in C#. How do I translate the phrase:

    (HttpRuntime) typeof(System.Web.HttpRuntime).InvokeMember(...

    to VB??

    I've tried a bunch of flavors of cast like:

    dim runtime as HttpRuntime = DirectCast(runtime, System.Web.HttpRuntime).GetType.Invoke...

    or
    CType(runtime, System.Web.HttpRuntime).Gettype...

    That C# syntax is apparently doing something else? Could I trouble you for a quick hint on the conversion?

    Many thanks,

    Phil

  • Hi Phil,

    Any chance you could send me email on this if you are still having a problem? I can then try and see if I can port a VB version for you.

    Thanks,

    Scott

  • it seems you just need enable healthMonitoring , it will tell you the reason why an application shutdown

  • Hi Markus,

    If the application pool restarts, it could be that the application_end event doesn't get a chance to execute (this sometimes happens if the worker process needs to be shutdown immediately).

    The other thing to check is to make sure you aren't assuming that HttpContext is available in either the Session_End or Application_End events. I've seen a few cases where people are trying to use it here -- and it isn't available and so throws an exception. It could be that you have a runtime error in these events which is why it isn't showing up as logged.

    Hope this helps,

    Scott

  • This is quite interesting.
    Are there similar stacks available for say session end, etc?

  • Hi Chris,

    When sessions end we raise the Session_End event within Global.asax. That is probably the best way to log them.

    Hope this helps,

    Scott

  • Scott,
    I can't get it to work under my existing global.asax....I added the contents of the asax included in the zip file, but am not allowed to use two application directives such as "". The existing directive is: "". Help! Cause it looks very helpfull getting the info this way!!

    Thanks,
    Keston

  • Hi Keston,

    Can you send me an email containing your global.asax source code to look at, as well as the exact error you are seeing?

    Thanks,

    Scott

  • Hi there. When I log Application_End events, I usually notice 2-3 log entries per server within a second of each other in our production environment. Any ideas as to why this might be, rather than Application_End being called just once?

    Thanks,
    Chris

  • what email address can i use?

  • Hi Chris,

    Do you have 2-3 applications running at the same time? It could be that you are updating a global setting (like the machine web.config file) which resets all the apps at the same time.

    Hope this helps,

    Scott

  • Hi Keston,

    My email address is: scottgu@microsoft.com

    Thanks,

    Scott

  • Hi Scott.

    Thanks for the reply. It's amazing the thought process your simple question sparked in me:
    "Do you have 2-3 applications running at the same time?"

    I actually have 3 iis websites all pointing at the same directory, and sure enough, that is why I was getting 3 app_end events. Go figure.

    On a different note, any ideas on my other problem? Basically, where do find more info when you just get a generic "Shutdown Message: HostingEnvironment caused shutdown Shutdown Stack" message?

    Thanks a million for the help.
    -Chris

  • Hi Chris,

    Could you check a couple of things:

    1) Are you updating any files on disk in the application when the shutdown happens?

    2) Do you have any anti-virus software running on the system that might end up touching the timestamps of files (which could cause applications to auto-restart)?

    3) Do you have any timeouts set on your IIS worker proceses? By default I believe these are set to recycle every 20 minutes or so.

    Thanks,

    Scott

  • Hi, my query is not directly related to this topic.

    I want to check is there any way we can figure out what is the amount of memory consumed by the worker process?

  • Hi Gaurav,

    You could use the System.Diagnostics namespace to obtain Process information about any Process running in Windows. You could then use this to lookup the amount of memory used.

    Hope this helps,

    Scott

  • Hi,
    my asp.net application ends abruptly and redirects to the login page. I tried logging the _shutdown message and the _shutdowntrack message, but when i reviewed the log file both the messages were empty? can you help me why are they not trapped by the runtime?

  • Hi Scott,

    What happens on AppDomain remainging idle for any length of time? Also, is there any way to stop AppDomain unload through inactivity?

  • Hi Surjit,

    You can configure ASP.NET to shutdown the app domain once a ceretain about of inactivity has elapsed. I believe you configure this using the attribute in your config file: http://msdn.microsoft.com/en-us/library/e1f13641.aspx

    Hope this helps,

    Scott

  • I am wondering how to tie this event into an HttpModule.
    public void Application_End() {....}

    In my case, for example, I have a rootsite, and 100's of child applications.

    From my testing, it would appear that I need a global.asax file in each individual application, as opposed to definining one HttpModule in my top level web.config.

    It is not realistic or efficent for me to duplicate the global.asax file, so I am trying to attach this event from an httpmodule, but I am not finding any way to do so.

    Anyone have an idea?

    Perhaps I am missing something with regards to the global.asax.

    Child apps pick up on Config (with it's own config instance, but reads the top level), that is about it, they do not pick up Provider instances, or Global. each child app appears to get it's own instances of everything.

  • Hi scott,
    i tried your code out in my sample application,but getting shutDownMessage and shutDownStack strings as null though am getting the runtime.Could you pls tell me were i went wrong.
    thanks

  • Great thing. I did R&D on this code and find lot of amazing things.

    I will post these all afer some more R&D.

    Thnx Scott

  • I have also been experiencing application shutdowns and it seems that it occurs like a minute after we upload files (e.g a PDF).

    Is this something related to the anti-virus or the fact the upload folder is under the application root?

    Would it help if I store the files under a different domain/server?

Comments have been disabled for this content.