“Yama-Hama! It's fright night!” #1
Since today is Halloween I thought I’d share a scary ghost bug story that I ran into this week. What better way to get everyone into the Halloween spirit?
Background:
- Running a load test, using one webtest, to gauge how the application handles heavy loads for a very specific scenario
Problem:
- Application is throwing an average of 4 exceptions/sec per user spiking CPU usage
Story:
- Ran load test and confirmed baseline of ~ 4 exceptions/sec per user on average
- Reviewed code and saw that Response.Redirect(url) was being used
- Replaced Response.Redirect(url) with Response.Redirect(url, false) and ran load test
- Saw improvement to ~3 exceptions/sec per user
- Added break point in custom error handler, to see the exceptions being caught, and ran load test
- Caught a FileNotFound exception with no valuable information
- Checked the stack trace and saw “System.Web.StaticFileHandler”
- Guessed that there must have been a problem loading a .css or image file which are static
- Used ieHttpHeaders and saw a funky GET: GET /~/foo/bar.css
- Fixed the style sheet reference and ran load test
- Saw improvement to ~2 exceptions/sec per user
- Ran load test again
- Caught an EventValidation exception
- Observed custom error handler throwing a NullReferenceException because Server.GetLastError() was null but there was no check before calling Server.GetLastError.GetBaseException()
- Added null check
- Ran load test and saw improvement to ~1 exception/sec per user
- Disabled EventValidation on pages and ran load test and saw improvement to ~0 exceptions/sec per users
- Enabled EventValidation on pages because I didn’t like the solution and began investigating
- 30mins later –I was baffled
- Manually ran the webtest and did not get any EventValidation exceptions
- Eureka! moment – checked webtest and saw that the ThinkTime between requests was set to 0 causing the next page request to be called before the first page request had finished loading
- Changed ThinkTime between request to 5sec
- App now had ~0 exceptions/sec per user
Ok, so it wasn’t that scary but it was painful! Happy Halloween!