IIS based Web Service - Delayed Response

After a long break updating my blog, I'm back.
I had some interesting issues in the last few month and I hope I'll have the time posting them here....

I'll begin with the oldest one.
Few months back one of the developers in the development team was complaining about a very slow response times from a web service he developed and was running in our Test environment.
He told me that the problem only occurs in the first request, or on the first request after long time (30 mins or above), but he cannot be exact and can't reproduce it.
The normal response time is about 1-2 seconds but to the first one, which takes almost 30!
He also told me that there is nothing 'heavy' or complex in the initialization process and he suspects that the problem is in IIS or the CLR...

I followed his directions and the first thing came in my mind is the "Idle timeout" that can be configured to IIS Application Pools. Surprisingly it was actually configured to 20 mins! I turned it off (to never shutdown an idle application pool).

After couple of hours he came back complaining that the problem still occurs.
My next step was to look for some ASP.NET configuration options in web.config and machine.config but I found nothing really related...

I had no direction and the only thing in my mind was that maybe something is wrong with the server(IIS or .NET Framewok), so I set up a virtual server with other OS version(Wndows 2003 Web Edition) and tried causing the problem to appear again.

I noticed that the only way to reproduce the behavior was to boot the server - Killing the worker process or restarting the IIS didn't trigger it.

Now with the option to reproduce in my hands, I could really start digging. I started (Microsoft's) System Internals' Process Explorer and watched the w3wp.exe of the WebService (w3wp.exe is the instance of the "Application Pool" you see in IIS). I checked to see if it was working and using CPU and not just waiting or hang on something. I drilled down to its threads tring to detect where the problem is. If the problem was in his code I could see it, but it was happening very long time before his code was invoked. I saw some threads waiting on a method named CompareAssemblyIdentity (a method used to "compares two assembly identities to determine whether they are equivalent"). Few moments later I noticed some new child processes running under the worker process, named csc.exe (C# Compiler) and they (in their turn) created another process as well !! Only then I realized the problem! ASP.NET is Recompiling the classes for the first time the Web Service is used! It is that simple.

Boot is not the only trigger for ASP.NET recompilation and it wasn't the case of the poor developer as well. The other trigger is much more relevant to his case - changes in the code. He just forget telling me he was still working on it and publishing his code relative frequently to the test server...

Kernel Memory leaks - Part 2

After a while not able to reproduce the case I've managed to get it again and copy all the data from Poolmon. The total of Paged-pool shown in the upper-right corner says 207MB but when summing each allocation for each tag (only Paged of curse) I got 270MB !!

Still have no answer.

Kernel Memory Leaks

For the last couple of years we are getting alot of servers' hangs ,due to both Paged and Non-Paged Pool depleted. I've managed to trace the leaking applications and either it an application we wrote or a third party application - closing it (killing it's process) just frees part of the leaked handles.

When monitoring with poolmon I saw the handles getting freed but not as they should have been.
Lets say that the total memory those handles were taking is 100MB in paged pool and the total use of paged pool was 150MB. Killing the process that created them should free them all right? - Wrong. While killing the process freed 90MB of the handles, the total should have dropped to 60MB, but its not! There seems to be unlisted handles either by purpose or by mistake/bug of the OS.

I intend to get to the bottom of this.

How to debug .net services ?

You wrote new windows service in .Net and you want to debug or run it from within the Visual Studio environment (just as you do when debugging console app) ? You want to debug a service that has been already installed and runs through the SCM(Service Control Manager)?

In this post I will cover two methods to debug a service developed in .Net. Each has it purpose.

Method 1:
After searching the web a bit and found no good way to debug a .Net service in a natural way, I did what I was familiar with since the Visual Studio 6 days - Add main() to the project.
Just add the following code to the service class:

static void Main(string[] args)
{
ServiceBase[] servicesToRun;
servicesToRun = new ServiceBase[] { new Service1() };
((Service1)servicesToRun[0]).OnStart(args);
}

Now just put a break point on that method and hit F5.

Method 2:
Microsoft's way (described in the MSDN) to debug a service is to add a Sleep(30) in the OnStart() method of the service class and attach the debugger within these 30 secs. This is a problem - every time we start that service we'll need to wait those 30 sec! They sovles it by telling you to create 2 compiled versions of the same service. One with Sleep and the other without it. So why should we use this way if we can debug from the Visual Studio? A simple reason is when you experience different behavior when the service runs as a real service (through the SCM).
What if we could tell the service when we want it to wait 30 secs when it starts? This is a very good example for the use of the args that are in the OnStart(string[] args) .
Add the following to the beginning of the OnStart method:

for (int i = 0; i < args.Length; i++)
{
if (args[i].ToLower() == "/debug")
{
System.Threading.Thread.Sleep(30 * 1000);
}
}

Now after the service has been installed, run services.msc and double click the service.

On the Start parameters field just write "/debug" and hit start.
Now you have a single project that allows you to attach to it's OnStart whenever you need.

Enjoy!

First post

Hey my name is Shmuel Krakover and I'm a System Programmer. For the last couple of years I've encountered an enormous number of issues all the way from

coding issues (ASP, JSP, C++, .NET ...)

to Application/Web servers issues (IIS, OAS, MTS)

and recently to OS related issues (Windows).

In my world there is no problem that cannot be solved.
Actually, any problem must be solved and even more, the cause for it must be completely understood. This means that I get allot reading architectures of platforms, whether its about MDAC, IIS or even as low as the OS itself.

Those technologies will probably change with time, so I'll have to deal with new problems in new areas and I'm about to share them because I'm absolutely sure that those problems are not mine only.

I hope this blog will be usefull for everyone.