McAfee Antivirus with IIS6.0 and ASP.NET Performance

A scenario came up at work when I went to roll the latest project up to production from our QA environment.
 
Our application was the first .NET 2.0 application to be built in this environment. The high-level executives had been pushing for a transition of all development to Java. A few managers had managed to convince the executives to let us take this particular application into .NET 2.0. You can imagine my suprise after 6 months of development, I roll to production and performance was awful. It had never been great in our QA environment, but it always ran at an acceptable pace. It did however seem to run great on our development boxes. I chalked it up to our QA environment not being up to par with our development boxes or our production servers (Mistake #1). After going through and hunting down the worst performing methods using Red-Gate ANTS .NET Profiler, I revamped the code and pushed a new release to a production server not in service yet and performed some load tests. Performance had improved but was still in the red zone.
 
After a long discussion with the server hosting team, I began to question the production environments. I knew they were running VMWare instances, and I thought perhaps that could be degrading performance. But why was it quicker on our QA servers then our production servers? Those too are VMWare instances. Weeks went by where I was on the phone and exchanging emails with the hosting team trying to figure it out. I needed help. I talked with some other members of our development team who had launched a .NET 1.1 application on similar VMWare instances, and they had mentioned running into issues with McAfee Antivirus and the ScriptProxy.dll issue (http://www.kbalertz.com/kb_917848.aspx).
 
A meeting was scheduled in which we would go have a sit down with the server hosting guys and take a look at the configuration of the servers. In the meantime, I gathered some load tests using Microsoft’s Application Center Test application on a Dell desktop machine. Our application performed quite well on that machine, and that wasn’t even server hardware. When I got to the meeting and started poking around on the server, I saw McAfee Antivirus running. One of the server hosting guys said that ScriptProxy.dll wasn’t registered, and that couldn’t be the issue. I urged them to disable McAfee Antivirus for a few minutes just to run some tests, but after a long argument, I finally caved in (Mistake #2).
 
We looked over everything else and it all seemed to be setup just fine. We left four hours later with heavy heads. As we were leaving, the hosting team sent some memory dumps over to Microsoft to take a look. I feared the worst. He’ll find something wrong with my code, I can see it now. We grabbed a quick bite to eat for lunch on the way home at a local deli and tried to imagine every scenario we could, and what we would do next. I got back to the office and went to tell the project manager the bad news…
 
My phone rang mid-sentance and I recognized the number. I interrupted the project manager and answered it. It was one of the guys who had gone to the hosting team with us, who was still in the car on the way back to his office. He said Microsoft had called… McAfee was injecting itself into the worker processes.
 
The hosting guys had disabled McAfee Antivirus on one of the servers. I could not have typed faster if my life depended on it. Response times were immaculate. Sub-second page responses. It was like a dream come true. I could hear him laughing in his car. He was the one who had gone to the executives to convince them to give us a chance to do it in .NET. We had done it. Moral of the story, watch what those Antivirus applications do. In this particular case, it had almost destroyed two seperate applications. The one done by other members of my team, and the one I had done. I still don’t know if McAfee has released a patch for this issue. If they do, I’ll post a link here.