Every so often I started seeing very slow response times and in some cases the WCF app didn’t respond at all and at the same time the w3wp.exe process was sitting at very high CPU usage 90-100%. This started happening under high load, and to get the application to start responding again i needed to restart IIS.
I attached WinDbg and got the following Call Stacks from the offending threads:
------------------------------------------------------------------------------------------------------------------------------------------
OS Thread Id: 0x820 (27)
Child SP IP Call Site
00000000151dd0c0 000007fe997d1c1a System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)
00000000151dd130 000007fe997d1a84 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].TryGetValue(System.__Canon, System.__Canon ByRef)
00000000151dd170 000007fe9ab24d79 Spring.Aop.Framework.HashtableCachingAdvisorChainF actory.GetInterceptors(Spring.Aop.Framework.IAdvis ed, System.Object, System.Reflection.MethodInfo, System.Type)
00000000151dd1d0 000007fe9b2b0966 CompositionAopProxy_f0785c6734574f8ca838bd53706b74 9b.GetPageSourcePermission(System.String, Int64)
00000000151dd260 000007fe9b2b0807 XXXXX.XXX.UP.BLL.Impl.GetPageSourcePermission(Syst em.String, Int64)
00000000151dd2b0 000007fe9a0f641d DynamicClass.SyncInvokeGetPageSourcePermission(Sys tem.Object, System.Object[], System.Object[])
00000000151dd300 000007fe9b0225a4 System.ServiceModel.Dispatcher.SyncMethodInvoker.I nvoke(System.Object, System.Object[], System.Object[] ByRef)
00000000151dd4f0 000007fe9b0188d8 System.ServiceModel.Dispatcher.DispatchOperationRu ntime.InvokeBegin(System.ServiceModel.Dispatcher.M essageRpc ByRef)
00000000151dd8a0 000007fe9b017fe0 System.ServiceModel.Dispatcher.ImmutableDispatchRu ntime.ProcessMessage5(System.ServiceModel.Dispatch er.MessageRpc ByRef)
00000000151dd950 000007fe9b016f37 System.ServiceModel.Dispatcher.ImmutableDispatchRu ntime.ProcessMessage31(System.ServiceModel.Dispatc her.MessageRpc ByRef)
00000000151ddb30 000007fe9b013c12 System.ServiceModel.Dispatcher.MessageRpc.Process( Boolean)
00000000151ddbf0 000007fe9b0122b7 System.ServiceModel.Dispatcher.ChannelHandler.Disp atchAndReleasePump(System.ServiceModel.Channels.Re questContext, Boolean, System.ServiceModel.OperationContext)
00000000151ddfd0 000007fe9af3d343 System.ServiceModel.Dispatcher.ChannelHandler.Hand leRequest(System.ServiceModel.Channels.RequestCont ext, System.ServiceModel.OperationContext)
00000000151de050 000007fe9af3c907 System.ServiceModel.Dispatcher.ChannelHandler.Asyn cMessagePump(System.IAsyncResult)
00000000151de0d0 000007fe9af220d2 System.Runtime.Fx+AsyncThunk.UnhandledExceptionFra me(System.IAsyncResult)
00000000151de120 000007fe9a82d39b System.Runtime.AsyncResult.Complete(Boolean)
00000000151de1a0 000007fe9af3c5f1 System.Runtime.InputQueue`1+AsyncQueueReader[[System.__Canon, mscorlib]].Set(Item<System.__Canon>)
00000000151de1f0 000007fe9af20f6c System.Runtime.InputQueue`1[[System.__Canon, mscorlib]].EnqueueAndDispatch(Item<System.__Canon>, Boolean)
00000000151de360 000007fe9af2072e System.Runtime.InputQueue`1[[System.__Canon, mscorlib]].EnqueueAndDispatch(System.__Canon, System.Action, Boolean)
00000000151de3d0 000007fe9af3c399 System.ServiceModel.Channels.SingletonChannelAccep tor`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Enqueue(System.__Canon, System.Action, Boolean)
00000000151de430 000007fe9af3b987 System.ServiceModel.Channels.HttpPipeline+EnqueueM essageAsyncResult.CompleteParseAndEnqueue(System.I AsyncResult)
00000000151de4a0 000007fe9af3b7e2 System.ServiceModel.Channels.HttpPipeline+EnqueueM essageAsyncResult.HandleParseIncomingMessage(Syste m.IAsyncResult)
00000000151de4d0 000007fe9af3b6b8 System.Runtime.AsyncResult.SyncContinue(System.IAs yncResult)
00000000151de520 000007fe9af25250 System.ServiceModel.Channels.HttpPipeline+EmptyHtt pPipeline.BeginProcessInboundRequest(System.Servic eModel.Channels.ReplyChannelAcceptor, System.Action, System.AsyncCallback, System.Object)
00000000151de570 000007fe9af22ab0 System.ServiceModel.Channels.HttpChannelListener`1 +HttpContextReceivedAsyncResult`1[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].ProcessHttpContextAsync()
00000000151de600 000007fe9af21bfa System.ServiceModel.Channels.HttpChannelListener`1[[System.__Canon, mscorlib]].BeginHttpContextReceived(System.ServiceModel.Chan nels.HttpRequestContext, System.Action, System.AsyncCallback, System.Object)
00000000151de660 000007fe9af2038b System.ServiceModel.Activation.HostedHttpTransport Manager.HttpContextReceived(System.ServiceModel.Ac tivation.HostedHttpRequestAsyncResult)
00000000151de700 000007fe9ac12527 System.ServiceModel.Activation.HostedHttpRequestAs yncResult.HandleRequest()
00000000151de790 000007fe9ac11fb5 System.ServiceModel.Activation.HostedHttpRequestAs yncResult.BeginRequest()
00000000151de840 000007fe9ac11db3 System.ServiceModel.Activation.HostedHttpRequestAs yncResult.OnBeginRequest(System.Object)
00000000151de890 000007fe9ac11951 System.Runtime.IOThreadScheduler+ScheduledOverlapp ed.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
00000000151de8f0 000007fe9ac11850 System.Runtime.Fx+IOCompletionThunk.UnhandledExcep tionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
00000000151de950 000007fe9ac11276 System.Threading._IOCompletionCallback.PerformIOCo mpletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
00000000151deb08 000007fef8dbf713 [GCFrame: 00000000151deb08]
00000000151decd8 000007fef8dbf713 [DebuggerU2MCatchHandlerFrame: 00000000151decd8]
00000000151deeb8 000007fef8dbf713 [ContextTransitionFrame: 00000000151deeb8]
00000000151df0a8 000007fef8dbf713 [DebuggerU2MCatchHandlerFrame: 00000000151df0a8]
----------------------------------------------------------------------------------------------------------------------------
Order to stacks infomation, the method HashtableCachingAdvisorChainFactory.GetInterceptor s causing the high CPU is that the FindEntry method walks through the dictionary, trying to find the key. If multiple threads are doing this at the same time, especially if the dictionary is modified in the meantime you may end up in an infinite loop in FindEntry causing the high CPU behavior and the process may hang.
Is this a known problem or a bug? please help me to fix this problem.