Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: Connection Caching

  1. #1
    Join Date
    Mar 2008
    Posts
    108

    Default Connection Caching

    I'm trying to use cached connections but can't seem to figure out what I'm doing wrong.

    Here's an excerpt from my configuration:

    Code:
    	<object id="mqConnFactoryNoCache" type="Apache.NMS.ActiveMQ.ConnectionFactory, Apache.NMS.ActiveMQ">
    		<constructor-arg index="0" value="tcp://localhost:61616"/>		
    	</object>
    	<object id="mqConnFactory" type="Spring.Messaging.Nms.Connections.CachingConnectionFactory, Spring.Messaging.Nms">
    		<constructor-arg name="targetConnectionFactory" ref="mqConnFactoryNoCache"/>
    		<property name="SessionCacheSize" value="10"/>
    	</object>
    	<object id="nmsTemplate" type="Spring.Messaging.Nms.Core.NmsTemplate, Spring.Messaging.Nms">
    		<constructor-arg name="connectionFactory" ref="mqConnFactory"/>
    		<property name="DefaultDestinationName" value="application.default"/>
    		<property name="Persistent" value="true"/>
    		<property name="ReceiveTimeout" value="3000"/>
    		<property name="SessionAcknowledgeMode" value="ClientAcknowledge"/>
    	</object>
    And here's the exception I get when I use SendAndConvert() from NmsTemplate

    Code:
    Apache.NMS.ActiveMQ.ConnectionClosedException: The connection is already closed!
       at Apache.NMS.ActiveMQ.MessageProducer.CheckClosed() in c:\project\3rdparty\asf\activemq-dotnet\Apache.NMS.ActiveMQ\trunk\src\main\csharp\MessageProducer.cs:line 114
       at Apache.NMS.ActiveMQ.MessageProducer.Send(IDestination destination, IMessage message, Boolean persistent, Byte priority, TimeSpan timeToLive, Boolean specifiedTimeToLive) in c:\project\3rdparty\asf\activemq-dotnet\Apache.NMS.ActiveMQ\trunk\src\main\csharp\MessageProducer.cs:line 141
       at Apache.NMS.ActiveMQ.MessageProducer.Send(IDestination destination, IMessage message) in c:\project\3rdparty\asf\activemq-dotnet\Apache.NMS.ActiveMQ\trunk\src\main\csharp\MessageProducer.cs:line 126
       at Apache.NMS.ActiveMQ.MessageProducer.Send(IMessage message) in c:\project\3rdparty\asf\activemq-dotnet\Apache.NMS.ActiveMQ\trunk\src\main\csharp\MessageProducer.cs:line 121
       at Spring.Messaging.Nms.Connections.CachedMessageProducer.Send(IMessage message) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Connections\CachedMessageProducer.cs:line 77
       at Spring.Messaging.Nms.Core.NmsTemplate.DoSend(IMessageProducer producer, IMessage message) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 575
       at Spring.Messaging.Nms.Core.NmsTemplate.DoSend(ISession session, IDestination destination, IMessageCreator messageCreator, MessageCreatorDelegate messageCreatorDelegate) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 545
       at Spring.Messaging.Nms.Core.NmsTemplate.DoSend(ISession session, IDestination destination, IMessageCreator messageCreator) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 511
       at Spring.Messaging.Nms.Core.SendDestinationCallback.DoInNms(ISession session) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 1479
       at Spring.Messaging.Nms.Core.NmsTemplate.Execute(ISessionCallback action, Boolean startConnection) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 187
       at Spring.Messaging.Nms.Core.NmsTemplate.Send(String destinationName, IMessageCreator messageCreator) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 736
       at Spring.Messaging.Nms.Core.NmsTemplate.ConvertAndSend(String destinationName, Object message) in c:\project\3rdparty\spring.net\src\Spring\Spring.Messaging.Nms\Messaging\Nms\Core\NmsTemplate.cs:line 783
    The application has multiple producers, consumers, and queues. Is there anything obviously wrong with the above?

  2. #2
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,

    Smells like a bug, despite those unit tests! I've created a JIRA issue and will have some time tonight to take a look.

    Cheers,
    Mark

  3. #3
    Join Date
    Mar 2008
    Posts
    108

    Default Test case

    I have a project that demonstrates the issue. I don't know if you will find it helpful but I'll include it here.

    There is a Producer.cs class that puts a message on a queue then sleeps. The Consumer receives the message then sleeps.

    Using the normal connection factory or the SingleConnectionFactory, it works fine. Switching to the CachingConnectionFactory leads to the original error.

    In our case we are currently using the standard connection factory. Thankfully our message volume is low and we can afford the inefficiency (for now). We tried using the SingleConnectionFactory and it works for some amount of time (hours or more), but then it seems like some of the consumers get stuck. The admin console shows there are pending messages and registered consumers, but no delivery takes place. I have not determined if this is a Spring.NET problem, but I suspect we are doing something on our side to "poison" the connection and make it unusable. If we do get in such a state basically the producers/consumers are blocked. The service continues to run but no message activity takes place and so far I have not found anything amiss in the logs.

    I'm hoping that if we can get the CachingConnectionFactory working we can do additional testing and see which of our components is misbehaving.
    Attached Files Attached Files

  4. #4
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,
    Thanks! What version of activemq are you using?
    Mark

  5. #5
    Join Date
    Mar 2008
    Posts
    108

    Default

    I'm using ActiveMQ version 5.1.0

    We are testing a variety of configurations:

    Windows XP SP3 (local development only)
    Windows Server 2003
    Ubuntu 8.04 Server

    with

    MySQL 5.0 Community Server
    PostgreSQL 8.3
    AMQ Store (the default persistence adapter)

    I believe we saw the same results with any of the above combinations, but I know for sure Windows XP with AMQ store yields the error.

  6. #6
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,
    I didn't get a chance to look into the issue, however SingleConnectionFactory is incredibly simple so I suspect the issue maybe with ActiveMQ. As it takes hours to reproduce it seems, I hate to bug you, but since I don't have the time today to set that up and let it bake, if you re-write the example using straight-up NMS API, and cache the connection, I suspect you will have the same issue. How about trying a 5.2 snapshot?
    Cheers,
    Mark

  7. #7
    Join Date
    Mar 2008
    Posts
    108

    Default

    My apologies, I was not clear enough in the original report.

    The test case I posted can be used to reproduce the problem with no delay. The error message occurs within seconds, at most-- probably within the first second.

    That issue involves the CachingConnectionFactory, and since the error occurs every time and immediately you should be able to reproduce it easily by running that solution against AMQ.

    The second half of the original post was discussing a separate problem-- using the SingleConnectionFactory gave us issues after "hours" of use. However, I'm not ready to point the finger yet at Spring.NET or AMQ because I feel the most likely problem is that our multi-threaded service is deadlocked or otherwise "churning" and that there is not an actual problem with the queues.

  8. #8
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,
    Yea, I was referring to the second problem...thanks for the test case w.r.t to the first.
    Mark

  9. #9
    Join Date
    Mar 2008
    Posts
    108

    Default

    I'll give it the old college try.

    The problem is intermittent, which makes writing a test case slightly more difficult. So far we have observed the problem most often when the system is under moderate to heavy load. We are using queueing in our load process, which starts with several dozen GB of text files and end up with millions of rows in the database. The messages themselves are small (just filenames) and there are a few thousand per load that move from queue to queue.

    Is there a known practical limit to the number of producers/consumers can share a single connection? Also, I'm using client acknowledge but some of the asynchronous receivers wait in a Thread.Sleep() loop until a condition is true, at which point the message is processed. I have a theory that might be part of the problem.

  10. #10
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,
    Thanks mate for the ol' college try i don't know of what the practical limits in activemq are for the number of session/consumers/producers per connection but it should be quite high and it doesn't sound like you are using that many. If you are doing very long processing tasks before doing the client ack, I can see how that might cause some issues in as much as that is probably not that good practice to wait very long. The possible retry of an long running data-load (due to error) should be separated from the fact that the message was delivered successfully to the process.
    Mark

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •