Using a mixture of synchronous and asynchronous messaging

[keenan]

If an application makes use of both synchronous and asynchronous messaging, for example by calling NmsTemplate.Receive() methods and also having one or more listener containers, then it is not possible to use the same CachingConnectionFactory instance for both. Is the a correct statement?

The reason I make this claim is based on errors I see in our logs. As I mentioned in a different forum post, the ActiveMQ Connection class does not lock on SyncRoot in any of the methods that enumerate the sessions collection.

In a multi-threaded service if one synchronous consumer thread is in the CreateSession() method and hits the sessions.Add() line while another thread is in the Start() method in the loop that is enumeratins sessions, an error occurs. In fact, the error is swallowed by AbstractListenerContainer:



Ignoring Connection start exception - assuming already started
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.Collections.ArrayList.ArrayListEnumeratorSi mple.MoveNext()


This was happening quite often in our case during service startup while the context was initializing objects.

The easy and straightforward solution was to create two CachingConnectionFactory instances in the configuration and use one for the asynchronous listeners and use the other for NmsTemplate.

Is there any downside or problems with that approach?

[Mark Pollack]

Hi,
The Java JMS spec states that the ConnectionFactory, Connection, and Destination should support concurrent usage and the expectation would be that .NET APIs based on this would support that level (or more) of concurrent usage directly in the APIs. Given what you mention and that I saw issues related to threading were raised on the ActiveMQ JIRA, this sounds like an ActiveMQ issue. Do you know if the fixes for that issue worked in this case?
Cheers,
Mark

[keenan]

I agree this is an NMS issue. I did try version 1.1 with the threading fixes but it caused more problems than it solved. When version 1.1 is closer to release I'll try again.

In the meantime version 1.0 of NMS is stable for us as long as:

1) singleton="false" on CachingConnectionFactory
2) singleton="false" on NmsTemplate
3) After starting a listener container, moving messages from the dead letter queue back to the input queue (this is automated)

I don't see any of those as onerous and we have been running for a solid week without queue problems.

[Mark Pollack]

Hi,

Setting those to singleton=false isn't great but you just end up using more resources than you should, effectively giving you a connectionfactory/connection/session/producer per instance of NmsTemplate. The default number of sessions to create is 1, so don't change the property SessionCacheSize to a higher value otherwise you will have created extra sessions for no purpose.

I didn't yet reply to the issues regarding ack mode, but what settings are you running which such that you move stuff back from the dead letter queue upon startup? Is that just to get basic behavior to work or is it for truly exceptional circumstances?

Cheers,
Mark

[keenan]

you just end up using more resources than you should, effectively giving you a connectionfactory/connection/session/producer per instance of NmsTemplate


Exactly. I don't know if the NMS 1.1 thread fixes will be back-ported to 1.0, but in 1.0 the Connection is documented as not thread safe so I think the "extra" resources are necessary if multiple threads access the NmsTemplate.



I didn't yet reply to the issues regarding ack mode, but what settings are you running which such that you move stuff back from the dead letter queue upon startup? Is that just to get basic behavior to work or is it for truly exceptional circumstances?


We are using transactional acknowledgment mode.

Moving messages from "DLQ.myqueue" to "myqueue" is something we do after we call Start() on the listener because when Stop() is called the pending messages are rejected and end up on the DLQ. More on that here (http://forum.springframework.net/showthread.php?t=4998).

To give you some context, some queues contain messages that cause database changes. We also have a daily automated process which creates a new database partition and drops an old partition because the volume is huge and we can only afford to have a sliding window of data.

In order for these not to interfere we:

1) call Stop() on the listener containers of queues that could potentially modify the database
2) make schema/partition changes
3) call Start() on the stopped listener containers
4) move messages from DLQ to original queue


As previously mentioned this has been running for about a week and seems to be stable.