PDA

View Full Version : simple chaining of response-destination


keenan
08-29-2008, 04:08 PM
Hopefully someone can set me straight here because I'm having a lot of issues with queueing and I'm starting to think maybe I'm missing some fundamentals.

I have a producer which puts 50000 messages on a queue:


while (count < 50000)
{
count++;
string text = "message # " + count.ToString();
NmsTemplate.ConvertAndSend("step.1", text);
}


In my configuration I have a listener container which has a chain of listeners that simply copies a message from one queue to another:


<object id="echoHandler" type="MyMessageDemo.Test3.Echo, MyMessageDemo"/>

<nms:listener-container acknowledge="client" concurrency="20" connection-factory="mqConnFactory">
<nms:listener destination="step.1" ref="echoHandler" method="Receive" response-destination="step.2"/>
<nms:listener destination="step.2" ref="echoHandler" method="Receive" response-destination="step.3"/>
<nms:listener destination="step.3" ref="echoHandler" method="Receive" response-destination="step.4"/>
<nms:listener destination="step.4" ref="echoHandler" method="Receive" response-destination="step.5"/>
<nms:listener destination="step.5" ref="echoHandler" method="Receive" response-destination="step.6"/>
</nms:listener-container>


The Echo class is simply:


class Echo
{
public string Receive(string message)
{
Thread.Sleep(50);
return message;
}
}



When I run the program, various errors occur, such as:

Error: : Collection was modified; enumeration operation may not execute.

or, sometimes the program crashes. Other times it gets stuck because listeners get unregistered and I end up with 0 listeners on some of the queues and > 20 listeners (the concurrency value) on the others.

Here is my connection factory:


<object id="mqConnFactoryNoCache" type="Apache.NMS.ActiveMQ.ConnectionFactory, Apache.NMS.ActiveMQ">
<constructor-arg index="0" value="tcp://localhost:61616?jms.prefetchPolicy.all=100&amp;transpo rt.requesttimeout=15000"/>
</object>

<object id="mqConnFactory" type="Spring.Messaging.Nms.Connections.SingleConnectionF actory, Spring.Messaging.Nms">
<constructor-arg name="targetConnectionFactory" ref="mqConnFactoryNoCache"/>
<property name="ReconnectOnException" value="true"/>
</object>


Surely I'm missing something basic here?

keenan
08-29-2008, 11:10 PM
I don't know if this is helpful, but I wanted to document it. In a test similar to the above I ran in debug mode and the output window had 13,000+ lines of:

A first chance exception of type 'antlr.MismatchedTokenException' occurred in antlr.runtime.dll

And several hundred lines of:

A first chance exception of type 'System.Threading.ThreadAbortException' occurred in Spring.Messaging.Nms.dll
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in Apache.NMS.ActiveMQ.dll

keenan
09-02-2008, 03:37 PM
I've had a chance to dive into this with the debugger and the results will probably not surprise anyone.

The NMS documentation (http://activemq.apache.org/nms/ndoc/ActiveMQ.Connection.html) clearly indicates the Connection class is not guaranteed to be thread-safe.

In this case I was using SingleConnectionFactory in the example code and had 20 consumers on 5 queues all sharing the same connection, so the "Collection was modified" error was a natural result.

Even though the Apache.NMS.ActiveMQ.Connection class uses ArrayList.Synchronized() to create an IList of sessions, enumeration is not thread-safe and the Connection class does not lock on SyncRoot in any of the methods that enumerate the sessions collection. DispatchMessage is one such method, and that is where most of the problems in the demonstration occurred.

keenan
09-02-2008, 08:30 PM
As a final follow up to this thread I tried the Spring.Messaging.Nms.Connections.CachingConnection Factory class now that the closed connection (http://forum.springframework.net/showpost.php?p=13165&postcount=16) defect is fixed (which was actually an AMQ/NMS issue), and everything worked great.

.ben
09-03-2008, 07:31 AM
Thanks for the update

steinard
09-03-2008, 07:39 AM
Hi!


A first chance exception of type 'antlr.MismatchedTokenException' occurred in antlr.runtime.dll
I also see this in my log, and I wonder what could be going wrong. It happens quite frequently and I cannot help wonder if I would get a more efficient application without these exceptions (we do live market simulations as stock bid/ask and price changes happening in the market).

Thanks,
Steinar.

keenan
09-03-2008, 04:46 PM
...we do live market simulations as stock bid/ask and price changes happening in the market).


If you don't mind me asking, what is your configuration from a messaging perspective? Are you using NMS heavily? I'm really curious how other people are using messaging and how well it is working for them.



I also see this in my log, and I wonder what could be going wrong. It happens quite frequently


I put the debugger on it and here are the gory details. Hopefully an insider can comment.

The error occurs in ExpressionParser.cs in the functionOrVar() method. Near the top is the following section:


try {
{
match(POUND);
match(ID);
match(LPAREN);
}
}


it's the match(LPAREN) line which causes this error:


antlr.MismatchedTokenException occurred
Message="expecting \"LPAREN\", found ')'"
Source="antlr.runtime"
StackTrace:
at antlr.Parser.match(Int32 t)
at Spring.Expressions.ExpressionParser.functionOrVar( )


Here is the call stack at the time (I XXX'd out the message details for brevity):


Spring.Core.dll!Spring.Expressions.ExpressionParse r.functionOrVar() Line 1499 + 0x10 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.startNode() Line 1254 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.primaryExpression() Line 983 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.unaryExpression() Line 947 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.powExpr() Line 853 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.prodExpr() Line 771 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.sumExpr() Line 572 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.relationalExpression() Line 507 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.logicalAndExpression() Line 451 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.logicalOrExpression() Line 366 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.expression() Line 201 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.argument() Line 2623 + 0x8 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.methodArgs() Line 2469 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.methodOrProperty() Line 1445 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.startNode() Line 1104 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.primaryExpression() Line 983 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.unaryExpression() Line 947 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.powExpr() Line 853 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.prodExpr() Line 771 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.sumExpr() Line 572 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.relationalExpression() Line 507 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.logicalAndExpression() Line 451 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.logicalOrExpression() Line 366 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.expression() Line 201 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.ExpressionParse r.expr() Line 170 + 0x9 bytes C#
Spring.Core.dll!Spring.Expressions.Expression.Pars e(string expression = "Receive(#convertedObject)") Line 103 + 0xa bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.List ener.Adapter.MessageListenerAdapter.OnMessage(Apac he.NMS.IMessage message = {ActiveMQTextMessage[ XXX ] Text=XXX}, Apache.NMS.ISession session = {Cached NMS Session: Apache.NMS.ActiveMQ.Session}) Line 296 + 0x1d bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.List ener.AbstractMessageListenerContainer.DoInvokeList ener(Spring.Messaging.Nms.Listener.ISessionAwareMe ssageListener listener = {Spring.Messaging.Nms.Listener.Adapter.MessageList enerAdapter}, Apache.NMS.ISession session = {Cached NMS Session: Apache.NMS.ActiveMQ.Session}, Apache.NMS.IMessage message = {ActiveMQTextMessage[ XXX ] Text=XXX}) Line 409 + 0xf bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.List ener.AbstractMessageListenerContainer.InvokeListen er(Apache.NMS.ISession session = {Cached NMS Session: Apache.NMS.ActiveMQ.Session}, Apache.NMS.IMessage message = {ActiveMQTextMessage[ XXX ] Text=XXX}) Line 362 + 0x21 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.List ener.AbstractMessageListenerContainer.DoExecuteLis tener(Apache.NMS.ISession session = {Cached NMS Session: Apache.NMS.ActiveMQ.Session}, Apache.NMS.IMessage message = {ActiveMQTextMessage[ XXX ] Text=XXX}) Line 339 + 0x11 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.List ener.AbstractMessageListenerContainer.ExecuteListe ner(Apache.NMS.ISession session = {Cached NMS Session: Apache.NMS.ActiveMQ.Session}, Apache.NMS.IMessage message = {ActiveMQTextMessage[ XXX ] Text=XXX}) Line 304 + 0x11 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.List ener.SimpleMessageListener.OnMessage(Apache.NMS.IM essage message = {ActiveMQTextMessage[ XXX ] Text=PH2008PA09004454.fin}) Line 391 + 0x1b bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Messag eConsumer.DispatchAsyncMessages() Line 222 + 0xd bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Sessio n.DispatchAsyncMessages() Line 508 + 0xb bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Dispat chingThread.MyThreadFunc() Line 118 + 0xe bytes C#


It looks like the expression we are trying to parse is:

Receive(#convertedObject)

so I'm not sure why we are expecting LPAREN and finding ')', but that is as far as I got when looking into it.

As you mentioned, this occurs frequently and even if it does not represent an error if it could be avoided it might have a positive performance impact.

steinard
09-03-2008, 08:20 PM
Hi!

Thanks for looking into it. I did not get very much wiser though, although I believe that I could squeeze some more performance out of the application without these exceptions.


If you don't mind me asking, what is your configuration from a messaging perspective? Are you using NMS heavily? I'm really curious how other people are using messaging and how well it is working for them.
Most exchanges and broker houses publish the market auction on some kind of messaging based system, we simply call these messaging systems for feeds as they feed our system with market data. As there are different kind of feeds or data sources (asynchronous messaging feeds, web pages, files from ftps, excel sheets on shared locations etc..), we create adapters that connect to the various feeds/data sources and adapt the messages/data to an internal format and sends it to our messaging server (ActiveMQ at the moment but this is likely to change to MSMQ). Then a market pricing server is listening to the queue all these adapters send information onto and automatically stores and updates a live cache of the total market being monitored (the cache is an implementation of the identity map pattern).

Together with historical data we do heavy simulations to predict the forward prices of the market and how the market will move (the simulations are based on market models and theories published by researches working in the company). If you look at my profile you will see the web page to my workplace and be able to read more about the software in detail. But the messaging part is quite central in gathering information about how the market is at the moment and performance is quite important, especially for traders that need information about market changes as quickly as possible.

So how is your usage of NMS?

Cheers,
Steinar.

Mark Pollack
09-03-2008, 09:46 PM
Hi,
The exception in antlr is 'normal' in that the 2.x series uses exceptions for their internal flow control. The code in question is generated from antlr. There is a JIRA issue to move to use a newer version of antlr that doesn't use exceptions for flow control. Work started on that issue but it was based on the 3.0 version of antlr that is apparently 'dead' on the .NET side. (I had an email exchange on the antlr-dev list about that). At the time 3.1 wasn't out yet so work was stopped on moving to the new version. I just checked and on Aug 12th 3.1 was released so now we can move ahead.

I used the spring expression language here just to avoid the low-level reflection calls, but if you think performance is a concern we can switch to direct (+cached) reflection calls. What did you measure as the peformance impact?

Cheers,
Mark

keenan
09-04-2008, 12:05 AM
I used the spring expression language here just to avoid the low-level reflection calls, but if you think performance is a concern we can switch to direct (+cached) reflection calls. What did you measure as the peformance impact?


I do not have any performance numbers. My conjecture is that it might be an issue if the volume was heavy enough, based on my belief that the above stack occurs with each message dispatch.

The ActiveMQ performance (http://activemq.apache.org/performance.html) page indicates that ~2000 messages/sec on a durable queue was achieved with a single producer and consumer thread on a Celeron. With multiple producers and consumers on modern server hardware it seems like the throughput could get high enough that the ANTLR exception could become an issue.

In my case we are just starting to integrate messaging into our application and only process a few thousand messages per day so the impact (if any) is negligible.

Having said that, I'm personally supportive of upgrading to the latest ANTLR release on the principal that using exceptions for control flow is not always wrong but certainly not in the list of best practices.

steinard
09-04-2008, 12:14 AM
Hi!

I profiled the pricing server in December 2007 but I did not focus on the antlr exceptions, but rather what performance gains I could get out of my own code. I doubt that I will have time to look at this until mid October, it is a reason for why I'm still working at 1am...

Cheers,
Steinar.

P.S I'm going to Linz...

Mark Pollack
09-04-2008, 12:55 AM
Hi,
Also check if your activemq build has this fix (https://issues.apache.org/activemq/browse/AMQNET-109) for perf reasons...
Mark

keenan
09-04-2008, 02:01 AM
So how is your usage of NMS?


Our usage could best be categorized as extremely light. The project is not inherently message based and we have only started to integrate messaging recently.

Due to the fact NMS is not officially released we have been hesitant to dive into the deep end of messaging. With that in mind we decided to use our load process as a proof of concept.

Previously we had a monolithic load process which was something like:


Load(string file)
{
object result1 = Step1(file);
object result2 = Step2(result1);
object result3 = Step3(result2);
...
}

That is a gross simplification but the point is that it was a multi-step sequential process where the result of each step was needed for the following step. The steps also differ in resource utilization (disk, CPU, network).

The absolute beauty of Spring.NET is how unobtrusive the container is. Although we did make a few code changes to incorporate automatic retry and queue based error handling, these were mostly additional features. We could have gotten away with almost no code changes if that were a priority.

By adding the necessary connection factory and listener container:


<object id="mqConnFactoryNoCache" type="Apache.NMS.ActiveMQ.ConnectionFactory, Apache.NMS.ActiveMQ">
<constructor-arg index="0" value="tcp://localhost:61616"/>
</object>
<nms:listener-container acknowledge="client" concurrency="5" connection-factory="mqConnFactory">
<nms:listener destination="step.1" ref="Loader" method="Step1" response-destination="step.2"/>
<nms:listener destination="step.2" ref="Loader" method="Step2" response-destination="step.3"/>
<nms:listener destination="step.3" ref="Loader" method="Step3" />
</nms:listener-container>


the only missing piece to get it all wired up was to change the way the load process was launched. By adding a directory watcher which put a message on the "step.1" queue, the service is able to load files as they are dumped into an input folder.

Really, our use of queues in this case was a bit frivolous because the producers and consumers are in the same process. We could have accomplished the same goals using events and thread pools. However, the real mission was to get our feet wet with queueing and on that front it was a good decision.

If you've read my other posts you know that I have run into several problems along the way. Thankfully, the tireless ActiveMQ/NMS/Spring.NET developers are very responsive and I have no doubt that the remaining issues will be ironed out in short order.

When they do I plan to take further advantage of messaging. I have not thought it all the way through, but I have considered using messaging instead of remoting as the primary interface between the client and server.

steinard
09-04-2008, 07:59 AM
Hi!

We have not experienced any problems yet with how our stack is configured in regards of NMS. The setup of the single qeue is also very simple and light, and the adapters may run anywhere (which is nice when you consider all the feed vendors and their licensing policies).

For the rest of the application suit we use remoting and som legacy com+/MTS services as the primary interface between the client and server. Our clients use remoting to access the pricing server as well. Messaging is only used between the feeds/data sources and the pricing server. We plan to upgrade to WCF as soon as I get done with some features, although I guess there are some considerations needed to be taken as I cannot use the default WCF DataContractSerializer (I'm using NHibernate to persist my domain objects). I agree with you that Spring.Net as a dependency injection container is very unintrusive, I also like the unintrusiveness of NHibernate. Seeing that NMS and Quartz are made in the same spirit is really cool. I have some scheduling tasks that are running but all the scheduling logic is hidden in a scheduling config file. Really smart, I only had to write the methods that do the job (business logic), while the infrastructural concerns of starting the job, when to start the job and everything else a scheduler needs to take care of was fulfilled with a few xml lines. Cute ;)

Cheers,
Steinar.

Mark Pollack
09-04-2008, 08:11 PM
Hi,
I've made an issue (http://jira.springframework.org/browse/SPRNET-1021) to remove the use of SpEL for finding the method to dispatch to when using MessageListenerAdapter to avoid the cost of the internal antlr exception that is thrown for flow control.
Mark