Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)

[Rob Davies]

yep - being able to reproduce - this is now fixed in trunk - but too  
late for the 5.1 release

thanks,
Rob

On 16 Apr 2008, at 13:54, yaussy wrote:


Rob,

While I don't yet have a solid / consistent test case for you, our
application, using a particular test scenario the app guys have, can
reproduce the problem every time.  If you have anything you want me  
to try
out, let me know.

I'm still working with my test code to get something more  
reproduceable,
while adding some debug code to IndexManager.

Kevin


yaussy wrote:

Still working on a consistent test case.  But, I've run into  
something
along the way, and I'm not sure if it's related to the original  
exception
I've logged here.  I was able to get the problem to happen when I  
expanded
my test case to a publisher of 6 topics, at 100+/sec/topic, and a  
consumer
that consumes all those topics.  I have this configured to use a
connection for each topic, for both the publisher and the  
consumer.  I've
only been able to get the original exception to happen once, but  
I've more
consistently hit a problem wherein no exception is thrown, but the  
broker
just stops forwarding events for one of the topics.  After this  
point, the
publisher continues, but all events for this one topic are queued  
to disk.

I'm trying to see if I can reduce this test case down right now.   
But,
maybe you could give this a go with the supplied consumer /  
supplier test
code?

Kevin


rajdavies wrote:

ok - thx for checking! - let me know when you have a test case

On 15 Apr 2008, at 15:06, yaussy wrote:


Bad news - problem still happens with "5.2" snapshot dated 4/13.  I
did not
see anymore 5.1 snapshots, but saw the 5.2 directory.  The archive
still has
5.1 for the path names, so I figured this is what I should take.

I'm still working with it and am trying to see if I can get any  
more
debug
information.  I have not been able to reproduce it with a test
program yet.



yaussy wrote:

Rob,

This is good news - I will try using last nights snapshot.

I started out using AMQMessageStore, but I've never had good
performance
luck with AMQs Journal - it always seems to have this frequent,
high CPU
cost as it checkpoints.  Our Durable rate requirements are fairly
significant, such as a few hundred events per second to a  
handful of
consumers, on one of our clusters.  It has not performed well,
whereas
Kaha by itself is excellent.

Kevin


rajdavies wrote:


On 14 Apr 2008, at 14:07, yaussy wrote:


I have not been able to reproduce this problem outside of our
application,
but I'm still trying.

Anyway, I'm using regular Kaha persistence (not  
AMQMessageStore),
and during
a Durable Topic test, I'm eventually getting the following
exception
in the
AMQBroker, after which the broker stops forwarding events to the
consumer.

Anyone seen this?


<
org
.apache
.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
<Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>  
Failed
to fill
batch
Stack Trace follows:
java.lang.RuntimeException: Failed to get next index from
IndexManager:(index-topic-subs) for offset=7681977, key=(2,
20080006, 47),
value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
     at
org
.apache
.activemq
.kaha
.impl
.index 
.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
267)
     at
org
.apache
.activemq
.kaha
.impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
449)
     at
org
.apache
.activemq
.store
.kahadaptor 
.TopicSubContainer.getNextEntry(TopicSubContainer.java:
95)
     at
org
.apache
.activemq
.store
.kahadaptor
.KahaTopicMessageStore
.recoverNextMessages(KahaTopicMessageStore.java:165)
     at
org
.apache
.activemq
.store
.ProxyTopicMessageStore
.recoverNextMessages(ProxyTopicMessageStore.java:97)
     at
org
.apache
.activemq
.broker
.region
.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
107)
     at
org
.apache
.activemq
.broker
.region
.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
188)
     at
org
.apache
.activemq
.broker
.region
.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
104)
     at
org
.apache
.activemq
.broker
.region
.cursors
.StoreDurableSubscriberCursor
.hasNext(StoreDurableSubscriberCursor.java:210
)
     at
org
.apache
.activemq
.broker
.region
.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
479)
     at
org
.apache
.activemq
.broker
.region 
.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
357)
     at
org
.apache
.activemq
.broker.region.AbstractRegion.acknowledge(AbstractRegion.java: 
349)
     at
org
.apache
.activemq 
.broker.region.RegionBroker.acknowledge(RegionBroker.java:
474)
     at
org
.apache
.activemq
.broker.TransactionBroker.acknowledge(TransactionBroker.java: 
194)
     at
org
.apache
.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
     at
org
.apache
.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
     at
org
.apache
.activemq
.broker 
.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
84)
     at
org
.apache
.activemq
.broker
.TransportConnection.processMessageAck(TransportConnection.java:
444)
     at
org.apache.activemq.command.MessageAck.visit(MessageAck.java: 
196)
     at
org
.apache
.activemq
.broker.TransportConnection.service(TransportConnection.java: 
293)
     at
org.apache.activemq.broker.TransportConnection
$1.onCommand(TransportConnection.java:181)
     at
org
.apache
.activemq
.transport.TransportFilter.onCommand(TransportFilter.java:68)
     at
org
.apache
.activemq
.transport
.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
143)
     at
org
.apache
.activemq
.transport.InactivityMonitor.onCommand(InactivityMonitor.java: 
206)
     at
org
.apache
.activemq
.transport.TransportSupport.doConsume(TransportSupport.java:
84)
     at
org
.apache
.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
196)
     at
org
.apache 
.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
183)
     at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
     at  
java.io.RandomAccessFile.readFully(RandomAccessFile.java:
383)
     at  
java.io.RandomAccessFile.readFully(RandomAccessFile.java:
361)
     at
org
.apache
.activemq
.kaha 
.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
46)
     at
org
.apache
.activemq 
.kaha.impl.index.IndexManager.getIndex(IndexManager.java:
66)
     at
org
.apache
.activemq
.kaha
.impl
.index 
.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
265)
     ... 27 more

--
View this message in context:
http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
Sent from the ActiveMQ - User mailing list archive at  
Nabble.com.


There was a bug fixed in this area last Friday - would be  
interested
if this is still a problem for you with a later version?
If it is - Ill digg a little deeper.

btw - why are you using Kaha instead of AMQStore ?




cheers,

Rob

http://open.iona.com/ -Enterprise Open Integration
http://rajdavies.blogspot.com/








--
View this message in context:
http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.







--
View this message in context: 
http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.