Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
Clebert, We see the AMQ222162 message after restarting on a JVM crash. In that case we can recover the messages. Maybe this is any help to Gašper The last time we really lost messages we didn’t notice any particular errors at server not client side. I think we’ll find some commercial support

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
I believe you're saying you got that message? are you getting into a failover scenario and perhaps a split brain? you could use ZooKeeper to avoid such thing. In case you're using retention, look at the replay method in the Address.. where you can extra messages from your queue in the period

Re: OpenMessaging Benchmark - Artemis Driver PR

2022-10-20 Thread Dave Fisher
> On Oct 19, 2022, at 11:04 PM, Clebert Suconic > wrote: > > The confirmation window size is definitely a change that needs to be done. > > Is your benchmark a single producer / consumer ? The benchmark program can be configured with many consumers and producers of many topics. You can se

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
Gašper, Did you see "AMQ222162 Moving data directory ..." in your logs? In that case you should use ./artemis data exp (and ./artemis data imp) to export the old replicas and find your messages there (and import them agian). See Artemis Data Tools in the manual. Regards, Walter

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
P.s. we'll also look at retention and enable message logging. We're not even sure now wether the missing messages have been accepted to the queue or not...

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
Clebert, Thanks for your reply. I'm aware you need a way to reproduce it. Unfortunately this only happens once in a while and we're not able to reproduce yet. We wil investigate further, thanks for the instructions! We'll keep you posted. We'll also have a look at OFF-WITH-REDISTRIBUTION, I

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
I just read the entire set of commits between 2.22.0 and 2.25.0... The only major change that happened in 2.25.0 was the paging improvement, removing cache... everything else is not really related to the core of the project (journalling, routing.. etc). There are fixes related to redistribution

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
Is there anything on DLQ or Expiry? Any exceptions in the logs? Things you need to understand in order to write a reproducer: How you're sending (protocol you're using? size, how often, how many producers, how you open the connection and how you commit (say if you are using XA). are you pooling

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
Regarding the reproducer. Write a test that mimics your payload and configuration In an independent way. You could use the examples for that. If you reproduce your issue I way someone can actually see what’s happening (even jf it’s misconfig) is the only way someone would be able to help you rem

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
Can you reproduce it in your test at least ? Use journal retention and look for the message your lost on the print data. It could be misconfiguration in your cluster. There are no changes between 2.22 and 2.26 that could lead to that. Run your test with retention enabled and I can help you out

RE: Messages getting lost on Artemis 2.25

2022-10-20 Thread Gašper Čefarin
I hope you don't mind if I reply to this thread. I'd also like to report messages getting lost. I've had 2 occurrences of losing messages when using simple replication (1 live and 1 backup server). I was using artemis v2.22.0. I was not able to replicate the issue, and I think it happened when I