Hi Gerda Thanks for sharing your solution.
I guess users like you whom dynamic add and remove routes more frequently could be affected by this deadlock when doing concurrent add and removes. On top of my head one aspect that could lead to a problem is that if your routes are using the same route ids, so you are doing a remove of "foo" and add "foo" at the same time. However that is needed to be investigated deep dive to really be sure. I think your solution is something we should consider for merging to the core project. You are welcome to create a JIRA and send a PR with the code changes. Thank you. On Fri, Sep 2, 2022 at 8:50 AM Gerda Ilger <gerda.il...@ecosio.com> wrote: > Hi Claus, > > Yes, we're now doing our own locking - essentially just straightening > out the order of locks Camel will need later on in our code. > It feels wrong, though: CamelContext is already putting a lot of > effort into protecting addRoutes/removeRoute-calls. > Streamlining the order in which locks are obtained would fix this very > deadlock - and would probably be beneficial in other places too? > > I played around with it a bit: it felt less invasive to lock the model > for the "removeRoute" call than locking CamelContext for the whole > "addRoutes"-call (which is what we're doing now in our code, works as > well), but I know I don't have the big picture. > https://github.com/apache/camel/compare/main...gerdailger:camel:deadlock > > Either way, let me quickly thank you all for Camel at this point. It > has been working very well for us for years. > > Kind regards > Gerda > > Am Do., 1. Sept. 2022 um 18:36 Uhr schrieb Claus Ibsen < > claus.ib...@gmail.com>: > > > > Hi > > > > Do you own locking if you do concurrent model changes to Camel > > > > On Thu, Sep 1, 2022 at 6:21 PM Gerda Ilger <gerda.il...@ecosio.com> > wrote: > > > > > Dear Camel Community, > > > > > > We ran into a deadlock when concurrently adding and removing routes. > > > > > > We're dynamically adding and removing routes, whose (ftp) consumers > > > sometimes take a longer time to start up, and observed the following: > > > > > > * (Default/Abstract)CamelContext.addRoutes -> > > > DefaultCamelContext.addRouteDefinitions locks DefaultModel first, and > > > is then blocked trying to lock DefaultCamelContext > > > * (Default/Abstract)CamelContext.removeRoute locks DefaultCamelContext > > > first, and is then blocked trying to lock DefaultModel > > > > > > --- > > > > > > Found one Java-level deadlock: > > > ============================= > > > > > > "pool-2-thread-3": > > > waiting to lock monitor 0x00007fd21404a130 (object 0x000000068c002a20, > > > a org.apache.camel.impl.DefaultCamelContext), > > > which is held by "pool-2-thread-2" > > > > > > "pool-2-thread-2": > > > waiting to lock monitor 0x00007fd22000b0c0 (object 0x000000068c800170, > > > a org.apache.camel.impl.DefaultModel), > > > which is held by "pool-2-thread-3" > > > > > > --- > > > > > > > > > We are on Camel 3.7, but I was able to reproduce it with > > > 3.19.0-SNAPSHOT in a JUnit test. > > > > > > I haven't found any tickets or mailing list discussions about this > > > behaviour. > > > Is this a known issue? > > > > > > > > > Thanks, > > > Gerda > > > > > > > > > -- > > Claus Ibsen > > ----------------- > > http://davsclaus.com @davsclaus > > Camel in Action 2: https://www.manning.com/ibsen2 > -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
