It was a chicken egg case. The transaction manger wasn't the problem. But I am getting time outs now with tomcat nio exec threads being blocked in UtilCache and CachedClassloader those are major synchronization bottle necks but I have already updated then both to ofbiz 14. Anyway I suppose I should search the mail archive and then try a new thread.
On Wed, Jul 20, 2016 at 6:32 PM, Jacopo Cappellato < [email protected]> wrote: > Is it possible that your code is creating EntityListIterator objects and > not properly closing them? This may cause connections to stay open until > the SqlProcessor objects are finalized. > My guess is that the connection leaks are happening before the > finalization, and not as a consequence of it, because EntityListIterator > are not properly closed, and a db transaction is active when the finalize > method attempts to close the connection. > Based on the above assumptions, that could be wrong, I don't think that > there is an interference of transactions for different tenants, but rather > data that you would expect to see is rolled back when the connection is > closed before a commit. > Please try to investigate in this direction and let me know if you do any > progress. > > Good luck! > > Jacopo > > On Wed, Jul 20, 2016 at 1:51 PM, Jacopo Cappellato < > [email protected]> wrote: > > > Hi Justin, > > > > On Wed, Jul 20, 2016 at 12:37 PM, Justin Robinson < > > [email protected]> wrote: > > > >> I am really just interested in the architecture, it's no so much an > issue > >> with any version of ofbiz. (In fact I am working on a system based on > >> ofbiz, it would be better to replace it with pure ofbiz, but that is not > >> my > >> decision to make). I hope you will still be willing to discuss your > >> implementation regardless of that fact. > >> > > > > Definitely! > > > > > >> > >> Yes org.ofbiz.geronimo.GeronimoTransactionFactory > >> calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, > >> datasourceInfo.getInlineJdbc()); > >> which is implemented > >> through > >> > org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, > >> Element) which uses a > >> ConcurrentHashMap to store a connection pool per tenant datasource and > it > >> seems to work very well, the correct pool is provided to the appropriate > >> tenant threads. > >> > >> This is an intermittent issue when testing with 50-100 tenants, > >> transactions across tenants seem to collide occasionally. At least that > >> seems to be the case, I have stepped through the code in a debugger. > >> > >> It seem to happen in SqlProcessor finalize method, the gc calls finalize > >> and when the connection is closed (returned to the pool) it calls commit > >> first and then rolls back because a transaction is in progress as a > result > >> the connection is leaked and doesn't return to the pool. This is clearly > >> seen on a massive scale in jprofiler on the jdbc probe page. > >> > > > > Hmmm... this is an interesting detail, there could be an issue in the > > SqlProcessor class or maybe in the way that is used in your framework. > > > > I will dig into this more and let you know if I find something > interesting. > > > > Jacopo > > > > > >> > >> Is it a limitation of the architecture? Is it possible to provide a > >> TransactionManger per tenant in ThreadLocal or would that not work? > >> > >> Or am I missing something here? I realise it's a complex issue, but I am > >> trying to stay as close to ofbiz as I can I hope one day we can replace > >> the > >> framework with a pure ofbiz implementation. I'd rather not make a change > >> like this which diverges from ofbiz architecture. How would you solve > this > >> issue? > >> > >> > >> On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < > >> [email protected]> wrote: > >> > >> > Hi Justin, > >> > > >> > your understanding of the class implementation is correct and there > may > >> be > >> > a chance that some code may need to be improved, but I am not sure as > I > >> > could only give a cursory look at the code. > >> > By the way, I see that the code in GeronimoTransactionManager calls a > >> > "tenant-aware" method in DBCPConnectionFactory and this should provide > >> the > >> > correct connection to every thread. > >> > > >> > Could you please elaborate more on your concern? If you could share a > >> > usage/failure scenario to recreate the issue it would be great too. > >> > > >> > Thanks, > >> > > >> > Jacopo > >> > > >> > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < > >> > [email protected]> wrote: > >> > > >> > > I am trying to understand the database connection pooling in ofbiz > 14, > >> > > which uses dbcp2. > >> > > > >> > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there > >> is a > >> > > static ConcurrentHashMap which provides a ManagedDataSource for each > >> > > tenant. > >> > > > >> > > But when it comes to obtaining a transaction manager reference, it > >> looks > >> > > like there is one global instance for all tenants. > >> > > > >> > > getTransactionManager in > org.ofbiz.geronimo.GeronimoTransactionFactory > >> > > returns the static reference to a single GeronimoTransactionManager. > >> > > > >> > > In multi tenant testing overlapping from different tenants seem like > >> they > >> > > interact with eachother. > >> > > > >> > > Can you explain how the implementation is supposed to work in with > >> multi > >> > > tenancy ? > >> > > > >> > > >> > > > > >
