kevinrr888 commented on issue #5130: URL: https://github.com/apache/accumulo/issues/5130#issuecomment-2557302170
@keith-turner and I met yesterday to discuss the overall structure of the new changes to be made. Here is a summary of what we concluded. ### The current implementation in 4.0: 1 FATE instance for USER transactions, 1 FATE instance for META transactions. Each FATE instance has its own set of transaction runners __*which work on all fate operations*__ in a __*single thread pool*__, a work finder (scans the underlying storage for FATE transactions to work on and assigns the work to the runners), a pool watcher (watches for property change and dead threads, resizing pool and submitting more transaction runners if needed), and a dead reservation cleaner (finds and unreserves transactions which are reserved by a now-dead manager). ### The changes to be made: 1 FATE instance for USER transactions, 1 FATE instance for META transactions. Each FATE instance has its own set of executors (new class maybe called FateExecutor). Each FateExecutor __*works on a subset of fate operations*__ (meaning each has a pool of runners, a work finder, and a pool watcher). Each FATE instance only has one dead reservation cleaner since more than one per instance is unneeded (we are just scanning the storage layer for dead reservations). There will be two properties, one to configure the USER pools, and one for META pools. ### What else we considered: We considered a set of FATE instances for each transaction type (USER/META), but that was a bit bloated and had some less-desirable functionality as some operations wouldn't care about what FATE instance we operate on and only care about the underlying storage layer. For example, starting a transaction (Fate.startTransaction()) just creates the transaction in the store. With multiple FATE instances, this could be achieved the same through any of them, which makes it confusing for code using FATE. We considered one work finder per FATE instance but with the current structure of the code using a TransferQueue for the work, this wouldn't be possible. This does have the draw back of having multiple threads scanning the same store for the same things, then just filtering them out based on what work they are assigned to take. Maybe this structure could change in a follow on PR if desired, but as an initial PR, using multiple work finders per FATE instance. We considered sharing pools between USER/META transactions, but Keith noted that this could lead to dead locks. Best to keep resources for USER and META transactions separate. I think I covered everything. If anyone has any questions or concerns, please let me know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
