dlmarion opened a new issue, #5130:
URL: https://github.com/apache/accumulo/issues/5130
In main the `Manager` instantiates two instances of Fate, one for the
`MetaFateStore` and one for the `UserFateStore`. Each instance of Fate contains
a single thread pool of `Property.MANAGER_FATE_THREADPOOL_SIZE` size which is
used to process Fate transactions. The threads in the Fate thread pool try to
process transactions in a fair manner, by spending time on a different
transaction in the queue when done with the current transactions' step (see
#3852 and `FateInterleavingIT` for more information). The default size of
`MANAGER_FATE_THREADPOOL_SIZE` is 4, which may need to be increased in main,
could allow for some transactions to go a long time without making progress
(It's possible that this could be the cause of the long delete table
transaction times we are seeing in integration tests).
It may be advisable to dedicate some number of threads to specific
transaction types. For example, maybe 1 thread dedicated to creating and
deleting tables, and a larger thread pool dedicated to bulk imports and
compactions. I'm initially thinking of a change to the
`MANAGER_FATE_THREADPOOL_SIZE` property such that instead of it being a count
of the number of threads, the property is renamed to
`MANAGER_FATE_THREADPOOL_CONFIGURATION` and contains a json map where the key
is a comma separated list of Fate transaction names and the value is the number
of threads, like:
```
{
"SET_AVAILABILITY,CREATE_TABLE,DELETE_TABLE,CREATE_NAMESPACE,DELETE_NAMESPACE,RENAME_NAMESPACE":
1,
"BULK_IMPORT": 2,
"COMPACT": 4,
"MERGE,SPLIT,IMPORT_TABLE,EXPORT_TABLE": 2
}
```
The Manager would parse this configuration and start one Fate instance for
each Key,Value pair in the map. The Fate instance would be responsible for
managing only the types that it's given. Currently
`MANAGER_FATE_THREADPOOL_SIZE` is watched and the size of the threadpool
changes when the property changes. That may be harder to do with this idea. I
think the name of the transaction is currently serialized into the transaction
and I think the transactions can be filtered by type, so this would build upon
that capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
