dlmarion commented on code in PR #5129:
URL: https://github.com/apache/accumulo/pull/5129#discussion_r1873687117
##########
core/src/main/java/org/apache/accumulo/core/fate/Fate.java:
##########
@@ -372,6 +375,42 @@ public Fate(T environment, FateStore<T> store, boolean
runDeadResCleaner,
break;
}
}
+ queueSizeHistory.clear();
+ } else {
+ // The property did not change, but should it based on the queue size?
Maintain
+ // the last X minutes of queue sizes. If the queue size is always
larger than the number
+ // of Fate threads multiplied by some factor, then suggest that the
+ // MANAGER_FATE_THREADPOOL_SIZE be increased.
+ final long interval = Math.min(60, TimeUnit.MILLISECONDS
+
.toMinutes(conf.getTimeInMillis(Property.MANAGER_FATE_QUEUE_CHECK_INTERVAL)));
+ if (interval == 0) {
+ queueSizeHistory.clear();
+ } else {
+ final int sizeFactor =
conf.getCount(Property.MANAGER_FATE_QUEUE_CHECK_FACTOR);
+ if (queueSizeHistory.size() >= interval * 2) { // this task runs
every 30s
+ final int warnThreshold = configured * sizeFactor;
+ boolean needMoreThreads = true;
+ for (Integer i : queueSizeHistory) {
+ if (i < warnThreshold) {
+ needMoreThreads = false;
+ break;
+ }
+ }
+ if (needMoreThreads) {
+ log.warn(
+ "Fate queue size is {} times the number of Fate threads for
the last {} minutes,"
+ + " consider increasing property: {}",
+ sizeFactor, interval,
Property.MANAGER_FATE_THREADPOOL_SIZE.getKey());
+ // Clear the history so that we don't log for another 5 minutes.
+ queueSizeHistory.clear();
+ } else {
+ while (queueSizeHistory.size() >= interval * 2) {
+ queueSizeHistory.remove();
+ }
+ }
+ }
+ queueSizeHistory.add(workQueue.size());
Review Comment:
I forgot that we changed the Fate implementation to used a
LinkedTransferQueue and that the WorkFinder threads only put things on the
queue when there is an available consumer thread. I could invert this check,
instead of checking that the queue size is larger than the number of threads, I
could use
[getWaitingConsumerCount](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/TransferQueue.html#getWaitingConsumerCount()).
If the value is always zero, then log the warning to increase the threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
