dlmarion commented on code in PR #5129:
URL: https://github.com/apache/accumulo/pull/5129#discussion_r1873721343
##########
core/src/main/java/org/apache/accumulo/core/fate/Fate.java:
##########
@@ -372,6 +375,42 @@ public Fate(T environment, FateStore<T> store, boolean
runDeadResCleaner,
break;
}
}
+ queueSizeHistory.clear();
+ } else {
+ // The property did not change, but should it based on the queue size?
Maintain
+ // the last X minutes of queue sizes. If the queue size is always
larger than the number
+ // of Fate threads multiplied by some factor, then suggest that the
+ // MANAGER_FATE_THREADPOOL_SIZE be increased.
+ final long interval = Math.min(60, TimeUnit.MILLISECONDS
+
.toMinutes(conf.getTimeInMillis(Property.MANAGER_FATE_QUEUE_CHECK_INTERVAL)));
+ if (interval == 0) {
+ queueSizeHistory.clear();
+ } else {
+ final int sizeFactor =
conf.getCount(Property.MANAGER_FATE_QUEUE_CHECK_FACTOR);
+ if (queueSizeHistory.size() >= interval * 2) { // this task runs
every 30s
+ final int warnThreshold = configured * sizeFactor;
+ boolean needMoreThreads = true;
+ for (Integer i : queueSizeHistory) {
+ if (i < warnThreshold) {
+ needMoreThreads = false;
+ break;
+ }
+ }
+ if (needMoreThreads) {
+ log.warn(
+ "Fate queue size is {} times the number of Fate threads for
the last {} minutes,"
+ + " consider increasing property: {}",
+ sizeFactor, interval,
Property.MANAGER_FATE_THREADPOOL_SIZE.getKey());
+ // Clear the history so that we don't log for another 5 minutes.
+ queueSizeHistory.clear();
+ } else {
+ while (queueSizeHistory.size() >= interval * 2) {
+ queueSizeHistory.remove();
+ }
+ }
+ }
+ queueSizeHistory.add(workQueue.size());
Review Comment:
> That sounds good.
Updated in d1fdf91.
> that gives us the idle fate thread count w/o doing any work.
yeah, but it's just a point in time approximation. If the property is 60m,
then if it's zero for 120 intervals, then log the warning. I feel like that
might be a good solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
