I have been doing some work on a cluster that is impacted by
https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through the
ticket prompted me to take a closer look at
org.apache.cassandra.concurrent.SEPExecutor. I am looking at the 3.0.14
code. I am a little confused about the Blocked and All Time Blocked columns
reported in nodetool tpstats and reported by StatusLogger. I understand
that there is a queue for tasks. In the case of RequestThreadPoolExecutor,
the size of that queue can be controlled via the
cassandra.max_queued_native_transport_requests system property.
I have been looking at SEPExecutor.addTask(FutureTask<?> task), and here is
my question. If the queue is full, as defined by SEPExector.maxTasksQueued,
are tasks rejected? I do not fully grok the code, but it looks like it is
possible for tasks to be rejected here (some code and comments omitted for
brevity):
public void addTask(FutureTask<?> task)
{
tasks.add(task);
...
else if (taskPermits >= maxTasksQueued)
{
WaitQueue.Signal s = hasRoom.register();
if (taskPermits(permits.get()) > maxTasksQueued)
{
if (takeWorkPermit(true))
pool.schedule(new Work(this))
metrics.totalBlocked.inc();
metrics.currentBlocked.inc();
s.awaitUninterruptibly();
metrics.currentBlocked.dec();
}
else
s.cancel();
}
}
The first thing that happens is that the task is added to the tasks queue.
pool.schedule() only gets called if takeWorkPermit() returns true. I am
still studying the code, but can someone explain what exactly happens when
the queue is full?
- John
