[
https://issues.apache.org/jira/browse/CASSANDRA-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895405#action_12895405
]
Mike Malone edited comment on CASSANDRA-1358 at 8/4/10 4:26 PM:
----------------------------------------------------------------
So are the message deserializer threads blocking on a response from (or pushing
a task onto a queue for) a different thread pool then? The problem we saw / are
seeing is that the deserializer pool backs up and it causes the server to flap
because (apparently) it's unable to process gossip requests / responses in a
timely manner. Since all inter-node RPC goes through the deserializer, this
queue ends up causing all sorts of crazy havoc when it gets backed up.
Even supposing the MDP _isn't_ the bottleneck, the current code seems to be a
mistake. If the intent is to have a single threaded executor, the second
argument should be 1. If the intent is to have a multiple-threaded executor,
the first argument should not be 1.
was (Author: mmalone):
So are the message deserializer threads blocking on a response from (or
pushing a task onto a queue for) a different thread pool then? The problem we
saw / are seeing is that the deserializer pool backs up and it causes the
server to flap because (apparently) it's unable to process get gossip requests
/ responses of in a timely manner. Since all inter-node RPC goes through the
deserializer, this queue ends up causing all sorts of crazy havoc when it gets
backed up.
Even supposing the MDP _isn't_ the bottleneck, the current code seems to be a
mistake. If the intent is to have a single threaded executor, the second
argument should be 1. If the intent is to have a multiple-threaded executor,
the first argument should not be 1.
> Message deserializer pool will never grow beyond a single thread.
> -----------------------------------------------------------------
>
> Key: CASSANDRA-1358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1358
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.3
> Environment: All.
> Reporter: Mike Malone
> Priority: Minor
>
> The message deserialization process can become a bottleneck that prevents
> efficient resource utilization because the executor that manages the
> deserialization process will never grow beyond a single thread. The message
> deserializer executor is instantiated in the MessagingService constructor as
> a JMXEnableThreadPoolExecutor, which extends
> java.util.concurrent.ThreadPoolExecutor. The thread pool is instantiated with
> a corePoolSize of 1 and a maximumPoolSize of
> Runtime.getRuntime().availableProcessors(). But, according to the
> ThreadPoolExecutor documentation "using an unbounded queue (for example a
> LinkedBlockingQueue without a predefined capacity) will cause new tasks to be
> queued in cases where all corePoolSize threads are busy. Thus, no more than
> corePoolSize threads will ever be created. (And the value of the
> maximumPoolSize therefore doesn't have any effect.)"
> The message deserializer pool uses a LinkedBlockingQueue, so there will never
> be more than one deserialization thread. This issue became a problem in our
> production cluster when the MESSAGE-DESERIALIZER-POOL began to back up on a
> node that was only lightly loaded. We increased the core pool size to 4 and
> the situation improved, but the deserializer pool was still backing up while
> the machine was not fully utilized (less than 100% CPU utilization). This
> leads me to think that the deserializer thread is blocking on some sort of
> I/O, which seems like it shouldn't happen.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
