[
https://issues.apache.org/jira/browse/CASSANDRA-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13829193#comment-13829193
]
Benedict commented on CASSANDRA-1632:
-------------------------------------
I'm dubious of this kind of performance critical code path ever having a
hardcoded sleep time - it should be reactive to the workload. What you're doing
is predicting the max work rate without knowing the actual conditions and, in
particular, any future improvements. 100ms right now is already dangerously
long, and would noticeably cap throughput after a brief pause in work, and also
potentially spike latency if you have such a peak; 10ms might well avoid either
for now, but hopefully that too will be dangerous soon, and either way what
does waiting for a fixed interval win for us?
IMO if you want to roll in this change (which, as I say, will probably be short
lived) it would be better to poll the queue in the normal fashion if we drain
no items, add it to the drain queue and loop back to re-attempt a drain. This
might mean processing smaller batches in some cases, but guarantees we never
rate limit. Given that a sleep will guarantee a context switch and pause time
of probably tens of micros (unless we're on a real-time scheduler, in which
case the user will probably consider this an even worse idea :-), that should
be enough time to populate the queue with enough items to warrant processing
immediately anyway, unless there's a low rate of additions and then we aren't
winning anything.
> Thread workflow and cpu affinity
> --------------------------------
>
> Key: CASSANDRA-1632
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1632
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Assignee: Jason Brown
> Labels: performance
> Attachments: 1632-v2.txt, 1632_batchRead-v1.diff,
> threadAff_reads.txt, threadAff_writes.txt
>
>
> Here are some thoughts I wanted to write down, we need to run some serious
> benchmarks to see the benefits:
> 1) All thread pools for our stages use a shared queue per stage. For some
> stages we could move to a model where each thread has its own queue. This
> would reduce lock contention on the shared queue. This workload only suits
> the stages that have no variance, else you run into thread starvation. Some
> stages that this might work: ROW-MUTATION.
> 2) Set cpu affinity for each thread in each stage. If we can pin threads to
> specific cores, and control the workflow of a message from Thrift down to
> each stage, we should see improvements on reducing L1 cache misses. We would
> need to build a JNI extension (to set cpu affinity), as I could not find
> anywhere in JDK where it was exposed.
> 3) Batching the delivery of requests across stage boundaries. Peter Schuller
> hasn't looked deep enough yet into the JDK, but he thinks there may be
> significant improvements to be had there. Especially in high-throughput
> situations. If on each consumption you were to consume everything in the
> queue, rather than implying a synchronization point in between each request.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
