[
https://issues.apache.org/jira/browse/CASSANDRA-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104135#comment-18104135
]
Dmitry Konstantinov commented on CASSANDRA-21569:
-------------------------------------------------
{quote}
It is incremented once per request by the single Netty thread that owns the
connection (no contention)
{quote}
unfortunately - it is not executed within Netty thread if we speak about actual
queries, not auth requests. The increment logic is invoked as a part of
org.apache.cassandra.transport.Dispatcher#processRequest within a common
org.apache.cassandra.transport.Dispatcher#requestExecutor. So, we may have a
contention here. In my old tests I observed ~0.12% of total CPU in cpu profile
("Dispatcher.\processRequest\:428" search pattern) - [^may17_allread_cpu.html]
At the same time I agree that for cases with 10k connections the read overhead
probably is too large and if such number of CQL connection per node is a real
scenario for big deployments then it makes sense to step back here. In my use
cases and my perf tests I have tens of connections per node, so the pattern is
very different.
Few comments regarding the benchmarks:
* ThreadLocalMetricsChurnBench assumes that we continuously start and stop
threads, is it a real scenario? (my impression is that majority of thread pools
in Cassandra are pretty stable from this point of view and the threads are just
parked, not started and stopped continuously)
* regarding "{*}Read{*} (the cost) – one {{system_views.clients}} scan,
us/op:" - it looks like we measure here only a counter read cost, what is about
total cost to invoke system_views.clients (or invoke nodetool clientstats) -
does this counter cost actually dominate in this total cost of nodetool
invocation or virtual table query?
> Revert ServerConnection.requests from ThreadLocalCounter to Dropwizard Counter
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-21569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21569
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
> Reporter: Francisco Guerrero
> Assignee: Francisco Guerrero
> Priority: Normal
> Attachments: ThreadLocalMetricsChurnBench.java,
> ThreadLocalMetricsReadScanBench.java, may17_allread_cpu.html
>
>
> h2. Summary
> {{ServerConnection.requests}} was changed from a Dropwizard {{Counter}}
> ({{LongAdder}}) to {{ThreadLocalCounter}} in CASSANDRA-21400. I would like to
> reconsider this change: it gains a negligible write speedup while making
> {{system_views.clients}} scans and metric reads under connection churn
> dramatically more expensive. I propose we revert *only this one site*.
> h2. The trade-off to consider
> {{ThreadLocalCounter}}'s only advantage is a cheaper {{inc()}} (a
> thread-local array bump vs a {{LongAdder}} cell). Its cost is that
> {{getCount()}} takes a process-wide shared read lock and sums the counter's
> slot across *every live thread*, and each counter close
> ({{recycleMetricId()}}) takes that lock's *write* side and scans every live
> thread. That trade only pays off for a small, fixed set of write-hot
> counters. {{ServerConnection.requests}} is the opposite:
> * *Benefit is negligible here.* It is incremented once per request by the
> single Netty thread that owns the connection (no contention), so
> {{LongAdder}} stays on its base cell. Measured delta is ~1-2 ns per increment
> (well under 0.1% of CQL request processing).
> * *Cost is large here.* It is per-connection (high cardinality, churny) and
> is read once-per-row by {{ClientsTable}} ({{system_views.clients}} /
> {{nodetool clientstats}}), so a single query becomes an O(connections x
> live-threads) scan under the shared lock, and connection open/close churn
> generates write-lock traffic that stalls reads of unrelated counters.
> h2. Evidence (JMH; single fork, JDK 11 -- read the ratios, not absolute
> numbers)
> *Write* (the benefit) -- per-increment cost is ~1-2 ns for a single
> low-contention counter; even the worst measured case (cache-polluted, 100
> counters) is ~35 ns/inc.
> *Read* (the cost) -- one {{system_views.clients}} scan, us/op:
> || Connections || Live threads || LongAdder || ThreadLocalCounter ||
> | 10,000 | 64 | 9.2 | 455 |
> | 10,000 | 256 | 9.6 | 6,786 |
> | 50,000 | 256 | 57.3 | 120,757 |
> {{LongAdder}} is flat in thread count; {{ThreadLocalCounter}} scales with
> connections x threads (~121 ms per scan at 50k connections / 256 threads).
> *Churn* (the cost) -- read throughput of a stable counter while N threads
> close counters, at 256 live threads: {{0 -> 10,667 ops/ms}}, {{1 -> 1,384
> (-7.7x)}}, {{4 -> 509 (-21x)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
