[
https://issues.apache.org/jira/browse/CASSANDRA-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028729#comment-18028729
]
Dmitry Konstantinov commented on CASSANDRA-20226:
-------------------------------------------------
Same test and env but withÂ
{code:java}
memtable_allocation_type: heap_buffers
{code}
[^test_results_m8i.4xlarge_heap_buffers.html]
!test_results_m8i.4xlarge_heap_buffers.png|width=1000!
h4. Before
{code:java}
op rate : 78,804 op/s [insert: 78,804 op/s]
partition rate : 78,804 pk/s [insert: 78,804 pk/s]
row rate : 788,037 row/s [insert: 788,037 row/s]
latency mean : 1.3 ms [insert: 1.3 ms]
latency median : 1.0 ms [insert: 1.0 ms]
latency 95th percentile : 2.3 ms [insert: 2.3 ms]
latency 99th percentile : 4.3 ms [insert: 4.3 ms]
latency 99.9th percentile : 21.5 ms [insert: 21.5 ms]
latency max : 431.8 ms [insert: 431.8 ms]
total gc count : 0
total gc memory : 0 B
total gc time : 0.0 seconds
avg gc time : NaN ms
stddev gc time : 0.0 ms
Total operation time : 00:02:06
{code}
h4. After
{code:java}
op rate : 131,069 op/s [insert: 131,069 op/s]
partition rate : 131,069 pk/s [insert: 131,069 pk/s]
row rate : 1,310,685 row/s [insert: 1,310,685 row/s]
latency mean : 0.7 ms [insert: 0.7 ms]
latency median : 0.5 ms [insert: 0.5 ms]
latency 95th percentile : 0.9 ms [insert: 0.9 ms]
latency 99th percentile : 2.5 ms [insert: 2.5 ms]
latency 99.9th percentile : 22.3 ms [insert: 22.3 ms]
latency max : 981.5 ms [insert: 981.5 ms]
total gc count : 0
total gc memory : 0 B
total gc time : 0.0 seconds
avg gc time : NaN ms
stddev gc time : 0.0 ms
Total operation time : 00:01:16
{code}
> Reduce contention in NativeAllocator.allocate
> ---------------------------------------------
>
> Key: CASSANDRA-20226
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20226
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Local/Memtable
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 5.x
>
> Attachments: 5.1_batch_LongAdder.html, 5.1_batch_addAndGet.html,
> 5.1_batch_alloc_batching.html, 5.1_batch_baseline.html,
> 5.1_batch_pad_allocated.html, cpu_profile_batch.html,
> image-2025-01-20-23-38-58-896.png, profile.yaml,
> test_results_m8i.4xlarge_heap_buffers.html,
> test_results_m8i.4xlarge_heap_buffers.png,
> test_results_m8i.4xlarge_offheap_objects.html,
> test_results_m8i.4xlarge_offheap_objects.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For a high insert batch rate it looks like we have a bottleneck in
> NativeAllocator.allocate probably caused by contention within the logic.
> !image-2025-01-20-23-38-58-896.png|width=300!
> [^cpu_profile_batch.html]
> The logic has at least the following 2 potential places to assess:
> # allocation cycle in MemtablePool.SubPool#tryAllocate. This logic has a
> while loop with a CAS, which can be non-efficient under a high contention,
> similar to CASSANDRA-15922 we can try to replace it with addAndGet (need to
> check if it does not break the allocator logic)
> # swap region logic in NativeAllocator.trySwapRegion (under a high insert
> rate 1MiB regions can be swapped quite frequently)
> Reproducing test details:
> * test logic
> {code:java}
> ./tools/bin/cassandra-stress "user profile=./profile.yaml no-warmup
> ops(insert=1) n=10m" -rate threads=100 -node somenode
> {code}
> * Cassandra version: 5.0.3
> * configuration changes compared to default:
> {code:java}
> memtable_allocation_type: offheap_objects
> memtable:
> configurations:
> skiplist:
> class_name: SkipListMemtable
> trie:
> class_name: TrieMemtable
> parameters:
> shards: 32
> default:
> inherits: trie
> {code}
> * 1 node cluster
> * OpenJDK jdk-17.0.12+7
> * Linux kernel: 4.18.0-240.el8.x86_64
> * CPU: 16 cores, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
> * RAM: 46GiB
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
