Jon Haddad created CASSANDRA-21656:
--------------------------------------
Summary: large mutations over-allocate on the commit log path
Key: CASSANDRA-21656
URL: https://issues.apache.org/jira/browse/CASSANDRA-21656
Project: Apache Cassandra
Issue Type: Bug
Reporter: Jon Haddad
Writing a large mutation to the commit log allocates far more memory than the
mutation itself, and it does so on every write. The allocation is native, not
heap, and it grows with mutation size. Mutations under about 1 MiB are
unaffected.
Measured on a 4 MiB mutation, 2400 writes, single thread, memory-mapped
segments, {{cassandra-6.0}} at {{{}38a7a783{}}}, async-profiler {{{}-e
nativemem{}}}:
||Measure||Value||
|native memory per write|10.42 MB|
|heap bytes per write|958.7|
|writes/sec, median of 5|850.5|
The commit log buffers the serialized mutation before copying it into the
segment, and the buffer is rebuilt from scratch on each large write. The
serialized size is already computed earlier in the same call and then
discarded, so it may be possible to size the segment slot from it and serialize
once, directly. Whether that size is exact enough to be trusted for the slot is
the open question.
I have a branch (wip) that removes the intermediate buffer. The serialized size
is already computed earlier in the same call, so the segment slot is sized from
it and the mutation is serialized straight into it.
Here are some stats:
||Measure||Before||After||Change||
|Native memory per write, 4 MiB mutation|10.42 MB|19.0 KB|-99.8%|
|Native memory total, 2400 writes|23.30 GiB|43.6 MiB|-99.8%|
|Heap bytes per write, 4 MiB mutation|958.7|547.3|-42.9%|
|Heap bytes per write, 200 B mutation|161.7|161.6|not resolved|
The branch is a little early, but it's my hope that this will help reduce the
pain people experience when writing large objects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
