Michael Semb Wever created CASSANDRA-21657:
----------------------------------------------
Summary: The hints write buffer is not released when the write
executor shuts down
Key: CASSANDRA-21657
URL: https://issues.apache.org/jira/browse/CASSANDRA-21657
Project: Apache Cassandra
Issue Type: Bug
Components: Consistency/Hints
Reporter: Michael Semb Wever
{{HintsWriteExecutor}} allocates a 256 KiB direct buffer that every write task
shares:
{code:java}
writeBuffer = ByteBuffer.allocateDirect(WRITE_BUFFER_SIZE);
{code}
{{shutdownBlocking}} shuts the executor down and leaves the buffer allocated. A
process that shuts the hints service down and starts it again loses 256 KiB of
direct memory each time. The in-JVM test framework does exactly that, once per
instance restart, so a long dtest run holds direct memory that no instance can
use.
A node shutdown in production ends the process, so the priority is Low; the
cost falls on the test framework and on any other caller that restarts the
service in one process.
The patch releases the buffer with {{FileUtils.clean}} after the executor
terminates. The release waits for termination, because a write task that still
runs holds the buffer, and a shutdown that gives up after its minute reports
the buffer it keeps rather than freeing memory a task may write to.
Patch:
[mck/upstream/hints-write-buffer-release/5.0|https://github.com/thelastpickle/cassandra/tree/mck/upstream/hints-write-buffer-release/5.0]
Provenance:
[00003d27fa|https://github.com/datastax/cassandra/commit/00003d27faf2241245436d9440d740e1cd18aaa8]
by [~jlewandowski]. That commit releases the buffer in a {{finally}} block and
carries further work on the buffer pool and on the in-JVM instance shutdown,
which this patch leaves out. The {{finally}} block frees the buffer even when a
write task is still running, so this patch tests the termination result
instead, and adds the regression test the commit lacks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
