C. Scott Andreas created CASSANDRA-21507:
--------------------------------------------
Summary: Flaky dtest -
largecolumn_test.TestLargeColumn::test_cleanup fails due to memory exhaustion
Key: CASSANDRA-21507
URL: https://issues.apache.org/jira/browse/CASSANDRA-21507
Project: Apache Cassandra
Issue Type: Bug
Components: Test/dtest/python
Reporter: C. Scott Andreas
{{largecolumn_test.TestLargeColumn::test_cleanup}} verifies that off-heap
(direct) memory does not grow proportionally across repeated large-column
requests. It attempts reads of multiple 64MB columns to assert this, and
occasionally exhausts available memory. But the exact column size is incidental
to that goal — the test just intends to assert that off-heap memory allocated
doesn't increase.
The ccm node heap is a fixed 500MB. At the original {{LARGE_COLUMN_SIZE =
63MiB}} (and even at 48MiB), a node exhausts its 500MB heap and OOMs before the
direct-memory assertion can run. The node is killed
({{{}-XX:OnOutOfMemoryError=kill -9{}}}, ~294MB heap dump written), and the
subsequent {{CL=ALL}} read fails with {{{}ReadTimeoutException{}}}("2 required,
1 responded") / cassandra-stress {{{}ToolError{}}}.
*Proposed Change:*
Since the column size is ~incidental to the test's intention, we can probably
lower it from 64MB to 32MB.
At 32MB, the test passes 100/100 runs.
*Potential patch (model-assisted):*
[https://github.com/cscotta/cassandra-dtest/pull/2/changes]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
