The performance implications would primarily be due to the challenge of handling mutations this
large themselves rather than the commitlog segment size. These would occupy large, contiguous
areas of heap and increase memory pressure in the process.Increasing
commit_log_segment_size_in_mb is likely the best / only approach, along with addressing
mutation size for future writes from the application.I'd also strongly recommend upgrading from
3.11.5 to 3.11.15. 3.11.5 was released about 3.5 years ago, with a large number of bugfixes
available in 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing the
version and performing a rolling restart of the instances.– ScottOn Mar 1, 2023, at 2:43 PM,
Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:Hi Cassandra Community,We have to expand
our cluster and I tried to add the first node to the cluster and when the new node was
bootstrapping , I noticed the error like below in the system.log, but the bootstrap process was
successful . We are on 3.11.5 .ERROR [MutationStage-7] 2023-03-01 07:01:40,026
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[MutationStage-7,5,main]java.lang.IllegalArgumentException: Mutation of 24.510MiB is too
large for the maximum size of 16.000MiBI tried to add another node after 5 days of adding 1st
node and the 2nd node also gave an error but this time the mutation size was bigger .ERROR
[BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 - Exception in thread
Thread[BatchlogTasks:1,5,main]java.lang.IllegalArgumentException: Mutation of 40.717MiB is too
large for the maximum size of 16.000MiBAnd on the first node( which was added 5 days ago), we
saw messages like below till the 2nd node bootstrap was completed. "Mutation of 24.510MiB
is too large for the maximum size of 16.000MiB"but after the 2nd node bootstarp process
was completed, we started seeing the message about hint byte is too large but there are no hint
files in any of the nodes on hints directory . And the message about mutation too large has
stopped popping in system.log on the first added node. Now we are seeing hints error as below
on the first newly added node. ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091
CassandraDaemon.java:228 - Exception in thread
Thread[BatchlogTasks:1,5,main]java.lang.IllegalArgumentException: Hint of 25700336 bytes is too
large - the maximum size is 16777216As per the application team , nothing changed . As per the
workaround commitlog_segment_size_in_mb can be increased to accommodate the increased size but
that doesnt seem to be a concrete solution and it can have performance impact , because by
design intent the maximum allowed segment size is 50% of the configured
commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments with large amounts
of empty space. Looks like i am hitting https://issues.apache.org/jira/browse/CASSANDRA-15152
Anyone have any suggestions?ThanksSurbhi
