Hi Cassandra Community, We have to expand our cluster and I tried to add the first node to the cluster and when the new node was bootstrapping , I noticed the error like below in the system.log, but the bootstrap process was successful .
We are on 3.11.5 . ERROR [MutationStage-7] 2023-03-01 07:01:40,026 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-7,5,main] java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for the maximum size of 16.000MiB I tried to add another node after 5 days of adding 1st node and the 2nd node also gave an error but this time the mutation size was bigger . ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 - Exception in thread Thread[BatchlogTasks:1,5,main] java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for the maximum size of 16.000MiB And on the first node( which was added 5 days ago), we saw messages like below till the 2nd node bootstrap was completed. "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB" but after the 2nd node bootstarp process was completed, we started seeing the message about hint byte is too large but there are no hint files in any of the nodes on hints directory . And the message about mutation too large has stopped popping in system.log on the first added node. Now we are seeing hints error as below on the first newly added node. ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228 - Exception in thread Thread[BatchlogTasks:1,5,main] java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large - the maximum size is 16777216 As per the application team , nothing changed . As per the workaround commitlog_segment_size_in_mb can be increased to accommodate the increased size but that doesnt seem to be a concrete solution and it can have performance impact , because by design intent the maximum allowed segment size is 50% of the configured commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments with large amounts of empty space. Looks like i am hitting https://issues.apache.org/jira/browse/CASSANDRA-15152 <https://issues.apache.org/jira/browse/CASSANDRA-15152> Anyone have any suggestions? Thanks Surbhi
