Thanks Scott On Wed, Mar 1, 2023 at 4:00 PM C. Scott Andreas <sc...@paradoxica.net> wrote:
> The performance implications would primarily be due to the challenge of > handling mutations this large themselves rather than the commitlog segment > size. These would occupy large, contiguous areas of heap and increase > memory pressure in the process. > > Increasing commit_log_segment_size_in_mb is likely the best / only > approach, along with addressing mutation size for future writes from the > application. > > I'd also strongly recommend upgrading from 3.11.5 to 3.11.15. 3.11.5 was > released about 3.5 years ago, with a large number of bugfixes available in > 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing > the version and performing a rolling restart of the instances. > > > – Scott > > On Mar 1, 2023, at 2:43 PM, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: > > > Hi Cassandra Community, > > We have to expand our cluster and I tried to add the first node to the > cluster and when the new node was bootstrapping , I noticed the error like > below in the system.log, but the bootstrap process was successful . > > We are on 3.11.5 . > > ERROR [MutationStage-7] 2023-03-01 07:01:40,026 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[MutationStage-7,5,main] > > java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for > the maximum size of 16.000MiB > > I tried to add another node after 5 days of adding 1st node and the 2nd > node also gave an error but this time the mutation size was bigger . > > ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 > - Exception in thread Thread[BatchlogTasks:1,5,main] > > java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for > the maximum size of 16.000MiB > > And on the first node( which was added 5 days ago), we saw messages like > below till the 2nd node bootstrap was completed. > "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB" > but after the 2nd node bootstarp process was completed, we started seeing > the message about hint byte is too large but there are no hint files in any > of the nodes on hints directory . And the message about mutation too large > has stopped popping in system.log on the first added node. Now we are > seeing hints error as below on the first newly added node. > > ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228 > - Exception in thread Thread[BatchlogTasks:1,5,main] > > java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large - > the maximum size is 16777216 > > As per the application team , nothing changed . > > As per the workaround commitlog_segment_size_in_mb can be increased to > accommodate the increased size but that doesnt seem to be a concrete > solution and it can have performance impact , because by design intent > the maximum allowed segment size is 50% of the configured > commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments > with large amounts of empty space. > > Looks like i am hitting > https://issues.apache.org/jira/browse/CASSANDRA-15152 > <https://issues.apache.org/jira/browse/CASSANDRA-15152> > > Anyone have any suggestions? > > Thanks > Surbhi > > > > > > > > >
