Are you using materialized views or secondary indices? -- Jeff Jirsa
> On Aug 6, 2018, at 3:49 PM, Laszlo Szabo <laszlo.viktor.sz...@gmail.com> > wrote: > > Hello All, > > I'm having JVM unstable / OOM errors when attempting to auto bootstrap a 9th > node to an existing 8 node cluster (256 tokens). Each machine has 24 cores > 148GB RAM and 10TB (2TB used). Under normal operation the 8 nodes have JVM > memory configured with Xms35G and Xmx35G, and handle 2-4 billion inserts per > day. There are never updates, deletes, or sparsely populated rows. > > For the bootstrap node, I've tried memory values from 35GB to 135GB in 10GB > increments. I've tried using both memtable_allocation_types (heap_buffers and > offheap_buffers). I've not tried modifying the memtable_cleanup_threshold > but instead have tried memtable_flush_writers from 2 to 8. I've tried > memtable_(off)heap_space_in_mb from 20000 to 60000. I've tried both CMS and > G1 garbage collection with various settings. > > Typically, after streaming about ~2TB of data, CPU load will hit a maximum, > and the "nodetool info" heap memory will, over the course of an hour, > approach the maximum. At that point, CPU load will drop to a single thread > with minimal activity until the system becomes unstable and eventually the > OOM error occurs. > > Excerpt of the system log is below, and what I consistently see is the > MemtableFlushWriter and the MemtableReclaimMemory pending queues grow as the > memory becomes depleted, but the number of completed seems to stop changing a > few minutes after the CPU load spikes. > > One other data point is there seems to be a huge number of mutations that > occur after most of the stream has occured. Concurrent_writes is set at 256 > with the queue getting as high as 200K before dropping down. > > Any suggestions for yaml changes or jvm changes? JVM.options is currently > the default with the memory set to the max, the current YAML file is below. > > Thanks! > > >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,329 StatusLogger.java:51 - >> MutationStage 1 2 191498052 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,331 StatusLogger.java:51 - >> ViewMutationStage 0 0 0 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,338 StatusLogger.java:51 - >> PerDiskMemtableFlushWriter_0 0 0 5865 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,343 StatusLogger.java:51 - >> ReadStage 0 0 0 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,347 StatusLogger.java:51 - >> ValidationExecutor 0 0 0 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,360 StatusLogger.java:51 - >> RequestResponseStage 0 0 8 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,380 StatusLogger.java:51 - >> Sampler 0 0 0 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,382 StatusLogger.java:51 - >> MemtableFlushWriter 8 74293 4716 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,388 StatusLogger.java:51 - >> ReadRepairStage 0 0 0 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,389 StatusLogger.java:51 - >> CounterMutationStage 0 0 0 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,404 StatusLogger.java:51 - >> MiscStage 0 0 0 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,407 StatusLogger.java:51 - >> CompactionExecutor 8 13 493 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,410 StatusLogger.java:51 - >> InternalResponseStage 0 0 16 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,413 StatusLogger.java:51 - >> MemtableReclaimMemory 1 6066 356 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,421 StatusLogger.java:51 - >> AntiEntropyStage 0 0 0 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,430 StatusLogger.java:51 - >> CacheCleanupExecutor 0 0 0 0 >> 0 >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,431 StatusLogger.java:51 - >> PendingRangeCalculator 0 0 9 0 >> 0 >> INFO [Service Thread] 2018-08-06 17:49:26,436 StatusLogger.java:61 - >> CompactionManager 8 19 > > > > > Current Yaml >> num_tokens: 256 > >> hinted_handoff_enabled: true > >> hinted_handoff_throttle_in_kb: 10240 > >> max_hints_delivery_threads: 8 > >> hints_flush_period_in_ms: 10000 > >> max_hints_file_size_in_mb: 128 > >> batchlog_replay_throttle_in_kb: 10240 > >> authenticator: AllowAllAuthenticator > >> authorizer: AllowAllAuthorizer > >> role_manager: CassandraRoleManager > >> roles_validity_in_ms: 2000 > >> permissions_validity_in_ms: 2000 > >> credentials_validity_in_ms: 2000 > >> partitioner: org.apache.cassandra.dht.Murmur3Partitioner > >> data_file_directories: > >> - /data/cassandra/data > >> commitlog_directory: /data/cassandra/commitlog > >> cdc_enabled: false > >> disk_failure_policy: stop > >> commit_failure_policy: stop > >> prepared_statements_cache_size_mb: > >> thrift_prepared_statements_cache_size_mb: > >> key_cache_size_in_mb: > >> key_cache_save_period: 14400 > >> row_cache_size_in_mb: 0 > >> row_cache_save_period: 0 > >> counter_cache_size_in_mb: > >> counter_cache_save_period: 7200 > >> saved_caches_directory: /data/cassandra/saved_caches > >> commitlog_sync: periodic > >> commitlog_sync_period_in_ms: 10000 > >> commitlog_segment_size_in_mb: 32 > >> seed_provider: > >> - class_name: org.apache.cassandra.locator.SimpleSeedProvider > >> parameters: > >> - seeds: "10.1.1.11,10.1.1.12,10.1.1.13" > >> concurrent_reads: 128 > >> concurrent_writes: 256 > >> concurrent_counter_writes: 96 > >> concurrent_materialized_view_writes: 32 > >> disk_optimization_strategy: spinning > >> memtable_heap_space_in_mb: 61440 > >> memtable_offheap_space_in_mb: 61440 > >> memtable_allocation_type: heap_buffers > >> commitlog_total_space_in_mb: 81920 > >> memtable_flush_writers: 8 >
