High load on cassandra node
HI I have a 6 node cluster and I ran a major compaction on node 1 but I found that the load reached very high levels on node 2. Is this explainable? Attaching tpstats and metrics: cassandra-2 ~]$ nodetool tpstats Pool NameActive Pending Completed Blocked All time blocked MutationStage 0 0 185152938 0 0 ReadStage 0 0490 0 0 RequestResponseStage 0 0 168660091 0 0 ReadRepairStage 0 0 21247 0 0 ReplicateOnWriteStage32 6186 88699535 0 7163 MiscStage 0 0 0 0 0 HintedHandoff 0 1 1090 0 0 FlushWriter 0 0 2059 0 13 MemoryMeter 0 0 3922 0 0 GossipStage 0 02246873 0 0 CacheCleanupExecutor 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 CompactionExecutor0 0 12353 0 0 ValidationExecutor0 0 0 0 0 MigrationStage0 0 1 0 0 commitlog_archiver0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 PendingRangeCalculator0 0 16 0 0 MemtablePostFlusher 0 0 10932 0 0 Message type Dropped READ 49051 RANGE_SLICE 0 _TRACE 0 MUTATION 269 COUNTER_MUTATION 185 BINARY 0 REQUEST_RESPONSE 0 PAGED_RANGE 0 READ_REPAIR 0 Also I saw that the nativetransportreqests was 23 in active and 23 in pending, I found this in opscenter. Any settings i can make to keep the load under control? Appreciate any help.. Thanks
Re: High load on cassandra node
We heavily use counters, will upgrade and check if it solves the issue... The current cluster version is 2.0.14 We do a lot of delete operations and do major compaction to remove the tombstones. Is there any better way? On 1 July 2015 at 20:02, Sebastian Estevez wrote: > Looks like a CASSANDRA-6405 (replicate on write is the counter tp). Upgrade > to the latest 2.1 version and let us know if he situation improves. > > Major compactions are usually a bad idea by the way. Do you really want one > huge sstable? > > On Jul 1, 2015 10:03 AM, "Jayapandian Ponraj" wrote: >> >> HI I have a 6 node cluster and I ran a major compaction on node 1 but >> I found that the load reached very high levels on node 2. Is this >> explainable? >> >> Attaching tpstats and metrics: >> >> cassandra-2 ~]$ nodetool tpstats >> Pool NameActive Pending Completed Blocked >> All time blocked >> MutationStage 0 0 185152938 0 >> 0 >> ReadStage 0 0490 0 >> 0 >> RequestResponseStage 0 0 168660091 0 >> 0 >> ReadRepairStage 0 0 21247 0 >> 0 >> ReplicateOnWriteStage32 6186 88699535 0 >> 7163 >> MiscStage 0 0 0 0 >> 0 >> HintedHandoff 0 1 1090 0 >> 0 >> FlushWriter 0 0 2059 0 >>13 >> MemoryMeter 0 0 3922 0 >> 0 >> GossipStage 0 02246873 0 >> 0 >> CacheCleanupExecutor 0 0 0 0 >> 0 >> InternalResponseStage 0 0 0 0 >> 0 >> CompactionExecutor0 0 12353 0 >> 0 >> ValidationExecutor0 0 0 0 >> 0 >> MigrationStage0 0 1 0 >> 0 >> commitlog_archiver0 0 0 0 >> 0 >> AntiEntropyStage 0 0 0 0 >> 0 >> PendingRangeCalculator0 0 16 0 >> 0 >> MemtablePostFlusher 0 0 10932 0 >> 0 >> >> Message type Dropped >> READ 49051 >> RANGE_SLICE 0 >> _TRACE 0 >> MUTATION 269 >> COUNTER_MUTATION 185 >> BINARY 0 >> REQUEST_RESPONSE 0 >> PAGED_RANGE 0 >> READ_REPAIR 0 >> >> >> >> Also I saw that the nativetransportreqests was 23 in active and 23 in >> pending, I found this in opscenter. >> >> >> Any settings i can make to keep the load under control? >> Appreciate any help.. Thanks
Last two metrics of cfstats
The last two metrics of cfstats shows zero for all the tables we have Average live cells per slice (last five minutes): 0.0 Average tombstones per slice (last five minutes): 0.0 What do these mean and why are they always zero?
Change hostname in metrics reporter
How to override_hostname for ganglia reporter ganglia: - period: 300 timeunit: 'SECONDS' gmondConf: '/etc/ganglia/gmond.conf' groupPrefix: "cassandra" predicate: color: "white" useQualifiedName: true patterns: - "^org.apache.cassandra.metrics.Compaction.+"
Minor compaction not triggered
Hi I have a cassandra cluster of 6 nodes, with DateTiered compaction for the tables/CFs For some reason the minor compaction never happens. I have enabled debug logging and I don't see any debug logs related to compaction like the following https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L150 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java#L127 As a result of no compactions, now the cluster has more than 50K SStables per node. How do i debug this issue further? Appreciate any help..
Re: Minor compaction not triggered
Can anyone guide me with any pointers plz On 17 June 2015 at 17:05, Jayapandian Ponraj wrote: > Hi > > I have a cassandra cluster of 6 nodes, with DateTiered compaction for > the tables/CFs > For some reason the minor compaction never happens. > I have enabled debug logging and I don't see any debug logs related to > compaction like the following > > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L150 > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java#L127 > > As a result of no compactions, now the cluster has more than 50K > SStables per node. > How do i debug this issue further? > Appreciate any help..
Re: Minor compaction not triggered
Hi, Curently the cluster runs 2.0.14 on all nodes, will try to upgrade to 2.0.15 The issue is in all the nodes.. simple restart as in a rolling restart of all the nodes? On 18 June 2015 at 15:18, Alain RODRIGUEZ wrote: > Hi, > > From what I read there are some issue using DTCS. You should probably go to > the last minor version. > > You should also give more context so people can help. Have you tried a > simple restart ? Does it happen on all your nodes ? > > C*heers, > > Alain > > 2015-06-18 11:42 GMT+02:00 Jayapandian Ponraj : >> >> Can anyone guide me with any pointers plz >> >> On 17 June 2015 at 17:05, Jayapandian Ponraj wrote: >> > Hi >> > >> > I have a cassandra cluster of 6 nodes, with DateTiered compaction for >> > the tables/CFs >> > For some reason the minor compaction never happens. >> > I have enabled debug logging and I don't see any debug logs related to >> > compaction like the following >> > >> > >> > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L150 >> > >> > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java#L127 >> > >> > As a result of no compactions, now the cluster has more than 50K >> > SStables per node. >> > How do i debug this issue further? >> > Appreciate any help.. > >
