Are you using vnodes by any chance? If so, how many? How many nodes and what's the replication factor? How was data inserted (at what consistency level)?
Streaming might create a large number of sstables with vnodes (see CASSANDRA-10495), so in case data is inconsistent between nodes (detected during the validation phase) a large number of sstables might be created during the streaming phase of the repair. This might happen with or without incremental repairs. There is also a migration step from non-incremental to incremental repair that must be followed for LCS tables to avoid recompacting all sstables again. You may find more information in the migration procedure on http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html 2016-02-10 14:16 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: > Hello Kai > > This is for *cf3* > > nodetool cfstats pns_nonreg_bench.cf3 -H > Keyspace: pns_nonreg_bench > Read Count: 23594 > Read Latency: 1.2980987539204882 ms. > Write Count: 148161 > Write Latency: 0.04608940274431193 ms. > Pending Flushes: 0 > Table: cf3 > SSTable count: 11489 > SSTables in each level: [11485/4, 4, 0, 0, 0, 0, 0, 0, 0] > Space used (live): 954.89 MB > Space used (total): 954.89 MB > Space used by snapshots (total): 0 bytes > Off heap memory used (total): 3.74 MB > SSTable Compression Ratio: 0.4245371280396339 > Number of keys (estimate): 1051613 > Memtable cell count: 0 > Memtable data size: 0 bytes > Memtable off heap memory used: 0 bytes > Memtable switch count: 889 > Local read count: 23594 > Local read latency: 1.299 ms > Local write count: 148161 > Local write latency: 0.047 ms > Pending flushes: 0 > Bloom filter false positives: 16609 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 1.64 MB > Bloom filter off heap memory used: 1.55 MB > Index summary off heap memory used: 1.88 MB > Compression metadata off heap memory used: 318.56 KB > Compacted partition minimum bytes: 643 bytes > Compacted partition maximum bytes: 924 bytes > Compacted partition mean bytes: 914 bytes > Average live cells per slice (last five minutes): 1.0 > Maximum live cells per slice (last five minutes): 1.0 > Average tombstones per slice (last five minutes): 0.0 > Maximum tombstones per slice (last five minutes): 0.0 > > ---------------- > > This is for *cf1* > > nodetool cfstats pns_nonreg_bench.cf1 -H > Keyspace: pns_nonreg_bench > Read Count: 24207 > Read Latency: 0.9072712851654481 ms. > Write Count: 148380 > Write Latency: 0.04561746866154468 ms. > Pending Flushes: 0 > Table: cf1 > SSTable count: 1942 > > It seems that nodetool does not retrieve the rest of the information, I > think it is due to compactions > > nodetool tpstats > CompactionExecutor 2 83 1424 > 0 0 > > > > > > Saludos > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > > On Wed, Feb 10, 2016 at 6:00 PM, Kai Wang <dep...@gmail.com> wrote: > >> Jean, >> >> What does your cfstats look like? Especially "SSTables in each level" >> line. >> >> On Wed, Feb 10, 2016 at 8:33 AM, Jean Carlo <jean.jeancar...@gmail.com> >> wrote: >> >>> Hello guys! >>> >>> I am testing the repair inc in my custer cassandra. I am doing my test >>> over these tables >>> >>> *CREATE TABLE pns_nonreg_bench.cf3* ( >>> s text, >>> sp int, >>> d text, >>> dp int, >>> m map<text, text>, >>> t timestamp, >>> PRIMARY KEY (s, sp, d, dp) >>> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC) >>> >>> AND compaction = {'class': 'org.apache.cassandra.db.compaction. >>> *LeveledCompactionStrategy*'} >>> AND compression = {'sstable_compression': >>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'} >>> >>> *CREATE TABLE pns_nonreg_bench.cf1* ( >>> ise text PRIMARY KEY, >>> int_col int, >>> text_col text, >>> ts_col timestamp, >>> uuid_col uuid >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND compaction = {'class': 'org.apache.cassandra.db.compaction. >>> *LeveledCompactionStrategy*'} >>> AND compression = {'sstable_compression': >>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'} >>> >>> >>> >>> *table cf1 Space used (live): 665.7 MB* >>> >>> *table cf2* >>> * Space used (live): 697.03 MB* >>> >>> It happens that when I do repair -inc -par on theses tables, *cf2 got a >>> pick of 3k sstables*. When the repair finish, it takes 30 min or more >>> to finish all the compactations and return to 6 sstable. >>> >>> I am a little concern about if this will happen on production. is it >>> normal? >>> >>> Saludos >>> >>> Jean Carlo >>> >>> "The best way to predict the future is to invent it" Alan Kay >>> >> >> >
