I imagine 3.0.16 has THIS bug, but it has far fewer other real bugs.
On Tue, Apr 17, 2018 at 1:56 PM, Luigi Tagliamonte < luigi.tagliamont...@gmail.com> wrote: > Thank you Jeff, > my backup scripts works using the cf folders on disk :) > it parses all the keyspaces and for each performs: nodetool flush > ${keyspace} ${cf} and then nodetool snapshot ${keyspace} -cf ${cf} > Does 3.0.16 not having this "bug"? > Regards > L. > > On Tue, Apr 17, 2018 at 1:50 PM, Jeff Jirsa <jji...@gmail.com> wrote: > >> It's probably not ideal, but also not really a bug. We need to create the >> table in the schema to see if it exists on disk so we know whether or not >> to migrate it, and when we learn it's empty, we remove it from the schema >> but we dont delete the directory. It's not great, but it's not going to >> cause you any problems. >> >> That said: 3.0.11 may cause you problems, you should strongly consider >> 3.0.16 instead. >> >> On Tue, Apr 17, 2018 at 1:47 PM, Luigi Tagliamonte < >> luigi.tagliamont...@gmail.com> wrote: >> >>> Hello everybody, >>> i'm having a problem with a brand new cassandra:3.0.11 node. The >>> following tables belonging to the system keyspace: >>> >>> - schema_aggregates >>> - schema_columnfamilies >>> - schema_columns >>> - schema_functions >>> - schema_keyspaces >>> - schema_triggers >>> - schema_usertypes >>> >>> >>> get initialised on disk: >>> >>> *root@ip-10-48-93-149:/var/lib/cassandra/data/system# pwd* >>> /var/lib/cassandra/data/system >>> >>> *root@ip-10-48-93-149:/var/lib/cassandra/data/system# ls -1* >>> IndexInfo-9f5c6374d48532299a0a5094af9ad1e3 >>> available_ranges-c539fcabd65a31d18133d25605643ee3 >>> batches-919a4bc57a333573b03e13fc3f68b465 >>> batchlog-0290003c977e397cac3efdfdc01d626b >>> built_views-4b3c50a9ea873d7691016dbc9c38494a >>> compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca >>> hints-2666e20573ef38b390fefecf96e8f0c7 >>> local-7ad54392bcdd35a684174e047860b377 >>> paxos-b7b7f0c2fd0a34108c053ef614bb7c2d >>> peer_events-59dfeaea8db2334191ef109974d81484 >>> peers-37f71aca7dc2383ba70672528af04d4f >>> range_xfers-55d764384e553f8b9f6e676d4af3976d >>> schema_aggregates-a5fc57fc9d6c3bfda3fc01ad54686fea >>> schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697 >>> schema_columns-296e9c049bec3085827dc17d3df2122a >>> schema_functions-d1b675fe2b503ca48e49c0f81989dcad >>> schema_keyspaces-b0f2235744583cdb9631c43e59ce3676 >>> schema_triggers-0359bc7171233ee19a4ab9dfb11fc125 >>> schema_usertypes-3aa752254f82350b8d5c430fa221fa0a >>> size_estimates-618f817b005f3678b8a453f3930b8e86 >>> sstable_activity-5a1ff267ace03f128563cfae6103c65e >>> views_builds_in_progress-b7f2c10878cd3c809cd5d609b2bd149c >>> >>> >>> >>> but if I describe the system keyspace those cf are not present. >>> >>> cassandra@cqlsh> DESCRIBE KEYSPACE system; >>> >>> CREATE KEYSPACE system WITH replication = {'class': 'LocalStrategy'} >>> AND durable_writes = true; >>> >>> CREATE TABLE system.available_ranges ( >>> keyspace_name text PRIMARY KEY, >>> ranges set<blob> >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'available keyspace/ranges during bootstrap/replace >>> that are ready to be served' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.batches ( >>> id timeuuid PRIMARY KEY, >>> mutations list<blob>, >>> version int >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'batches awaiting replay' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '2'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system."IndexInfo" ( >>> table_name text, >>> index_name text, >>> PRIMARY KEY (table_name, index_name) >>> ) WITH COMPACT STORAGE >>> AND CLUSTERING ORDER BY (index_name ASC) >>> AND bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'built column indexes' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.views_builds_in_progress ( >>> keyspace_name text, >>> view_name text, >>> generation_number int, >>> last_token text, >>> PRIMARY KEY (keyspace_name, view_name) >>> ) WITH CLUSTERING ORDER BY (view_name ASC) >>> AND bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'views builds current progress' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.peers ( >>> peer inet PRIMARY KEY, >>> data_center text, >>> host_id uuid, >>> preferred_ip inet, >>> rack text, >>> release_version text, >>> rpc_address inet, >>> schema_version uuid, >>> tokens set<text> >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'information about known peers in the cluster' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.compaction_history ( >>> id uuid PRIMARY KEY, >>> bytes_in bigint, >>> bytes_out bigint, >>> columnfamily_name text, >>> compacted_at timestamp, >>> keyspace_name text, >>> rows_merged map<int, bigint> >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'week-long compaction history' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 604800 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.sstable_activity ( >>> keyspace_name text, >>> columnfamily_name text, >>> generation int, >>> rate_120m double, >>> rate_15m double, >>> PRIMARY KEY ((keyspace_name, columnfamily_name, generation)) >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'historic sstable read rates' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.peer_events ( >>> peer inet PRIMARY KEY, >>> hints_dropped map<uuid, int> >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'events related to peers' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.paxos ( >>> row_key blob, >>> cf_id uuid, >>> in_progress_ballot timeuuid, >>> most_recent_commit blob, >>> most_recent_commit_at timeuuid, >>> most_recent_commit_version int, >>> proposal blob, >>> proposal_ballot timeuuid, >>> proposal_version int, >>> PRIMARY KEY (row_key, cf_id) >>> ) WITH CLUSTERING ORDER BY (cf_id ASC) >>> AND bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'in-progress paxos proposals' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.LeveledCompactionStrategy'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.batchlog ( >>> id uuid PRIMARY KEY, >>> data blob, >>> version int, >>> written_at timestamp >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = '*DEPRECATED* batchlog entries' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '2'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.size_estimates ( >>> keyspace_name text, >>> table_name text, >>> range_start text, >>> range_end text, >>> mean_partition_size bigint, >>> partitions_count bigint, >>> PRIMARY KEY (keyspace_name, table_name, range_start, range_end) >>> ) WITH CLUSTERING ORDER BY (table_name ASC, range_start ASC, range_end >>> ASC) >>> AND bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'per-table primary range size estimates' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.built_views ( >>> keyspace_name text, >>> view_name text, >>> PRIMARY KEY (keyspace_name, view_name) >>> ) WITH CLUSTERING ORDER BY (view_name ASC) >>> AND bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'built views' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.range_xfers ( >>> token_bytes blob PRIMARY KEY, >>> requested_at timestamp >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'ranges requested for transfer' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.local ( >>> key text PRIMARY KEY, >>> bootstrapped text, >>> broadcast_address inet, >>> cluster_name text, >>> cql_version text, >>> data_center text, >>> gossip_generation int, >>> host_id uuid, >>> listen_address inet, >>> native_protocol_version text, >>> partitioner text, >>> rack text, >>> release_version text, >>> rpc_address inet, >>> schema_version uuid, >>> thrift_version text, >>> tokens set<text>, >>> truncated_at map<uuid, blob> >>> ) WITH bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = 'information about the local node' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>> 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> CREATE TABLE system.hints ( >>> target_id uuid, >>> hint_id timeuuid, >>> message_version int, >>> mutation blob, >>> PRIMARY KEY (target_id, hint_id, message_version) >>> ) WITH COMPACT STORAGE >>> AND CLUSTERING ORDER BY (hint_id ASC, message_version ASC) >>> AND bloom_filter_fp_chance = 0.01 >>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>> AND comment = '*DEPRECATED* hints awaiting delivery' >>> AND compaction = {'class': 'org.apache.cassandra.db.compa >>> ction.SizeTieredCompactionStrategy', 'enabled': 'false', >>> 'max_threshold': '32', 'min_threshold': '4'} >>> AND compression = {'chunk_length_in_kb': '64', 'class': ' >>> org.apache.cassandra.io.compress.LZ4Compressor'} >>> AND crc_check_chance = 1.0 >>> AND dclocal_read_repair_chance = 0.0 >>> AND default_time_to_live = 0 >>> AND gc_grace_seconds = 0 >>> AND max_index_interval = 2048 >>> AND memtable_flush_period_in_ms = 3600000 >>> AND min_index_interval = 128 >>> AND read_repair_chance = 0.0 >>> AND speculative_retry = '99PERCENTILE'; >>> >>> >>> The boot logs show: >>> >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_keyspaces >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_columnfamilies >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_columns >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_triggers >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_usertypes >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_functions >>> Apr 17 19:58:22 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system.schema_aggregates >>> >>> >>> but also: >>> >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.keyspaces >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.tables >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.columns >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.triggers >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.dropped_columns >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.views >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.types >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.functions >>> Apr 17 19:58:23 ip-10-48-93-149.us-west-2.compute.internal >>> docker[1469]: cassandra: INFO Initializing system_schema.aggregates >>> >>> >>> basically the "same" cf in the system_schema namespace. >>> >>> I believe that the at the beginning the cfs where part of the system >>> keyspace and then moved to the system_schema keyspace, am I right? >>> >>> If yes, is this a bug to open in order to avoid to create those cf in >>> the system data folder? >>> I already searched in Jira without any luck. >>> >>> Thank you in advance. >>> Regards >>> L. >>> >> >> >