It is a Cassandra bug. The workaround is to change system_distributed
keyspce replication strategy to something as below:
alter keyspace system_distributed with replication = {'class':
'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};
You may see similar problem for other system keyspaces. Do the same thing.
Cheers,
Yabin
On Thu, Sep 22, 2016 at 1:44 PM, Timo Ahokas <timo.aho...@gmail.com> wrote:
> Hi Alain,
>
> Our normal user keyspaces have RF3 in all DCs, e.g:
>
> create keyspace reporting with replication = {'class':
> 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};
>
> Any idea would it be safe to change the system_distributed keyspace to
> match this?
>
> -Timo
>
> On 22 September 2016 at 19:23, Timo Ahokas <timo.aho...@gmail.com> wrote:
>
>> Hi Alain,
>>
>> Thanks a lot for a helping out!
>>
>> Some of the basic keyspace / cluster info you requested:
>>
>> # echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
>>
>> CREATE KEYSPACE system_distributed WITH replication = {'class':
>> 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
>>
>> CREATE TABLE system_distributed.repair_history (
>>
>> keyspace_name text,
>>
>> columnfamily_name text,
>>
>> id timeuuid,
>>
>> coordinator inet,
>>
>> exception_message text,
>>
>> exception_stacktrace text,
>>
>> finished_at timestamp,
>>
>> parent_id timeuuid,
>>
>> participants set<inet>,
>>
>> range_begin text,
>>
>> range_end text,
>>
>> started_at timestamp,
>>
>> status text,
>>
>> PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>>
>> ) WITH CLUSTERING ORDER BY (id ASC)
>>
>> AND bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = 'Repair history'
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.0
>>
>> AND default_time_to_live = 0
>>
>> AND gc_grace_seconds = 0
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 3600000
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>> CREATE TABLE system_distributed.parent_repair_history (
>>
>> parent_id timeuuid PRIMARY KEY,
>>
>> columnfamily_names set<text>,
>>
>> exception_message text,
>>
>> exception_stacktrace text,
>>
>> finished_at timestamp,
>>
>> keyspace_name text,
>>
>> requested_ranges set<text>,
>>
>> started_at timestamp,
>>
>> successful_ranges set<text>
>>
>> ) WITH bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = 'Repair history'
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.0
>>
>> AND default_time_to_live = 0
>>
>> AND gc_grace_seconds = 0
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 3600000
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>>
>> CREATE TABLE system_distributed.repair_history (
>>
>> keyspace_name text,
>>
>> columnfamily_name text,
>>
>> id timeuuid,
>>
>> coordinator inet,
>>
>> exception_message text,
>>
>> exception_stacktrace text,
>>
>> finished_at timestamp,
>>
>> parent_id timeuuid,
>>
>> participants set<inet>,
>>
>> range_begin text,
>>
>> range_end text,
>>
>> started_at timestamp,
>>
>> status text,
>>
>> PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>>
>> ) WITH CLUSTERING ORDER BY (id ASC)
>>
>> AND bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = 'Repair history'
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.0
>>
>> AND default_time_to_live = 0
>>
>> AND gc_grace_seconds = 0
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 3600000
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>> CREATE TABLE system_distributed.parent_repair_history (
>>
>> parent_id timeuuid PRIMARY KEY,
>>
>> columnfamily_names set<text>,
>>
>> exception_message text,
>>
>> exception_stacktrace text,
>>
>> finished_at timestamp,
>>
>> keyspace_name text,
>>
>> requested_ranges set<text>,
>>
>> started_at timestamp,
>>
>> successful_ranges set<text>
>>
>> ) WITH bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = 'Repair history'
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.0
>>
>> AND default_time_to_live = 0
>>
>> AND gc_grace_seconds = 0
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 3600000
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>>
>>
>> # nodetool status
>>
>> Datacenter: DC1
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns Host ID
>> Rack
>>
>> UN xxx.xxx.145.5 693,63 GB 256 ?
>> 6f1a0fdd-e3f9-474d-9a49-7bfeeadb3f56 RAC1
>>
>> UN xxx.xxx.145.225 648,55 GB 256 ?
>> f900847a-63e4-44c5-b4d7-e439c7cb6a8e RAC1
>>
>> UN xxx.xxx.145.160 608,31 GB 256 ?
>> d257e76d-9e40-4215-94c7-3076c8ff4b7f RAC1
>>
>> UN xxx.xxx.145.67 552,93 GB 256 ?
>> 1d47cbdd-cdf1-45b6-aa0e-0c6123899dca RAC1
>>
>> UN xxx.xxx.145.227 636,68 GB 256 ?
>> 47e5f207-f9fd-4a86-be8a-66e7630d1baa RAC1
>>
>> UN xxx.xxx.146.105 610,9 GB 256 ?
>> 8edf1aaa-49d1-4e4b-9f09-99c4ab6136c2 RAC1
>>
>> UN xxx.xxx.147.136 666,82 GB 256 ?
>> bafbf6a2-cff9-489f-a2dd-fc6e8cb08ff6 RAC1
>>
>> UN xxx.xxx.146.213 609,79 GB 256 ?
>> 6416275c-7570-48a9-957f-2daca71d31aa RAC1
>>
>> UN xxx.xxx.146.20 664,44 GB 256 ?
>> b016df7e-f694-4ef3-928c-8783853e9a07 RAC1
>>
>> UN xxx.xxx.146.209 615,44 GB 256 ?
>> 898e6d98-1b92-4e86-b52c-f851fd4fda71 RAC1
>>
>> UN xxx.xxx.146.241 668,91 GB 256 ?
>> 0b5d4c6c-4b7c-4265-92bc-ad74464d85cc RAC1
>>
>> UN xxx.xxx.147.211 641,33 GB 256 ?
>> 16cdc4a7-b694-4125-91d6-05b9099cb765 RAC1
>>
>> UN xxx.xxx.147.125 647,03 GB 256 ?
>> 2e97ed0a-039c-413b-9693-a87fadf40f82 RAC1
>>
>> Datacenter: DC2
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns Host ID
>> Rack
>>
>> UN xxx.xxx.7.99 18,76 MB 256 ?
>> d7b907ad-15f5-4c79-962c-c604a5723a7b RAC1
>>
>> UN xxx.xxx.6.135 16,04 MB 256 ?
>> 463f480a-baf3-4230-86b7-1106251ebfad RAC1
>>
>> UN xxx.xxx.7.229 17,36 MB 256 ?
>> 9487a975-6183-43b8-9208-cd8e09a0ae18 RAC1
>>
>> UN xxx.xxx.7.5 14,01 MB 256 ?
>> ae039e49-4d79-4e4e-87bd-921cd6b3291a RAC1
>>
>> UN xxx.xxx.7.4 14,93 MB 256 ?
>> 122a47fb-b5ca-46d1-aae9-e6993ab58b66 RAC1
>>
>> UN xxx.xxx.6.10 16,77 MB 256 ?
>> bbb66068-bf06-438d-81ee-965e201e8fff RAC1
>>
>> UN xxx.xxx.6.15 14,95 MB 256 ?
>> 668a864d-9fd3-41b7-88fb-824e75e71953 RAC1
>>
>> UN xxx.xxx.7.140 17,38 MB 256 ?
>> 7b016c96-eaa1-4ee1-8657-f4260c70ed37 RAC1
>>
>> UN xxx.xxx.7.113 19,14 MB 256 ?
>> 46c06c44-ce2f-4ab6-9597-a1314cecf9bc RAC1
>>
>> UN xxx.xxx.6.118 16,7 MB 256 ?
>> 9c3c3107-a1d3-4254-ad10-909713a38f8c RAC1
>>
>> UN xxx.xxx.6.248 17,29 MB 256 ?
>> 35ff4d3d-d993-468b-9a54-88b40ceec6d4 RAC1
>>
>> UN xxx.xxx.5.24 16,55 MB 256 ?
>> 5f1f34bd-110f-4d60-9af5-a3abd01b55a5 RAC1
>>
>> UN xxx.xxx.7.189 16,63 MB 256 ?
>> be7cbf84-5838-487a-8bd4-b340a1c70fab RAC1
>>
>> UN xxx.xxx.5.124 20,37 MB 256 ?
>> 638f2656-fb92-4b70-ba2a-251a749c4c58 RAC1
>>
>> UN xxx.xxx.6.60 24,57 MB 256 ?
>> cf16209a-a9a0-4f27-9341-c76d47e50261 RAC1
>>
>> Datacenter: DC3
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns Host ID
>> Rack
>>
>> UN xxx.xxx.151.102 389,41 GB 256 ?
>> 1740a473-e304-467c-a682-d1b4b0595ffa RAC1
>>
>> UN xxx.xxx.149.161 367,82 GB 256 ?
>> 3a5322d4-e49f-45ed-85b5-fd658502859c RAC1
>>
>> UN xxx.xxx.149.226 390,88 GB 256 ?
>> b8ca4576-2632-4198-ac87-10243c0c554e RAC1
>>
>> UN xxx.xxx.151.162 408,35 GB 256 ?
>> 54d3dd90-f9ab-47c2-ae31-5f3e87b91e2a RAC1
>>
>> UN xxx.xxx.149.109 369,33 GB 256 ?
>> 9172c7d8-0c55-4e8e-a17b-89fdb0dce878 RAC1
>>
>> UN xxx.xxx.150.172 362,32 GB 256 ?
>> ba394a29-1a0c-4f50-ab85-4db19011b190 RAC1
>>
>> UN xxx.xxx.149.238 388,98 GB 256 ?
>> a3d7228c-ccb4-4787-a4bb-f7720aeedc8e RAC1
>>
>> UN xxx.xxx.151.232 435,31 GB 256 ?
>> 500a43ab-ae77-4a07-876c-171cb34c549b RAC1
>>
>> UN xxx.xxx.151.43 410,69 GB 256 ?
>> b8bc80e2-2107-447a-85e4-57a39dc9c595 RAC1
>>
>> UN xxx.xxx.151.139 407,47 GB 256 ?
>> ecfa4ba7-7783-47a4-8b17-aadc91a3e776 RAC1
>>
>> UN xxx.xxx.151.213 375,05 GB 256 ?
>> 9bf53ee1-53d4-4d18-a58e-0b0a17e18a69 RAC1
>>
>> UN xxx.xxx.149.177 401,91 GB 256 ?
>> b903faf1-1ae9-45ad-bdce-3c9377458a03 RAC1
>>
>> UN xxx.xxx.150.145 388,76 GB 256 ?
>> 1c4e4232-db27-4cc1-9985-9eb7f0b984d1 RAC1
>>
>> UN xxx.xxx.149.48 385,43 GB 256 ?
>> ad3ea388-203c-4b26-a368-934a6105cc6e RAC1
>>
>> UN xxx.xxx.150.189 384,52 GB 256 ?
>> f361ebad-b0a6-47b7-a55c-245c98f84508 RAC1
>>
>> UN xxx.xxx.151.220 357,56 GB 256 ?
>> feb814e6-6d2f-4cef-ae3b-4924c1cbac60 RAC1
>>
>> UN xxx.xxx.149.121 355,64 GB 256 ?
>> 47fbb104-6a5a-49c0-b086-3f14c853c83b RAC1
>>
>> UN xxx.xxx.151.218 416,57 GB 256 ?
>> bbb21d16-da85-4cfd-87d4-2333c8b02dad RAC1
>>
>> UN xxx.xxx.150.26 383,06 GB 256 ?
>> 1ca0085d-93a5-4650-891a-b45f988150a4 RAC1
>>
>> Note: Non-system keyspaces don't have the same replication settings,
>> effective ownership information is meaningless
>>
>>
>>
>> # nodetool status system_distributed
>>
>> Datacenter: DC1
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN xxx.xxx.145.5 693,63 GB 256 6,2%
>> 6f1a0fdd-e3f9-474d-9a49-7bfeeadb3f56 RAC1
>>
>> UN xxx.xxx.145.225 648,55 GB 256 6,8%
>> f900847a-63e4-44c5-b4d7-e439c7cb6a8e RAC1
>>
>> UN xxx.xxx.145.160 608,31 GB 256 6,5%
>> d257e76d-9e40-4215-94c7-3076c8ff4b7f RAC1
>>
>> UN xxx.xxx.145.67 552,93 GB 256 6,1%
>> 1d47cbdd-cdf1-45b6-aa0e-0c6123899dca RAC1
>>
>> UN xxx.xxx.145.227 636,68 GB 256 6,0%
>> 47e5f207-f9fd-4a86-be8a-66e7630d1baa RAC1
>>
>> UN xxx.xxx.146.105 610,9 GB 256 6,1%
>> 8edf1aaa-49d1-4e4b-9f09-99c4ab6136c2 RAC1
>>
>> UN xxx.xxx.147.136 666,82 GB 256 6,3%
>> bafbf6a2-cff9-489f-a2dd-fc6e8cb08ff6 RAC1
>>
>> UN xxx.xxx.146.213 609,79 GB 256 6,0%
>> 6416275c-7570-48a9-957f-2daca71d31aa RAC1
>>
>> UN xxx.xxx.146.20 664,44 GB 256 7,0%
>> b016df7e-f694-4ef3-928c-8783853e9a07 RAC1
>>
>> UN xxx.xxx.146.209 615,44 GB 256 6,6%
>> 898e6d98-1b92-4e86-b52c-f851fd4fda71 RAC1
>>
>> UN xxx.xxx.146.241 668,91 GB 256 6,2%
>> 0b5d4c6c-4b7c-4265-92bc-ad74464d85cc RAC1
>>
>> UN xxx.xxx.147.211 641,33 GB 256 6,5%
>> 16cdc4a7-b694-4125-91d6-05b9099cb765 RAC1
>>
>> UN xxx.xxx.147.125 647,03 GB 256 6,3%
>> 2e97ed0a-039c-413b-9693-a87fadf40f82 RAC1
>>
>> Datacenter: DC2
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN xxx.xxx.7.99 18,76 MB 256 6,3%
>> d7b907ad-15f5-4c79-962c-c604a5723a7b RAC1
>>
>> UN xxx.xxx.6.135 16,04 MB 256 6,1%
>> 463f480a-baf3-4230-86b7-1106251ebfad RAC1
>>
>> UN xxx.xxx.7.229 17,36 MB 256 5,9%
>> 9487a975-6183-43b8-9208-cd8e09a0ae18 RAC1
>>
>> UN xxx.xxx.7.5 14,01 MB 256 6,2%
>> ae039e49-4d79-4e4e-87bd-921cd6b3291a RAC1
>>
>> UN xxx.xxx.7.4 14,93 MB 256 6,4%
>> 122a47fb-b5ca-46d1-aae9-e6993ab58b66 RAC1
>>
>> UN xxx.xxx.6.10 16,77 MB 256 6,4%
>> bbb66068-bf06-438d-81ee-965e201e8fff RAC1
>>
>> UN xxx.xxx.6.15 14,95 MB 256 6,1%
>> 668a864d-9fd3-41b7-88fb-824e75e71953 RAC1
>>
>> UN xxx.xxx.7.140 17,38 MB 256 6,7%
>> 7b016c96-eaa1-4ee1-8657-f4260c70ed37 RAC1
>>
>> UN xxx.xxx.7.113 19,14 MB 256 6,8%
>> 46c06c44-ce2f-4ab6-9597-a1314cecf9bc RAC1
>>
>> UN xxx.xxx.6.118 16,7 MB 256 6,7%
>> 9c3c3107-a1d3-4254-ad10-909713a38f8c RAC1
>>
>> UN xxx.xxx.6.248 17,29 MB 256 6,9%
>> 35ff4d3d-d993-468b-9a54-88b40ceec6d4 RAC1
>>
>> UN xxx.xxx.5.24 16,55 MB 256 6,8%
>> 5f1f34bd-110f-4d60-9af5-a3abd01b55a5 RAC1
>>
>> UN xxx.xxx.7.189 16,63 MB 256 6,2%
>> be7cbf84-5838-487a-8bd4-b340a1c70fab RAC1
>>
>> UN xxx.xxx.5.124 20,37 MB 256 6,3%
>> 638f2656-fb92-4b70-ba2a-251a749c4c58 RAC1
>>
>> UN xxx.xxx.6.60 24,57 MB 256 6,4%
>> cf16209a-a9a0-4f27-9341-c76d47e50261 RAC1
>>
>> Datacenter: DC3
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN xxx.xxx.151.102 389,41 GB 256 6,4%
>> 1740a473-e304-467c-a682-d1b4b0595ffa RAC1
>>
>> UN xxx.xxx.149.161 367,82 GB 256 6,3%
>> 3a5322d4-e49f-45ed-85b5-fd658502859c RAC1
>>
>> UN xxx.xxx.149.226 390,88 GB 256 6,2%
>> b8ca4576-2632-4198-ac87-10243c0c554e RAC1
>>
>> UN xxx.xxx.151.162 408,35 GB 256 6,4%
>> 54d3dd90-f9ab-47c2-ae31-5f3e87b91e2a RAC1
>>
>> UN xxx.xxx.149.109 369,33 GB 256 6,2%
>> 9172c7d8-0c55-4e8e-a17b-89fdb0dce878 RAC1
>>
>> UN xxx.xxx.150.172 362,32 GB 256 6,0%
>> ba394a29-1a0c-4f50-ab85-4db19011b190 RAC1
>>
>> UN xxx.xxx.149.238 388,98 GB 256 6,4%
>> a3d7228c-ccb4-4787-a4bb-f7720aeedc8e RAC1
>>
>> UN xxx.xxx.151.232 435,31 GB 256 6,6%
>> 500a43ab-ae77-4a07-876c-171cb34c549b RAC1
>>
>> UN xxx.xxx.151.43 410,69 GB 256 6,2%
>> b8bc80e2-2107-447a-85e4-57a39dc9c595 RAC1
>>
>> UN xxx.xxx.151.139 407,47 GB 256 6,2%
>> ecfa4ba7-7783-47a4-8b17-aadc91a3e776 RAC1
>>
>> UN xxx.xxx.151.213 375,05 GB 256 6,5%
>> 9bf53ee1-53d4-4d18-a58e-0b0a17e18a69 RAC1
>>
>> UN xxx.xxx.149.177 401,91 GB 256 6,6%
>> b903faf1-1ae9-45ad-bdce-3c9377458a03 RAC1
>>
>> UN xxx.xxx.150.145 388,76 GB 256 7,1%
>> 1c4e4232-db27-4cc1-9985-9eb7f0b984d1 RAC1
>>
>> UN xxx.xxx.149.48 385,43 GB 256 6,2%
>> ad3ea388-203c-4b26-a368-934a6105cc6e RAC1
>>
>> UN xxx.xxx.150.189 384,52 GB 256 6,4%
>> f361ebad-b0a6-47b7-a55c-245c98f84508 RAC1
>>
>> UN xxx.xxx.151.220 357,56 GB 256 6,1%
>> feb814e6-6d2f-4cef-ae3b-4924c1cbac60 RAC1
>>
>> UN xxx.xxx.149.121 355,64 GB 256 6,4%
>> 47fbb104-6a5a-49c0-b086-3f14c853c83b RAC1
>>
>> UN xxx.xxx.151.218 416,57 GB 256 6,3%
>> bbb21d16-da85-4cfd-87d4-2333c8b02dad RAC1
>> UN xxx.xxx.150.26 383,06 GB 256 6,7%
>> 1ca0085d-93a5-4650-891a-b45f988150a4 RAC1
>>
>> DC1 and DC3 are the old data centers. DC2 is the new one being added (as
>> seen from the data loads).
>>
>> For the snitch we are using GossipingPropertyFileSnitch and a
>> cassandra-rackdc.properties with config such as:
>> dc=DC1
>> rack=RAC1
>>
>> Just noticed that we also have cassandra-topology.properties present on
>> the nodes, but it's up-to-date with all the nodes from the 3 data centers.
>>
>> I was wondering on whether the replication settings for the
>> system_distributed keyspace might need a change, but didn't find any yet
>> documentation pointing to that.
>>
>> Best regards,
>> Timo
>>
>> On 22 September 2016 at 18:00, Alain RODRIGUEZ <arodr...@gmail.com>
>> wrote:
>>
>>> It could be a bug.
>>>
>>> Yet I am not very aware of this system_distributed keyspace, but from
>>> what I see, it is using a simple strategy:
>>>
>>> root@tlp-cassandra-2:~# echo "DESCRIBE KEYSPACE system_distributed;" |
>>> cqlsh $(hostname -I | awk '{print $1}')
>>>
>>> CREATE KEYSPACE system_distributed WITH replication = {'class':
>>> 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
>>>
>>> Let's first check some stuff. Could you share the output of:
>>>
>>>
>>> - echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
>>> [ip_address_of_the_server]
>>> - nodetool status
>>> - nodetool status system_distributed
>>> - Let us know about the snitch you are using and the corresponding
>>> configuration.
>>>
>>>
>>> I am trying to make sure the command you used is expected to work, given
>>> your setup.
>>>
>>> My guess is this you might need to alter this keyspace accordingly to
>>> your cluster setup.
>>>
>>> Just guessing, hope that helps.
>>>
>>> C*heers,
>>> -----------------------
>>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>>> France
>>>
>>> The Last Pickle - Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>> 2016-09-22 15:47 GMT+02:00 Timo Ahokas <timo.aho...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> We have a Cassandra 3.0.8 cluster (recently upgraded from 2.1.15)
>>>> currently running in two data centers (13 and 19 nodes, RF3 in both). We
>>>> are adding a third data center before decommissioning one of the earlier
>>>> ones. Installing Cassandra (3.0.8) goes fine and all the nodes join the
>>>> cluster (not set to bootstrap, as documented in
>>>> https://docs.datastax.com/en/cassandra/3.0/cassandra/operati
>>>> ons/opsAddDCToCluster.html).
>>>>
>>>> When trying to rebuild nodes in the new DC from a previous DC (nodetool
>>>> rebuild -- DC1), we get the following error:
>>>>
>>>> Unable to find sufficient sources for streaming range
>>>> (597769692463489739,597931451954862346] in keyspace system_distributed
>>>>
>>>> The same error occurs which ever of the 2 existing DCs we try to
>>>> rebuild from.
>>>>
>>>> We run pr repairs (nodetool repair -pr) on all nodes twice a week via
>>>> cron.
>>>>
>>>> Any advice on how to get the rebuild started?
>>>>
>>>> Best regards,
>>>> Timo
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
