Hi Ajay, Please take a look at the cassandra.yaml configuration reference regarding intial_token and num_tokens:
http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__initial_token This is basically what Michael was referring to in his earlier message. Setting an initial token overrode your num_tokens setting on initial startup, but after initial startup, the initial token setting is ignored, so num_tokens comes into play, attempting to start up with 256 vnodes. That's where your error comes from. It's likely that all of your nodes started up like this since you have the same config on all of them (hopefully, you at least changed initial_token for each node). After reviewing the doc on the two sections above, you'll need to decide which path to take to recover. You can likely bring the downed node up by setting num_tokens to 1 (which you'd need to do on all nodes), in which case you're not really running vnodes. Alternately, you can migrate the cluster to vnodes: http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configVnodesProduction_t.html BTW, I recommend carefully reviewing the cassandra.yaml configuration reference for ANY change you make from the default. As you've experienced here, not all settings are intended to work together. HTH, Steve On Fri, Oct 23, 2015 at 12:07 PM, Ajay Garg <ajaygargn...@gmail.com> wrote: > Any ideas, please? > To repeat, we are using the exact same cassandra-version on all 4 nodes > (2.1.10). > > On Fri, Oct 23, 2015 at 9:43 AM, Ajay Garg <ajaygargn...@gmail.com> wrote: > >> Hi Michael. >> >> Please find below the contents of cassandra.yaml for CAS11 (the files on >> the rest of the three nodes are also exactly the same, except the >> "initial_token" and "listen_address" fields) :: >> >> CAS11 :: >> >> #################################### >> cluster_name: 'InstaMsg Cluster' >> num_tokens: 256 >> initial_token: -9223372036854775808 >> hinted_handoff_enabled: true >> max_hint_window_in_ms: 10800000 # 3 hours >> hinted_handoff_throttle_in_kb: 1024 >> max_hints_delivery_threads: 2 >> batchlog_replay_throttle_in_kb: 1024 >> authenticator: AllowAllAuthenticator >> authorizer: AllowAllAuthorizer >> permissions_validity_in_ms: 2000 >> partitioner: org.apache.cassandra.dht.Murmur3Partitioner >> data_file_directories: >> - /var/lib/cassandra/data >> >> commitlog_directory: /var/lib/cassandra/commitlog >> >> disk_failure_policy: stop >> commit_failure_policy: stop >> key_cache_size_in_mb: >> key_cache_save_period: 14400 >> row_cache_size_in_mb: 0 >> row_cache_save_period: 0 >> counter_cache_size_in_mb: >> counter_cache_save_period: 7200 >> saved_caches_directory: /var/lib/cassandra/saved_caches >> commitlog_sync: periodic >> commitlog_sync_period_in_ms: 10000 >> commitlog_segment_size_in_mb: 32 >> seed_provider: >> - class_name: org.apache.cassandra.locator.SimpleSeedProvider >> parameters: >> - seeds: "104.239.200.33,119.9.92.77" >> >> concurrent_reads: 32 >> concurrent_writes: 32 >> concurrent_counter_writes: 32 >> >> memtable_allocation_type: heap_buffers >> >> index_summary_capacity_in_mb: >> index_summary_resize_interval_in_minutes: 60 >> trickle_fsync: false >> trickle_fsync_interval_in_kb: 10240 >> storage_port: 7000 >> ssl_storage_port: 7001 >> listen_address: 104.239.200.33 >> start_native_transport: true >> native_transport_port: 9042 >> start_rpc: true >> rpc_address: localhost >> rpc_port: 9160 >> rpc_keepalive: true >> >> rpc_server_type: sync >> thrift_framed_transport_size_in_mb: 15 >> incremental_backups: false >> snapshot_before_compaction: false >> auto_snapshot: true >> >> tombstone_warn_threshold: 1000 >> tombstone_failure_threshold: 100000 >> >> column_index_size_in_kb: 64 >> batch_size_warn_threshold_in_kb: 5 >> >> compaction_throughput_mb_per_sec: 16 >> compaction_large_partition_warning_threshold_mb: 100 >> >> sstable_preemptive_open_interval_in_mb: 50 >> >> read_request_timeout_in_ms: 5000 >> range_request_timeout_in_ms: 10000 >> >> write_request_timeout_in_ms: 2000 >> counter_write_request_timeout_in_ms: 5000 >> cas_contention_timeout_in_ms: 1000 >> truncate_request_timeout_in_ms: 60000 >> request_timeout_in_ms: 10000 >> cross_node_timeout: false >> endpoint_snitch: PropertyFileSnitch >> >> dynamic_snitch_update_interval_in_ms: 100 >> dynamic_snitch_reset_interval_in_ms: 600000 >> dynamic_snitch_badness_threshold: 0.1 >> >> request_scheduler: org.apache.cassandra.scheduler.NoScheduler >> >> server_encryption_options: >> internode_encryption: none >> keystore: conf/.keystore >> keystore_password: cassandra >> truststore: conf/.truststore >> truststore_password: cassandra >> >> client_encryption_options: >> enabled: false >> keystore: conf/.keystore >> keystore_password: cassandra >> >> internode_compression: all >> inter_dc_tcp_nodelay: false >> #################################### >> >> >> What changes need to be made, so that whenever a downed server comes back >> up, the missing data comes back over to it? >> >> Thanks and Regards, >> Ajay >> >> >> >> On Fri, Oct 23, 2015 at 9:05 AM, Michael Shuler <mich...@pbandjelly.org> >> wrote: >> >>> On 10/22/2015 10:14 PM, Ajay Garg wrote: >>> >>>> However, CAS11 refuses to come up now. >>>> Following is the error in /var/log/cassandra/system.log :: >>>> >>>> >>>> ################################################################ >>>> ERROR [main] 2015-10-23 03:07:34,242 CassandraDaemon.java:391 - Fatal >>>> configuration error >>>> org.apache.cassandra.exceptions.ConfigurationException: Cannot change >>>> the number of tokens from 1 to 256 >>>> >>> >>> Check your cassandra.yaml - this node has vnodes enabled in the >>> configuration when it did not, previously. Check all nodes. Something >>> changed. Mixed vnode/non-vnode clusters is bad juju. >>> >>> -- >>> Kind regards, >>> Michael >>> >> >> >> >> -- >> Regards, >> Ajay >> > > > > -- > Regards, > Ajay > -- Steve Robenalt Software Architect sroben...@highwire.org <bza...@highwire.org> (office/cell): 916-505-1785 HighWire Press, Inc. 425 Broadway St, Redwood City, CA 94063 www.highwire.org Technology for Scholarly Communication
