Did you solved this issue ? I guess nobody answers you because this is very weird. I also guess you've made some mistake on the configuration.
Anyway, let me know if you managed to get out of the mess somehow or if you still need help. C*heers 2014-12-03 15:57 GMT+01:00 Castelain, Alain <alain.castel...@xerox.com>: > > > Hi, > > I had a three node cluster in cassandra 1.2 16 version running well > So I have changed the range_request_timeout_in_ms form 10000 to 20000 on > two nodes and this nodes restarted well. > On the last node I received this messages from the output.log file : > > INFO 19:27:18,691 Cassandra shutting down... > INFO 19:27:18,706 Stop listening to thrift clients > INFO 19:27:18,905 Stop listening for CQL clients > INFO 19:27:25,153 Announcing shutdown > INFO 19:27:25,244 GC for ConcurrentMarkSweep: 6227 ms for 3 collections, > 1200395040 used; max is 21045379072 > INFO 19:27:25,244 Pool Name Active Pending > Completed Blocked All Time Blocked > INFO 19:27:25,245 ReadStage 0 0 > 7960169 0 0 > INFO 19:27:25,245 RequestResponseStage 0 0 > 29973176 0 0 > INFO 19:27:25,245 ReadRepairStage 0 0 > 4741703 0 0 > INFO 19:27:25,245 MutationStage 0 0 > 26456216 0 0 > INFO 19:27:25,245 ReplicateOnWriteStage 0 0 > 13 0 0 > INFO 19:27:25,245 GossipStage 0 0 > 1656431 0 0 > INFO 19:27:25,245 AntiEntropyStage 0 0 > 84461 0 0 > INFO 19:27:25,246 MigrationStage 0 0 > 2534350 0 0 > INFO 19:27:25,246 MemtablePostFlusher 0 0 > 40644 0 0 > INFO 19:27:25,246 MemoryMeter 0 0 > 145 0 0 > INFO 19:27:25,246 FlushWriter 0 0 > 2869 0 20 > INFO 19:27:25,246 MiscStage 0 0 > 1651 0 0 > INFO 19:27:25,246 PendingRangeCalculator 0 0 > 6 0 0 > INFO 19:27:25,246 commitlog_archiver 0 0 > 0 0 0 > INFO 19:27:25,246 InternalResponseStage 0 0 > 4633968 0 0 > INFO 19:27:25,247 AntiEntropySessions 0 0 > 15360 0 0 > INFO 19:27:25,247 HintedHandoff 0 0 > 200 0 0 > INFO 19:27:25,247 CompactionManager 0 0 > INFO 19:27:25,247 Commitlog n/a 0 > INFO 19:27:25,247 MessagingService n/a 0/0 > INFO 19:27:25,247 Cache Type Size > Capacity KeysToSave > Provider > INFO 19:27:25,247 KeyCache 104857412 > 104857600 all > INFO 19:27:25,247 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 19:27:25,247 ColumnFamily Memtable ops,data > INFO 19:27:25,248 Test_Sib.Files 0,0 > INFO 19:27:25,248 idfx.report_file_counters 0,0 > INFO 19:27:25,248 idfx.report_file 0,0 > INFO 19:27:25,248 SibBackup2.Files 0,0 > INFO 19:27:25,248 SibTss2.Files 0,0 > INFO 19:27:25,248 SibLeroyMerlin.Files 6316,87364119 > INFO 19:27:25,248 SibCasto.Files 0,0 > INFO 19:27:25,248 idfxr01.system_properties 0,0 > INFO 19:27:25,248 idfxr01.edgestore 0,0 > INFO 19:27:25,248 idfxr01.edgeindex 0,0 > INFO 19:27:25,248 idfxr01.titan_ids 0,0 > INFO 19:27:25,248 idfxr01.edgestore_lock_ 0,0 > INFO 19:27:25,248 idfxr01.vertexindex 0,0 > INFO 19:27:25,248 idfxr01.vertexindex_lock_ 0,0 > INFO 19:27:25,248 SibBackupTest.Files 0,0 > INFO 19:27:25,248 SibNorauto.SibCassandraColumnFamilyName > 1444,753849990 > INFO 19:27:25,248 SibNorauto.Files 0,0 > INFO 19:27:25,248 SibTss.Files 0,0 > INFO 19:27:25,248 SibTss1.Files 0,0 > INFO 19:27:25,249 OpsCenter.bestpractice_results 0,0 > INFO 19:27:25,249 OpsCenter.events 12,3520 > INFO 19:27:25,249 OpsCenter.rollups60 406517,77862510 > INFO 19:27:25,249 OpsCenter.settings 216,38195 > INFO 19:27:25,249 OpsCenter.pdps 168056,12582912 > INFO 19:27:25,249 OpsCenter.rollups86400 0,0 > INFO 19:27:25,249 OpsCenter.events_timeline 2,320 > INFO 19:27:25,249 OpsCenter.rollups300 81242,44035928 > INFO 19:27:25,249 OpsCenter.rollups7200 4365,1488742 > INFO 19:27:25,249 SibLamy.Files 24750,800384035 > INFO 19:27:25,249 SibLamy.SibCassandraColumnFamilyName 0,0 > INFO 19:27:25,249 system.local 2,520 > INFO 19:27:25,249 system.peers 16,1860 > INFO 19:27:25,249 system.batchlog 0,0 > INFO 19:27:25,249 system.NodeIdInfo 0,0 > INFO 19:27:25,249 system.LocationInfo 0,0 > INFO 19:27:25,249 system.Schema 0,0 > INFO 19:27:25,249 system.Migrations 0,0 > INFO 19:27:25,249 system.schema_keyspaces 0,0 > INFO 19:27:25,250 system.schema_columns 0,0 > INFO 19:27:25,250 system.schema_columnfamilies 0,0 > INFO 19:27:25,250 system.IndexInfo 0,0 > INFO 19:27:25,250 system.range_xfers 0,0 > INFO 19:27:25,250 system.peer_events 0,0 > INFO 19:27:25,250 system.hints 299,2880396 > INFO 19:27:25,250 system.HintsColumnFamily 0,0 > INFO 19:27:25,250 SibBackup.Files 59564,53786449 > INFO 19:27:25,250 SibBytel.Files 0,0 > INFO 19:27:25,250 SibColas.Files 0,0 > INFO 19:27:25,250 SibCassandraPierre.Files 0,0 > INFO 19:27:25,250 SibVeolia.Files 0,0 > INFO 19:27:25,250 idfxp01.vertexindex 0,0 > INFO 19:27:25,250 idfxp01.vertexindex_lock_ 0,0 > INFO 19:27:25,250 idfxp01.edgestore 0,0 > INFO 19:27:25,250 idfxp01.edgeindex 0,0 > INFO 19:27:25,250 idfxp01.system_properties 0,0 > INFO 19:27:25,250 idfxp01.edgestore_lock_ 0,0 > INFO 19:27:25,250 idfxp01.titan_ids 0,0 > INFO 19:27:25,251 system_traces.sessions 0,0 > INFO 19:27:25,251 system_traces.events 0,0 > INFO 19:27:25,251 SibCassandraTestPierre.Files 0,0 > INFO 19:27:27,153 Waiting for messaging service to quiesce > INFO 19:27:27,159 MessagingService shutting down server thread. > > After the service cassandra start command : > > INFO 19:27:33,739 Logging initialized > INFO 19:27:33,768 JVM vendor/version: Java HotSpot(TM) 64-Bit Server > VM/1.7.0_55 > INFO 19:27:33,770 Heap size: 21045379072/21045379072 > INFO 19:27:33,770 Classpath: > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.16.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.16.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO 19:27:41,717 JNA mlockall successful > INFO 19:27:41,739 Loading settings from file:/etc/cassandra/cassandra.yaml > INFO 19:27:42,052 Data files directories: [/data/cassandra/data] > INFO 19:27:42,052 Commit log directory: /commitlog > INFO 19:27:42,052 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 19:27:42,052 disk_failure_policy is stop > INFO 19:27:42,058 Global memtable threshold is enabled at 6690MB > INFO 19:27:42,263 Not using multi-threaded compaction > INFO 19:27:42,799 Initializing key cache with capacity of 100 MBs. > INFO 19:27:42,818 Scheduling key cache save to each 14400 seconds (going > to save all keys). > INFO 19:27:42,820 Initializing row cache with capacity of 0 MBs and > provider org.apache.cassandra.cache.SerializingCacheProvider > INFO 19:27:42,820 Initializing row cache with capacity of 0 MBs and > provider org.apache.cassandra.cache.SerializingCacheProvider > INFO 19:27:42,831 Scheduling row cache save to each 0 seconds (going to > save all keys). > INFO 19:27:43,269 Initializing system.batchlog > INFO 19:27:43,318 Initializing system.peer_events > INFO 19:27:43,327 Initializing system.hints > INFO 19:27:43,336 Opening > /data/cassandra/data/system/hints/system-hints-ic-22 (7813966 bytes) > INFO 19:27:43,337 Opening > /data/cassandra/data/system/hints/system-hints-ic-22 (7813966 bytes) > INFO 19:27:43,436 Initializing system.Schema > INFO 19:27:43,445 Initializing system.schema_keyspaces > INFO 19:27:43,449 Opening > /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-208 > (623 bytes) > INFO 19:27:43,449 Opening > /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-207 > (703 bytes) > INFO 19:27:43,450 Opening > /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-208 > (623 bytes) > INFO 19:27:43,450 Opening > /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-207 > (703 bytes) > INFO 19:27:43,556 reading saved cache > /data/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db > INFO 19:27:43,568 Initializing system.range_xfers > INFO 19:27:43,574 Initializing system.HintsColumnFamily > INFO 19:27:43,583 Initializing system.schema_columnfamilies > INFO 19:27:43,587 Opening > /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-265 > (15533 bytes) > INFO 19:27:43,587 Opening > /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-266 > (10706 bytes) > INFO 19:27:43,587 Opening > /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-265 > (15533 bytes) > INFO 19:27:43,587 Opening > /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-266 > (10706 bytes) > INFO 19:27:43,640 reading saved cache > /data/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db > INFO 19:27:43,650 Initializing system.NodeIdInfo > INFO 19:27:43,652 Opening > /data/cassandra/data/system/NodeIdInfo/system-NodeIdInfo-ic-1 (64 bytes) > INFO 19:27:43,652 Opening > /data/cassandra/data/system/NodeIdInfo/system-NodeIdInfo-ic-1 (64 bytes) > INFO 19:27:43,680 Initializing system.schema_columns > INFO 19:27:43,682 Opening > /data/cassandra/data/system/schema_columns/system-schema_columns-ic-113 > (3266 bytes) > INFO 19:27:43,683 Opening > /data/cassandra/data/system/schema_columns/system-schema_columns-ic-113 > (3266 bytes) > INFO 19:27:43,689 reading saved cache > /data/cassandra/saved_caches/system-schema_columns-KeyCache-b.db > INFO 19:27:43,704 Initializing system.IndexInfo > INFO 19:27:43,708 Opening > /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-6 (156 bytes) > INFO 19:27:43,709 Opening > /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-7 (72 bytes) > INFO 19:27:43,709 Opening > /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-5 (71 bytes) > INFO 19:27:43,709 Opening > /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-6 (156 bytes) > INFO 19:27:43,710 Opening > /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-5 (71 bytes) > INFO 19:27:43,709 Opening > /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-7 (72 bytes) > INFO 19:27:43,740 Initializing system.Migrations > INFO 19:27:43,748 Initializing system.peers > INFO 19:27:43,751 Opening > /data/cassandra/data/system/peers/system-peers-ic-131 (213 bytes) > INFO 19:27:43,752 Opening > /data/cassandra/data/system/peers/system-peers-ic-130 (19380 bytes) > INFO 19:27:43,752 Opening > /data/cassandra/data/system/peers/system-peers-ic-131 (213 bytes) > INFO 19:27:43,752 Opening > /data/cassandra/data/system/peers/system-peers-ic-130 (19380 bytes) > INFO 19:27:43,846 Initializing system.local > INFO 19:27:43,849 Opening > /data/cassandra/data/system/local/system-local-ic-265 (84 bytes) > INFO 19:27:43,849 Opening > /data/cassandra/data/system/local/system-local-ic-264 (10379 bytes) > INFO 19:27:43,850 Opening > /data/cassandra/data/system/local/system-local-ic-265 (84 bytes) > INFO 19:27:43,850 Opening > /data/cassandra/data/system/local/system-local-ic-264 (10379 bytes) > INFO 19:27:43,869 reading saved cache > /data/cassandra/saved_caches/system-local-KeyCache-b.db > INFO 19:27:43,895 Initializing system.LocationInfo > java.lang.AssertionError > at > org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:162) > at > org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1541) > at > org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1456) > at > org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:306) > at > org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:287) > at > org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:154) > at > org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:588) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:253) > at > org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212) > Cannot load daemon > Service exit with a return value of 3 > > After I tried to clean commit log and saved_cahes directories and restart > with no effect > > I have cleaned the all data directory and restarted the node with > auto_bootstrap and the node retrieved only two keyspaces . > A this point the previous error comes back when I try to restart without > auto_bootstrap parameter. > > If you could help me I don't have any idea for this node. > > The other nodes are running properly. > > Many thank's for your help. > > > > Cordialement, Regards, > > *Alain Castelain* > *Database Administrator* > Xerox - Global Document Outsourcing > 253, avenue du Président Wilson > La Plaine Saint Denis Cedex, 93211 > France > http://www.xerox.fr/services/frfr.html > > p (+33) 155 847 333 > m (+33) 682 999 617 > @ alain.castel...@xerox.com > > > >
