Hi,
I had a three node cluster in cassandra 1.2 16 version running well
So I have changed the range_request_timeout_in_ms form 10000 to 20000 on two
nodes and this nodes restarted well.
On the last node I received this messages from the output.log file :
INFO 19:27:18,691 Cassandra shutting down...
INFO 19:27:18,706 Stop listening to thrift clients
INFO 19:27:18,905 Stop listening for CQL clients
INFO 19:27:25,153 Announcing shutdown
INFO 19:27:25,244 GC for ConcurrentMarkSweep: 6227 ms for 3 collections,
1200395040 used; max is 21045379072
INFO 19:27:25,244 Pool Name Active Pending Completed
Blocked All Time Blocked
INFO 19:27:25,245 ReadStage 0 0 7960169
0 0
INFO 19:27:25,245 RequestResponseStage 0 0 29973176
0 0
INFO 19:27:25,245 ReadRepairStage 0 0 4741703
0 0
INFO 19:27:25,245 MutationStage 0 0 26456216
0 0
INFO 19:27:25,245 ReplicateOnWriteStage 0 0 13
0 0
INFO 19:27:25,245 GossipStage 0 0 1656431
0 0
INFO 19:27:25,245 AntiEntropyStage 0 0 84461
0 0
INFO 19:27:25,246 MigrationStage 0 0 2534350
0 0
INFO 19:27:25,246 MemtablePostFlusher 0 0 40644
0 0
INFO 19:27:25,246 MemoryMeter 0 0 145
0 0
INFO 19:27:25,246 FlushWriter 0 0 2869
0 20
INFO 19:27:25,246 MiscStage 0 0 1651
0 0
INFO 19:27:25,246 PendingRangeCalculator 0 0 6
0 0
INFO 19:27:25,246 commitlog_archiver 0 0 0
0 0
INFO 19:27:25,246 InternalResponseStage 0 0 4633968
0 0
INFO 19:27:25,247 AntiEntropySessions 0 0 15360
0 0
INFO 19:27:25,247 HintedHandoff 0 0 200
0 0
INFO 19:27:25,247 CompactionManager 0 0
INFO 19:27:25,247 Commitlog n/a 0
INFO 19:27:25,247 MessagingService n/a 0/0
INFO 19:27:25,247 Cache Type Size Capacity
KeysToSave
Provider
INFO 19:27:25,247 KeyCache 104857412 104857600
all
INFO 19:27:25,247 RowCache 0 0
all
org.apache.cassandra.cache.SerializingCacheProvider
INFO 19:27:25,247 ColumnFamily Memtable ops,data
INFO 19:27:25,248 Test_Sib.Files 0,0
INFO 19:27:25,248 idfx.report_file_counters 0,0
INFO 19:27:25,248 idfx.report_file 0,0
INFO 19:27:25,248 SibBackup2.Files 0,0
INFO 19:27:25,248 SibTss2.Files 0,0
INFO 19:27:25,248 SibLeroyMerlin.Files 6316,87364119
INFO 19:27:25,248 SibCasto.Files 0,0
INFO 19:27:25,248 idfxr01.system_properties 0,0
INFO 19:27:25,248 idfxr01.edgestore 0,0
INFO 19:27:25,248 idfxr01.edgeindex 0,0
INFO 19:27:25,248 idfxr01.titan_ids 0,0
INFO 19:27:25,248 idfxr01.edgestore_lock_ 0,0
INFO 19:27:25,248 idfxr01.vertexindex 0,0
INFO 19:27:25,248 idfxr01.vertexindex_lock_ 0,0
INFO 19:27:25,248 SibBackupTest.Files 0,0
INFO 19:27:25,248 SibNorauto.SibCassandraColumnFamilyName 1444,753849990
INFO 19:27:25,248 SibNorauto.Files 0,0
INFO 19:27:25,248 SibTss.Files 0,0
INFO 19:27:25,248 SibTss1.Files 0,0
INFO 19:27:25,249 OpsCenter.bestpractice_results 0,0
INFO 19:27:25,249 OpsCenter.events 12,3520
INFO 19:27:25,249 OpsCenter.rollups60 406517,77862510
INFO 19:27:25,249 OpsCenter.settings 216,38195
INFO 19:27:25,249 OpsCenter.pdps 168056,12582912
INFO 19:27:25,249 OpsCenter.rollups86400 0,0
INFO 19:27:25,249 OpsCenter.events_timeline 2,320
INFO 19:27:25,249 OpsCenter.rollups300 81242,44035928
INFO 19:27:25,249 OpsCenter.rollups7200 4365,1488742
INFO 19:27:25,249 SibLamy.Files 24750,800384035
INFO 19:27:25,249 SibLamy.SibCassandraColumnFamilyName 0,0
INFO 19:27:25,249 system.local 2,520
INFO 19:27:25,249 system.peers 16,1860
INFO 19:27:25,249 system.batchlog 0,0
INFO 19:27:25,249 system.NodeIdInfo 0,0
INFO 19:27:25,249 system.LocationInfo 0,0
INFO 19:27:25,249 system.Schema 0,0
INFO 19:27:25,249 system.Migrations 0,0
INFO 19:27:25,249 system.schema_keyspaces 0,0
INFO 19:27:25,250 system.schema_columns 0,0
INFO 19:27:25,250 system.schema_columnfamilies 0,0
INFO 19:27:25,250 system.IndexInfo 0,0
INFO 19:27:25,250 system.range_xfers 0,0
INFO 19:27:25,250 system.peer_events 0,0
INFO 19:27:25,250 system.hints 299,2880396
INFO 19:27:25,250 system.HintsColumnFamily 0,0
INFO 19:27:25,250 SibBackup.Files 59564,53786449
INFO 19:27:25,250 SibBytel.Files 0,0
INFO 19:27:25,250 SibColas.Files 0,0
INFO 19:27:25,250 SibCassandraPierre.Files 0,0
INFO 19:27:25,250 SibVeolia.Files 0,0
INFO 19:27:25,250 idfxp01.vertexindex 0,0
INFO 19:27:25,250 idfxp01.vertexindex_lock_ 0,0
INFO 19:27:25,250 idfxp01.edgestore 0,0
INFO 19:27:25,250 idfxp01.edgeindex 0,0
INFO 19:27:25,250 idfxp01.system_properties 0,0
INFO 19:27:25,250 idfxp01.edgestore_lock_ 0,0
INFO 19:27:25,250 idfxp01.titan_ids 0,0
INFO 19:27:25,251 system_traces.sessions 0,0
INFO 19:27:25,251 system_traces.events 0,0
INFO 19:27:25,251 SibCassandraTestPierre.Files 0,0
INFO 19:27:27,153 Waiting for messaging service to quiesce
INFO 19:27:27,159 MessagingService shutting down server thread.
After the service cassandra start command :
INFO 19:27:33,739 Logging initialized
INFO 19:27:33,768 JVM vendor/version: Java HotSpot(TM) 64-Bit Server
VM/1.7.0_55
INFO 19:27:33,770 Heap size: 21045379072/21045379072
INFO 19:27:33,770 Classpath:
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.16.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.16.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO 19:27:41,717 JNA mlockall successful
INFO 19:27:41,739 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO 19:27:42,052 Data files directories: [/data/cassandra/data]
INFO 19:27:42,052 Commit log directory: /commitlog
INFO 19:27:42,052 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 19:27:42,052 disk_failure_policy is stop
INFO 19:27:42,058 Global memtable threshold is enabled at 6690MB
INFO 19:27:42,263 Not using multi-threaded compaction
INFO 19:27:42,799 Initializing key cache with capacity of 100 MBs.
INFO 19:27:42,818 Scheduling key cache save to each 14400 seconds (going to
save all keys).
INFO 19:27:42,820 Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO 19:27:42,820 Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO 19:27:42,831 Scheduling row cache save to each 0 seconds (going to save
all keys).
INFO 19:27:43,269 Initializing system.batchlog
INFO 19:27:43,318 Initializing system.peer_events
INFO 19:27:43,327 Initializing system.hints
INFO 19:27:43,336 Opening /data/cassandra/data/system/hints/system-hints-ic-22
(7813966 bytes)
INFO 19:27:43,337 Opening /data/cassandra/data/system/hints/system-hints-ic-22
(7813966 bytes)
INFO 19:27:43,436 Initializing system.Schema
INFO 19:27:43,445 Initializing system.schema_keyspaces
INFO 19:27:43,449 Opening
/data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-208
(623 bytes)
INFO 19:27:43,449 Opening
/data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-207
(703 bytes)
INFO 19:27:43,450 Opening
/data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-208
(623 bytes)
INFO 19:27:43,450 Opening
/data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-207
(703 bytes)
INFO 19:27:43,556 reading saved cache
/data/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
INFO 19:27:43,568 Initializing system.range_xfers
INFO 19:27:43,574 Initializing system.HintsColumnFamily
INFO 19:27:43,583 Initializing system.schema_columnfamilies
INFO 19:27:43,587 Opening
/data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-265
(15533 bytes)
INFO 19:27:43,587 Opening
/data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-266
(10706 bytes)
INFO 19:27:43,587 Opening
/data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-265
(15533 bytes)
INFO 19:27:43,587 Opening
/data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-266
(10706 bytes)
INFO 19:27:43,640 reading saved cache
/data/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
INFO 19:27:43,650 Initializing system.NodeIdInfo
INFO 19:27:43,652 Opening
/data/cassandra/data/system/NodeIdInfo/system-NodeIdInfo-ic-1 (64 bytes)
INFO 19:27:43,652 Opening
/data/cassandra/data/system/NodeIdInfo/system-NodeIdInfo-ic-1 (64 bytes)
INFO 19:27:43,680 Initializing system.schema_columns
INFO 19:27:43,682 Opening
/data/cassandra/data/system/schema_columns/system-schema_columns-ic-113 (3266
bytes)
INFO 19:27:43,683 Opening
/data/cassandra/data/system/schema_columns/system-schema_columns-ic-113 (3266
bytes)
INFO 19:27:43,689 reading saved cache
/data/cassandra/saved_caches/system-schema_columns-KeyCache-b.db
INFO 19:27:43,704 Initializing system.IndexInfo
INFO 19:27:43,708 Opening
/data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-6 (156 bytes)
INFO 19:27:43,709 Opening
/data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-7 (72 bytes)
INFO 19:27:43,709 Opening
/data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-5 (71 bytes)
INFO 19:27:43,709 Opening
/data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-6 (156 bytes)
INFO 19:27:43,710 Opening
/data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-5 (71 bytes)
INFO 19:27:43,709 Opening
/data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-7 (72 bytes)
INFO 19:27:43,740 Initializing system.Migrations
INFO 19:27:43,748 Initializing system.peers
INFO 19:27:43,751 Opening
/data/cassandra/data/system/peers/system-peers-ic-131 (213 bytes)
INFO 19:27:43,752 Opening
/data/cassandra/data/system/peers/system-peers-ic-130 (19380 bytes)
INFO 19:27:43,752 Opening
/data/cassandra/data/system/peers/system-peers-ic-131 (213 bytes)
INFO 19:27:43,752 Opening
/data/cassandra/data/system/peers/system-peers-ic-130 (19380 bytes)
INFO 19:27:43,846 Initializing system.local
INFO 19:27:43,849 Opening
/data/cassandra/data/system/local/system-local-ic-265 (84 bytes)
INFO 19:27:43,849 Opening
/data/cassandra/data/system/local/system-local-ic-264 (10379 bytes)
INFO 19:27:43,850 Opening
/data/cassandra/data/system/local/system-local-ic-265 (84 bytes)
INFO 19:27:43,850 Opening
/data/cassandra/data/system/local/system-local-ic-264 (10379 bytes)
INFO 19:27:43,869 reading saved cache
/data/cassandra/saved_caches/system-local-KeyCache-b.db
INFO 19:27:43,895 Initializing system.LocationInfo
java.lang.AssertionError
at org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:162)
at
org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1541)
at
org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1456)
at
org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:306)
at
org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:287)
at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:154)
at
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:588)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:253)
at
org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
Cannot load daemon
Service exit with a return value of 3
After I tried to clean commit log and saved_cahes directories and restart with
no effect
I have cleaned the all data directory and restarted the node with
auto_bootstrap and the node retrieved only two keyspaces .
A this point the previous error comes back when I try to restart without
auto_bootstrap parameter.
If you could help me I don't have any idea for this node.
The other nodes are running properly.
Many thank's for your help.
Cordialement, Regards,
Alain Castelain
Database Administrator
Xerox - Global Document Outsourcing
253, avenue du Président Wilson
La Plaine Saint Denis Cedex, 93211
France
http://www.xerox.fr/services/frfr.html
p (+33) 155 847 333
m (+33) 682 999 617
@ alain.castel...@xerox.com
smime.p7s
Description: S/MIME cryptographic signature
