Hi, all
I'm trying to update our old version 0.6.5 to current 1.2.1
All nodes has been drained and stopped. Proper cassandra.yaml created,
schema file prepared.
Trying to start version 1.2.1 on the one node (full output attached to email):
...
ERROR 11:12:44,530 Exception encountered during startup
java.lang.NullPointerException
at
org.apache.cassandra.db.SystemTable.upgradeSystemData(SystemTable.java:161)
at
org.apache.cassandra.db.SystemTable.finishStartup(SystemTable.java:107)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:276)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:370)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:413)
java.lang.NullPointerException
at
org.apache.cassandra.db.SystemTable.upgradeSystemData(SystemTable.java:161)
at
org.apache.cassandra.db.SystemTable.finishStartup(SystemTable.java:107)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:276)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:370)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:413)
Exception encountered during startup: null
On the next attempts daemon started, but still with AssertionErrors
Question 1 - is it possible start the new version from the first attempt?
Then I loaded schema via cassandra-cli, and run nodetool scrub - which
caused a big number of warnings in log:
OutputHandler.java (line 52) Index file contained a different key
or row size; using key from data file
storage-conf.xml from 0.6.5 has column family defined as
<ColumnFamily Name="Invoices" CompareWith="BytesType"/>
for 1.2.1 I used
create column family Invoices with column_type = 'Standard' and
comparator = 'BytesType';
Question 2 - how to get rid of these warnings? Are they connected to
column family definition?
Thanks
--
Sergey
xss = -ea
-javaagent:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms750M -Xmx750M -Xmn187M
-XX:+HeapDumpOnOu
tOfMemoryError -Xss180k
INFO 11:12:42,775 Logging initialized
INFO 11:12:42,795 JVM vendor/version: OpenJDK 64-Bit Server VM/1.7.0_03-icedtea
INFO 11:12:42,796 Heap size: 766836736/766836736
INFO 11:12:42,796 Classpath:
/home/sm/cassandra/apache-cassandra-1.2.1/conf:/home/sm/cassandra/apache-cassandra-1.2.1/build/classes/main:/home/sm/cassandra/apache-cassandra-1.2.1/
build/classes/thrift:/home/sm/cassandra/apache-cassandra-1.2.1/lib/antlr-3.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/apache-cassandra-1.2.1.jar:/home/sm/cassandra/apache-
cassandra-1.2.1/lib/apache-cassandra-clientutil-1.2.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/apache-cassandra-thrift-1.2.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/
lib/avro-1.4.0-fixes.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/avro-1.4.0-sources-fixes.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/commons-cli-1.1.jar:/home/sm/cassa
ndra/apache-cassandra-1.2.1/lib/commons-codec-1.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/commons-lang-2.6.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/compress-lzf-
0.8.4.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/concurrentlinkedhashmap-lru-1.3.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/guava-13.0.1.jar:/home/sm/cassandra/apache
-cassandra-1.2.1/lib/high-scale-lib-1.1.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jackson-core-asl-1.9.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jackson-mapper-
asl-1.9.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jamm-0.2.5.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jline-1.0.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib
/json-simple-1.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/libthrift-0.7.0.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/log4j-1.2.16.jar:/home/sm/cassandra/apache-cass
andra-1.2.1/lib/metrics-core-2.0.3.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/netty-3.5.9.Final.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/servlet-api-2.5-20081211.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/slf4j-api-1.7.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/slf4j-log4j12-1.7.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/snakeyaml-1.6.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/snappy-java-1.0.4.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/snaptree-0.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jamm-0.2.5.jar
INFO 11:12:42,798 JNA not found. Native methods will be disabled.
INFO 11:12:42,814 Loading settings from
file:/home/sm/cassandra/apache-cassandra-1.2.1/conf/cassandra.yaml
INFO 11:12:43,275 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 11:12:43,275 disk_failure_policy is stop
INFO 11:12:43,282 Global memtable threshold is enabled at 243MB
INFO 11:12:43,921 Upgrade from pre-1.1 version detected: migrating sstables to
new directory layout
INFO 11:12:43,945 Initializing key cache with capacity of 36 MBs.
INFO 11:12:43,958 Scheduling key cache save to each 14400 seconds (going to
save all keys).
INFO 11:12:43,959 Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO 11:12:43,967 Scheduling row cache save to each 0 seconds (going to save
all keys).
INFO 11:12:44,151 Opening
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-2 (456 bytes)
INFO 11:12:44,159 Opening
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-1 (345 bytes)
INFO 11:12:44,449 Couldn't detect any schema definitions in local storage.
INFO 11:12:44,450 Found table data in data directories. Consider using the CLI
to define your schema.
INFO 11:12:44,501 No commitlog files found; skipping replay
ERROR 11:12:44,530 Exception encountered during startup
java.lang.NullPointerException
at
org.apache.cassandra.db.SystemTable.upgradeSystemData(SystemTable.java:161)
at
org.apache.cassandra.db.SystemTable.finishStartup(SystemTable.java:107)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:276)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:370)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:413)
java.lang.NullPointerException
at
org.apache.cassandra.db.SystemTable.upgradeSystemData(SystemTable.java:161)
at
org.apache.cassandra.db.SystemTable.finishStartup(SystemTable.java:107)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:276)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:370)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:413)
Exception encountered during startup: null
zsh: exit 3
xss = -ea
-javaagent:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms750M -Xmx750M -Xmn187M
-XX:+HeapDumpOnOu
tOfMemoryError -Xss180k
INFO 11:17:05,794 Logging initialized
INFO 11:17:05,813 JVM vendor/version: OpenJDK 64-Bit Server VM/1.7.0_03-icedtea
INFO 11:17:05,813 Heap size: 766836736/766836736
INFO 11:17:05,813 Classpath:
/home/sm/cassandra/apache-cassandra-1.2.1/conf:/home/sm/cassandra/apache-cassandra-1.2.1/build/classes/main:/home/sm/cassandra/apache-cassandra-1.2.1/
build/classes/thrift:/home/sm/cassandra/apache-cassandra-1.2.1/lib/antlr-3.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/apache-cassandra-1.2.1.jar:/home/sm/cassandra/apache-
cassandra-1.2.1/lib/apache-cassandra-clientutil-1.2.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/apache-cassandra-thrift-1.2.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/
lib/avro-1.4.0-fixes.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/avro-1.4.0-sources-fixes.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/commons-cli-1.1.jar:/home/sm/cassa
ndra/apache-cassandra-1.2.1/lib/commons-codec-1.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/commons-lang-2.6.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/compress-lzf-
0.8.4.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/concurrentlinkedhashmap-lru-1.3.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/guava-13.0.1.jar:/home/sm/cassandra/apache
-cassandra-1.2.1/lib/high-scale-lib-1.1.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jackson-core-asl-1.9.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jackson-mapper-
asl-1.9.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jamm-0.2.5.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/jline-1.0.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib
/json-simple-1.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/libthrift-0.7.0.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/log4j-1.2.16.jar:/home/sm/cassandra/apache-cass
andra-1.2.1/lib/metrics-core-2.0.3.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/netty-3.5.9.Final.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/servlet-api-2.5-20081211.ja
r:/home/sm/cassandra/apache-cassandra-1.2.1/lib/slf4j-api-1.7.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/slf4j-log4j12-1.7.2.jar:/home/sm/cassandra/apache-cassandra-1.2.1/
lib/snakeyaml-1.6.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/snappy-java-1.0.4.1.jar:/home/sm/cassandra/apache-cassandra-1.2.1/lib/snaptree-0.1.jar:/home/sm/cassandra/apache
-cassandra-1.2.1/lib/jamm-0.2.5.jar
INFO 11:17:05,815 JNA not found. Native methods will be disabled.
INFO 11:17:05,830 Loading settings from
file:/home/sm/cassandra/apache-cassandra-1.2.1/conf/cassandra.yaml
INFO 11:17:06,262 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 11:17:06,262 disk_failure_policy is stop
INFO 11:17:06,269 Global memtable threshold is enabled at 243MB
INFO 11:17:06,935 Initializing key cache with capacity of 36 MBs.
INFO 11:17:06,948 Scheduling key cache save to each 14400 seconds (going to
save all keys).
INFO 11:17:06,949 Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO 11:17:06,958 Scheduling row cache save to each 0 seconds (going to save
all keys).
INFO 11:17:07,124 Opening
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-2 (456 bytes)
INFO 11:17:07,132 Opening
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-1 (345 bytes)
ERROR 11:17:07,144 Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError
at
org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:61)
at
org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:1009)
at
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:420)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:357)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:194)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:238)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
ERROR 11:17:07,144 Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:61)
at
org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:1009)
at
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:420)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:357)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:194)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:238)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
INFO 11:17:07,401 Couldn't detect any schema definitions in local storage.
INFO 11:17:07,402 Found table data in data directories. Consider using the CLI
to define your schema.
INFO 11:17:07,460 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1360235564409.log,
/var/lib/cassandra/commitlog/CommitLog-2-1360235564410.log
INFO 11:17:07,474 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1360235564409.log
INFO 11:17:07,517 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1360235564409.log
INFO 11:17:07,518 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1360235564410.log
INFO 11:17:07,518 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1360235564410.log
INFO 11:17:07,522 Enqueuing flush of Memtable-local@338029290(294/294
serialized/live bytes, 15 ops)
INFO 11:17:07,530 Writing Memtable-local@338029290(294/294 serialized/live
bytes, 15 ops)
INFO 11:17:07,569 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ib-1-Data.db (283 bytes) for
commitlog position ReplayPosition(segmentId=1360235827373, position=278)
INFO 11:17:07,597 Log replay complete, 3 replayed mutations
INFO 11:17:07,813 Cassandra version: 1.2.1
INFO 11:17:07,814 Thrift API version: 19.35.0
INFO 11:17:07,814 CQL supported versions: 2.0.0,3.0.1 (default: 3.0.1)
INFO 11:17:07,861 Loading persisted ring state
INFO 11:17:07,865 Starting up server gossip
WARN 11:17:07,884 No host ID found, created
132a3ba6-4618-4c22-8a01-a47106cc3f6a (Note: This should happen exactly once per
node).
INFO 11:17:07,901 Enqueuing flush of Memtable-local@1377800073(284/284
serialized/live bytes, 11 ops)
INFO 11:17:07,901 Writing Memtable-local@1377800073(284/284 serialized/live
bytes, 11 ops)
INFO 11:17:07,909 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ib-2-Data.db (277 bytes) for
commitlog position ReplayPosition(segmentId=1360235827373, position=49268)
INFO 11:17:07,946 Starting Messaging Service on port 7000
INFO 11:17:07,965 Saved token not found. Using [0] from configuration
INFO 11:17:07,967 Enqueuing flush of Memtable-local@988223373(84/84
serialized/live bytes, 4 ops)
INFO 11:17:07,968 Writing Memtable-local@988223373(84/84 serialized/live
bytes, 4 ops)
INFO 11:17:07,989 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ib-3-Data.db (120 bytes) for
commitlog position ReplayPosition(segmentId=1360235827373, position=49543)
INFO 11:17:07,997 Enqueuing flush of Memtable-local@2116134604(32/32
serialized/live bytes, 2 ops)
INFO 11:17:07,997 Writing Memtable-local@2116134604(32/32 serialized/live
bytes, 2 ops)
INFO 11:17:08,008 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ib-4-Data.db (91 bytes) for
commitlog position ReplayPosition(segmentId=1360235827373, position=49699)
INFO 11:17:08,025 Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ib-2-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ib-4-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ib-1-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ib-3-Data.db')]
INFO 11:17:08,039 Node /192.168.14.11 state jump to normal
INFO 11:17:08,042 Startup completed! Now serving reads.
INFO 11:17:08,100 Not starting native transport as requested. Use JMX
(StorageService->startNativeTransport()) to start it
INFO 11:17:08,103 Binding thrift service to /192.168.14.11:9160
INFO 11:17:08,137 Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-ib-5,]. 771 bytes to 409
(~53% of original) in 103ms = 0,003787MB/s. 4 total rows, 1 unique. Row merge
counts were {1:0, 2:0, 3:0, 4:1, }
INFO 11:17:08,142 Using TFramedTransport with a max frame size of 15728640
bytes.
INFO 11:17:08,151 Using synchronous/threadpool thrift server on mr32.box : 9160
INFO 11:17:08,152 Listening for thrift clients...
