Adding secondary index: java.lang.ArithmeticException: / by zero
I was about to add a secondary index (which apparently failed) to existing data. When I restarted the node it crashed (!) with: INFO 09:21:36,510 Opening /var/lib/cassandra/data/test/tracking.6b6579-tmp-e-1 ERROR 09:21:36,512 Exception encountered during startup. java.lang.ArithmeticException: / by zero at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233) at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:284) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:200) at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:225) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448) at org.apache.cassandra.db.ColumnFamilyStore.addIndex(ColumnFamilyStore.java:305) at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:246) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:436) at org.apache.cassandra.db.Table.initCf(Table.java:360) at org.apache.cassandra.db.Table.(Table.java:290) at org.apache.cassandra.db.Table.open(Table.java:107) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138) at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134) Exception encountered during startup. java.lang.ArithmeticException: / by zero at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233) at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:284) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:200) at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:225) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448) at org.apache.cassandra.db.ColumnFamilyStore.addIndex(ColumnFamilyStore.java:305) at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:246) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:436) at org.apache.cassandra.db.Table.initCf(Table.java:360) at org.apache.cassandra.db.Table.(Table.java:290) at org.apache.cassandra.db.Table.open(Table.java:107) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138) at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134) So, I deleted the file which let's cassandra to startup again (and starting all over to build the secondary index). Since 0.7rc2 was too unstable I'm on SNAPSHOT from Dec 17.
Re: Cassandra Node Routinely Goes Down - 0.7 RC2
On Wed, Dec 22, 2010 at 7:21 AM, Dan Hendry wrote: > Can one of the Cassandra devs or anybody who knows about memory mapping > comment on this/my particular mmap situation? Pardon me if this has been covered or if you are already aware, but if not, you might find : https://issues.apache.org/jira/browse/CASSANDRA-1214 To be interesting background reading. =Rob
Re: Adding secondary index: java.lang.ArithmeticException: / by zero
On Dec 23, 2010, at 9:34, Timo Nentwig wrote: > I was about to add a secondary index (which apparently failed) to existing > data. When I restarted the node it crashed (!) with: It crashed because it ran out of heap space (2G). So I increased to 3.5G but after a whlie it's caught in full GC again. The node holds some 50G of data. So, if 3.5G isn't sufficient, how do I determine how much memory it's going to need and more important how can I cancel such indexing if I don't have enough memory available? Or did I hit some bug and should simply wait for rc3? > INFO 09:21:36,510 Opening /var/lib/cassandra/data/test/tracking.6b6579-tmp-e-1 > ERROR 09:21:36,512 Exception encountered during startup. > java.lang.ArithmeticException: / by zero ... > So, I deleted the file which let's cassandra to startup again (and starting > all over to build the secondary index). > > Since 0.7rc2 was too unstable I'm on SNAPSHOT from Dec 17.
Having trouble getting cassandra to stay up
Hi, I am a newbie to cassandra and am using cassandra RC 2. I initially have cassndra working on one node and was able to create keyspace, column families and populate the database fine. I tried adding a second node by changing the seed to point to another node and setting listen_address and rpc_address to blank. I then started up the second node and it seems to have connected fine using the node tool but after that I couldn't get it to accept any commands and whenever I tried to make a new keyspace or column family it would kill my initial node after a message like this: INFO 18:19:49,335 switching in a fresh Memtable for Schema at CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1293127746481.log', position=9143) INFO 18:19:49,335 Enqueuing flush of memtable-sch...@1358138608(2410 bytes, 5 operations) Killed and the next few time I start up the server a similar would pop up until I am guessing all the stuff is flushed out then it would start fine until I tried to add anything to it. I tried changing back the yaml file back to the original setup and this still happens. I don't know what to try to get it to work properly, if you guys can help I would be really grateful Alex
Re: cli 'list' command not returning all data, get_range_slices bug?
hi,i wrote a little test program to duplicate the problem and eliminate pelops (attached)
CClient.java
Description: Binary data
.i think the problem may be related tohttps://issues.apache.org/jira/browse/CASSANDRA-1198because watching the logs (at debug level) i see the remote node returningits data, but not the local node.here is the test program output:=== starting=== inserting row: test-key-0=== inserting row: test-key-1=== inserting row: test-key-2=== inserting row: test-key-3=== inserting row: test-key-4=== inserting row: test-key-5=== inserting row: test-key-6=== inserting row: test-key-7=== inserting row: test-key-8=== inserting row: test-key-9=== 1) got test-key-1 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-1)=== 2) got test-key-0 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-0)=== 3) got test-key-3 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-3)=== 4) got test-key-7 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-7)=== 5) got test-key-8 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-8)=== 6) got test-key-6 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-6)=== !! didnt retrieve all the keys, 4 not found=== crush all boxesand here is what i see in the local logsDEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 CassandraServer.java (line 479) range_sliceDEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 StorageProxy.java (line 412) RangeSliceCommand{keyspace='Harvest', column_family='TestCentroids', super_column=null, predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 0C 0C 00 01 0B 00 03 00 00 00 0D 54 65 73 74 43 65 6E 74 72 6F 69 64 73 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 0C 0C 00 01 0B 00 03 00 00 00 0D 54 65 73 74 43 65 6E 74 72 6F 69 64 73 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00, reversed:false, count:1)), range=[0,0], max_keys=11}DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 StorageProxy.java (line 597) restricted ranges for query [0,0] are [[0,0]]DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,959 StorageProxy.java (line 423) === endpoint: belize1.apple.com/17.224.36.17 for range.right 0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,963 StorageProxy.java (line 429) local range sliceDEBUG [pool-1-thread-4] 2010-12-23 12:54:26,964 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,964 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(95096621805687029609476958853893340535, 746573742d6b65792d31)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,966 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,967 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(102055691560625453887671437019240094911, 746573742d6b65792d30)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,969 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,970 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(117642756708984234120792643012013573692, 746573742d6b65792d33)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,971 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,971 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(120602233599396762327284348453927679148, 746573742d6b65792d37)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,971 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,972 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(137773384635200264868762149853169090749, 746573742d6b65792d38)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,972 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,972 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(145724861122056045272863275223173725919, 746573742d6b65792d36)DEBUG [pool-1-thread-4] 2010-12-23 12:54:27,013 ClientState.java (line 88) logged out: #and here are the remote node logsDEBUG [MutationStage:17] 2010-12-23 12:55:32,735 RowMutationVerbHandler.java (line 54) Applying RowMutation(keyspace='Harvest', key='746573742d6b65792d32', modifications=[ColumnFamily(TestCentroids [dummy-col:false:6...@0,])])DEBUG [MutationStage:17] 2010-12-23 12:55:32,736 Table.java (line 386) applying mutation of row 746573742d6b65792d32DEBUG [MutationStage:17] 2010-12-23 12:55:32,736 RowMutationVerbHandler.java (line 78) RowMutation(keyspace='Harvest', key='746573742d6b65792d32', modifications=[ColumnFamily(TestCentroids [dummy-col:false:6...@0,])]) applied. Sending response to 20602@/17.224.36.17DEBUG [MutationSt
Re: cli 'list' command not returning all data, get_range_slices bug?
sorry i should also have included my cassandra.ymland schema file too. here they are
cassandra.yaml-test
Description: Binary data
the only difference between the 2 nodes is the initial token, 0 for the local node (17.224.36.17)and 85070591730234615865843651857942052864 for the remote node (17.224.109.80)
harvest-setup-test
Description: Binary data
On Dec 23, 2010, at 1:00 PM, mike dooley wrote:hi,i wrote a little test program to duplicate the problem and eliminate pelops (attached).i think the problem may be related tohttps://issues.apache.org/jira/browse/CASSANDRA-1198because watching the logs (at debug level) i see the remote node returningits data, but not the local node.here is the test program output:=== starting=== inserting row: test-key-0=== inserting row: test-key-1=== inserting row: test-key-2=== inserting row: test-key-3=== inserting row: test-key-4=== inserting row: test-key-5=== inserting row: test-key-6=== inserting row: test-key-7=== inserting row: test-key-8=== inserting row: test-key-9=== 1) got test-key-1 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-1)=== 2) got test-key-0 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-0)=== 3) got test-key-3 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-3)=== 4) got test-key-7 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-7)=== 5) got test-key-8 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-8)=== 6) got test-key-6 = (dummy-col,twas brillig and the slithy toves did gyre and gimble in the wabe-6)=== !! didnt retrieve all the keys, 4 not found=== crush all boxesand here is what i see in the local logsDEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 CassandraServer.java (line 479) range_sliceDEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 StorageProxy.java (line 412) RangeSliceCommand{keyspace='Harvest', column_family='TestCentroids', super_column=null, predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 0C 0C 00 01 0B 00 03 00 00 00 0D 54 65 73 74 43 65 6E 74 72 6F 69 64 73 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 0C 0C 00 01 0B 00 03 00 00 00 0D 54 65 73 74 43 65 6E 74 72 6F 69 64 73 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00, reversed:false, count:1)), range=[0,0], max_keys=11}DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 StorageProxy.java (line 597) restricted ranges for query [0,0] are [[0,0]]DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,959 StorageProxy.java (line 423) === endpoint: belize1.apple.com/17.224.36.17 for range.right 0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,963 StorageProxy.java (line 429) local range sliceDEBUG [pool-1-thread-4] 2010-12-23 12:54:26,964 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,964 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(95096621805687029609476958853893340535, 746573742d6b65792d31)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,966 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,967 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(102055691560625453887671437019240094911, 746573742d6b65792d30)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,969 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,970 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(117642756708984234120792643012013573692, 746573742d6b65792d33)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,971 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,971 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(120602233599396762327284348453927679148, 746573742d6b65792d37)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,971 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,972 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(137773384635200264868762149853169090749, 746573742d6b65792d38)DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,972 SliceQueryFilter.java (line 129) collecting 0 of 1: dummy-col:false:6...@0DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,972 ColumnFamilyStore.java (line 1302) scanned DecoratedKey(145724861122056045272863275223173725919, 746573742d6b65792d36)DEBUG [pool-1-thread-4] 2010-12-23 12:54:27,013 ClientState.java (line 88) logged out: #and here are the remote node logsDEBUG [MutationStage:17] 2010-12-23 12:55:32,735 RowMutationVerbHandler.java (line 54) Applying RowMutation(keyspace='Harvest', key='746573742d6b65792d32', modifications=[ColumnFamily(TestCentroids [dummy-col:false:6...@0,])])DEBUG [MutationStage:17] 2010-12-23 12:
Re: Having trouble getting cassandra to stay up
Your details are rather vague, what do you mean by killed? Is the Cassandra java process still running? Any other warning or error log messages (from either node)? Could you provide the last few Cassandra log lines from each machine? Can you connect to the node via JMX? What is the output of nodetool ring from the second node (which is presumably still alive)? Is there any unusual system activity: high cpu usage, low cpu usage, problems with disk IO (can be checked with vmstat). Can you provide any further system information? Linux/windows, java version, 32/64 bit, amount of ram? On Thu, Dec 23, 2010 at 1:42 PM, Alex Quan wrote: > Hi, > > I am a newbie to cassandra and am using cassandra RC 2. I initially have > cassndra working on one node and was able to create keyspace, column > families and populate the database fine. I tried adding a second node by > changing the seed to point to another node and setting listen_address and > rpc_address to blank. I then started up the second node and it seems to have > connected fine using the node tool but after that I couldn't get it to > accept any commands and whenever I tried to make a new keyspace or column > family it would kill my initial node after a message like this: > > INFO 18:19:49,335 switching in a fresh Memtable for Schema at > CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1293127746481.log', > position=9143) > INFO 18:19:49,335 Enqueuing flush of memtable-sch...@1358138608(2410 > bytes, 5 operations) > Killed > > and the next few time I start up the server a similar would pop up until I > am guessing all the stuff is flushed out then it would start fine until I > tried to add anything to it. I tried changing back the yaml file back to the > original setup and this still happens. I don't know what to try to get it to > work properly, if you guys can help I would be really grateful > > Alex >
Re: cli 'list' command not returning all data, get_range_slices bug?
On Thu, Dec 23, 2010 at 3:00 PM, mike dooley wrote: > DEBUG [pool-1-thread-4] 2010-12-23 12:54:26,958 StorageProxy.java (line 597) > restricted ranges for query [0,0] are [[0,0]] This is the bug. It's not going to query the remote node unless 85070591730234615865843651857942052864 is part of the ranges generated. I've created https://issues.apache.org/jira/browse/CASSANDRA-1901 for this I suspect the bug is related to the one node having a token of 0, the minimum possible, which gets special treatment in some code paths. You could try changing it to 1 as a workaround. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
complexity
Hi, I've seen a similar question has been asked in this forum in Sept, but not answered. What is the complexity of get(row) and get(row, column-name) operations, and insert(row, column)? What about accessing or inserting a column within a SuperColumn by name? In Arin Sarkissian's "WTF is a Supercolumn" he stresses that columns are stored sorted by name. Are they simply stored in a sorted list, requiring a search when inserting and accessing (I forget the worst case complexity of searching/inserting into/deleting from a sorted list), or in some data structure with faster access? Even if you don't know the big-O complexity, a description of the implementation data structure(s), and discussion of what is fast and what is not, would help. Thanks for your help, Kevin.
