Hi Aaron, normally we use hector to access cassandra, but for debugging I switched to cassandra-cli.
Column can not be read by a simple get CFName['rowkey']['colname']; Response is "Value was not found" if i query another column, everything is just fine. Serverlog for unsuccessful read (keyspace and CF names replaced): DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,739 CassandraServer.java (line 280) get DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,744 StorageProxy.java (line 320) Command/ConsistencyLevel is SliceByNamesReadCommand(table='Keyspace', key=61636162626139322d396638312d343562382d396637352d393162303337383030393762, columnParent='QueryPath(columnFamilyName='ColumnFamily', superColumnName='null', columnName='null')', columns=[574c303030375030,])/ONE DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,750 ReadCallback.java (line 86) Blockfor/repair is 1/true; setting up requests to localhost/127.0.0.1 DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,750 StorageProxy.java (line 343) reading data locally DEBUG [ReadStage:33] 2011-10-10 23:15:29,751 StorageProxy.java (line 448) LocalReadRunnable reading SliceByNamesReadCommand(table='Keyspace', key=61636162626139322d396638312d343562382d396637352d393162303337383030393762, columnParent='QueryPath(columnFamilyName='ColumnFamily', superColumnName='null', columnName='null')', columns=[574c303030375030,]) DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,818 StorageProxy.java (line 393) Read: 67 ms. Log looks fine to me, but no result is returned. Best, Thomas On 10/10/2011 10:00 PM, aaron morton wrote: > How are they unreadable ? You need to go into some details about what is > going wrong. > > What sort of read ? > What client ? > What is in the logging on client and server side ? > > > Try turning the logging up to DEBUG on the server to watch what happens. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 10/10/2011, at 9:23 PM, Thomas Richter wrote: > >> Hi, >> >> no errors in the server logs. The columns are unreadable on all nodes at >> any consistency level (ONE, QUORUM, ALL). We started with 0.7.3 and >> upgraded to 0.7.6-2 two months ago. >> >> Best, >> >> Thomas >> >> On 10/10/2011 10:03 AM, aaron morton wrote: >>> What error are you seeing in the server logs ? Are the columns unreadable >>> at all Consistency Levels ? i.e. are the columns unreadable on all nodes. >>> >>> What is the upgrade history of the cluster ? What version did it start at ? >>> >>> Cheers >>> >>> >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 10/10/2011, at 7:42 AM, Thomas Richter wrote: >>> >>>> Hi, >>>> >>>> here is some further information. Compaction did not help, but data is >>>> still there when I dump the row with sstable2json. >>>> >>>> Best, >>>> >>>> Thomas >>>> >>>> On 10/08/2011 11:30 PM, Thomas Richter wrote: >>>>> Hi, >>>>> >>>>> we are running a 3 node cassandra (0.7.6-2) cluster and some of our >>>>> column families contain quite large rows (400k+ columns, 4-6GB row size). >>>>> Replicaton factor is 3 for all keyspaces. The cluster is running fine >>>>> for several months now and we never experienced any serious trouble. >>>>> >>>>> Some days ago we noticed, that some previously written columns could not >>>>> be read. This does not always happen, and only some dozen columns out of >>>>> 400k are affected. >>>>> >>>>> After ruling out application logic as a cause I dumped the row in >>>>> question with sstable2json and the columns are there (and are not marked >>>>> for deletion). >>>>> >>>>> Next thing was setting up a fresh single node cluster and copying the >>>>> column family data to that node. Columns could not be read either. >>>>> Right now I'm running a nodetool compact for the cf to see if data could >>>>> be read afterwards. >>>>> >>>>> Is there any explanation for such behavior? Are there any suggestions >>>>> for further investigation? >>>>> >>>>> TIA, >>>>> >>>>> Thomas >>>> >>> >> >
