kewl, > * Row is not deleted (other columns can be read, row survives compaction > with GCGraceSeconds=0)
IIRC row tombstones can hang around for a while (until gc grace has passed), and they only have an effect on columns that have a lower timstamp. So it's possible to read columns from a row with a tombstone. Can you read the column using a slice range rather than specifying it's name ? Aaron ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 11/10/2011, at 11:15 PM, Thomas Richter wrote: > Hi Aaron, > > i invalidated the caches but nothing changed. I didn't get the mentioned > log line either, but as I read the code SliceByNamesReadCommand uses > NamesQueryFilter and not SliceQueryFilter. > > Next, there is only one SSTable. > > I can rule out that the row is deleted because I deleted all other rows > in that CF to reduce data size and speed up testing. I set > GCGraceSeconds to zero and ran a compaction. All other rows are gone, > but i can still access at least one column from the left row. > So as far as I understand it, there should not be a tombstone on row level. > > To make it a list: > > * One SSTable, one row > * > * Row is not deleted (other columns can be read, row survives compaction > with GCGraceSeconds=0) > * Most columns can be read by get['row']['col'] from cassandra-cli > * Some columns can not be read by get['row']['col'] from cassandra-cli > but can be found in output of sstable2json > * unreadable data survives compaction with GCGraceSeconds=0 (checked > with sstable2json) > * Invalidation caches does not help > * Nothing in the logs > > Does that point into any direction where i should look next? > > Best, > > Thomas > > On 10/11/2011 10:30 AM, aaron morton wrote: >> Nothing jumps out. The obvious answer is that the column has been deleted. >> Did you check all the SSTables ? >> >> It looks like query returned from row cache, otherwise you would see this as >> well… >> >> DEBUG [ReadStage:34] 2011-10-11 21:11:11,484 SliceQueryFilter.java (line >> 123) collecting 0 of 2147483647: 1318294191654059:false:354@1318294191654861 >> >> Which would mean a version of the column was found. >> >> If you invalidate the cache with nodetool and run the query and the log >> message appears it will mean the column was read from (all of the) sstables. >> If you do not get a column returned I would say there is a tombstone in >> place. It's either a row level or a column level one. >> >> Hope that helps. >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 11/10/2011, at 10:35 AM, Thomas Richter wrote: >> >>> Hi Aaron, >>> >>> normally we use hector to access cassandra, but for debugging I switched >>> to cassandra-cli. >>> >>> Column can not be read by a simple >>> get CFName['rowkey']['colname']; >>> >>> Response is "Value was not found" >>> if i query another column, everything is just fine. >>> >>> Serverlog for unsuccessful read (keyspace and CF names replaced): >>> >>> DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,739 CassandraServer.java >>> (line 280) get >>> >>> DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,744 StorageProxy.java (line >>> 320) Command/ConsistencyLevel is >>> SliceByNamesReadCommand(table='Keyspace', >>> key=61636162626139322d396638312d343562382d396637352d393162303337383030393762, >>> columnParent='QueryPath(columnFamilyName='ColumnFamily', >>> superColumnName='null', columnName='null')', >>> columns=[574c303030375030,])/ONE >>> >>> DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,750 ReadCallback.java (line >>> 86) Blockfor/repair is 1/true; setting up requests to localhost/127.0.0.1 >>> >>> DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,750 StorageProxy.java (line >>> 343) reading data locally >>> >>> DEBUG [ReadStage:33] 2011-10-10 23:15:29,751 StorageProxy.java (line >>> 448) LocalReadRunnable reading SliceByNamesReadCommand(table='Keyspace', >>> key=61636162626139322d396638312d343562382d396637352d393162303337383030393762, >>> columnParent='QueryPath(columnFamilyName='ColumnFamily', >>> superColumnName='null', columnName='null')', columns=[574c303030375030,]) >>> >>> DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,818 StorageProxy.java (line >>> 393) Read: 67 ms. >>> >>> Log looks fine to me, but no result is returned. >>> >>> Best, >>> >>> Thomas >>> >>> On 10/10/2011 10:00 PM, aaron morton wrote: >>>> How are they unreadable ? You need to go into some details about what is >>>> going wrong. >>>> >>>> What sort of read ? >>>> What client ? >>>> What is in the logging on client and server side ? >>>> >>>> >>>> Try turning the logging up to DEBUG on the server to watch what happens. >>>> >>>> Cheers >>>> >>>> ----------------- >>>> Aaron Morton >>>> Freelance Cassandra Developer >>>> @aaronmorton >>>> http://www.thelastpickle.com >>>> >>>> On 10/10/2011, at 9:23 PM, Thomas Richter wrote: >>>> >>>>> Hi, >>>>> >>>>> no errors in the server logs. The columns are unreadable on all nodes at >>>>> any consistency level (ONE, QUORUM, ALL). We started with 0.7.3 and >>>>> upgraded to 0.7.6-2 two months ago. >>>>> >>>>> Best, >>>>> >>>>> Thomas >>>>> >>>>> On 10/10/2011 10:03 AM, aaron morton wrote: >>>>>> What error are you seeing in the server logs ? Are the columns >>>>>> unreadable at all Consistency Levels ? i.e. are the columns unreadable >>>>>> on all nodes. >>>>>> >>>>>> What is the upgrade history of the cluster ? What version did it start >>>>>> at ? >>>>>> >>>>>> Cheers >>>>>> >>>>>> >>>>>> ----------------- >>>>>> Aaron Morton >>>>>> Freelance Cassandra Developer >>>>>> @aaronmorton >>>>>> http://www.thelastpickle.com >>>>>> >>>>>> On 10/10/2011, at 7:42 AM, Thomas Richter wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> here is some further information. Compaction did not help, but data is >>>>>>> still there when I dump the row with sstable2json. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Thomas >>>>>>> >>>>>>> On 10/08/2011 11:30 PM, Thomas Richter wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> we are running a 3 node cassandra (0.7.6-2) cluster and some of our >>>>>>>> column families contain quite large rows (400k+ columns, 4-6GB row >>>>>>>> size). >>>>>>>> Replicaton factor is 3 for all keyspaces. The cluster is running fine >>>>>>>> for several months now and we never experienced any serious trouble. >>>>>>>> >>>>>>>> Some days ago we noticed, that some previously written columns could >>>>>>>> not >>>>>>>> be read. This does not always happen, and only some dozen columns out >>>>>>>> of >>>>>>>> 400k are affected. >>>>>>>> >>>>>>>> After ruling out application logic as a cause I dumped the row in >>>>>>>> question with sstable2json and the columns are there (and are not >>>>>>>> marked >>>>>>>> for deletion). >>>>>>>> >>>>>>>> Next thing was setting up a fresh single node cluster and copying the >>>>>>>> column family data to that node. Columns could not be read either. >>>>>>>> Right now I'm running a nodetool compact for the cf to see if data >>>>>>>> could >>>>>>>> be read afterwards. >>>>>>>> >>>>>>>> Is there any explanation for such behavior? Are there any suggestions >>>>>>>> for further investigation? >>>>>>>> >>>>>>>> TIA, >>>>>>>> >>>>>>>> Thomas >>>>>>> >>>>>> >>>>> >>>> >>> >> >
