Hi Viktor, We are only deleting columns. Rows are never deleted. We are continually adding new columns that are then deleted. Existing columns (deleted or otherwise) are never updated.
Can you provide any pointers as to what I should investigate to determine what is occurring? Our application logic did not change between Cassandra 0.8 and the upgrade to 1.0.8 (but that does not mean that there is some bug in our logic that is causing the problem). Thanks, Ross On 23 March 2012 19:04, Viktor Jevdokimov <viktor.jevdoki...@adform.com>wrote: > Yes, continued deletions of *the same* columns/rows will prevent > removing them from final sstable upon compaction due to new timestamp.**** > > You’re getting sliding tombstone gc grace period in that case.**** > > ** ** > > During compaction of selected sstables Cassandra checks the whole Column > Family for the latest timestamp of the column/row, including other sstables > and memtable.**** > > ** ** > > You need to review your application logic.**** > > ** ** > > **** > ** > > ** ** > > Best regards/ Pagarbiai**** > > ** ** > > *Viktor Jevdokimov* > > Senior Developer**** > > ** ** > > Email: viktor.jevdoki...@adform.com**** > > Phone: +370 5 212 3063****. Fax: +370 5 261 0453**** > > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania**** > > ** ** > > ** ** > > [image: Adform news] <http://www.adform.com/> > > [image: Visit us!] > > Follow: > > [image: twitter] <http://twitter.com/#%21/adforminsider> > > Visit our blog <http://www.adform.com/site/blog> > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies.**** > > *From:* Ross Black [mailto:ross.w.bl...@gmail.com] > *Sent:* Friday, March 23, 2012 07:16 > *To:* user@cassandra.apache.org > *Subject:* Re: tombstones problem with 1.0.8**** > > ** ** > > Hi Victor, > > Thanks for your response. > > Is there a possibility that continual deletions during compact could be > blocking removal of the tombstones? The full manual compact takes about 4 > hours per node for our data, so there is a large number of deletes > occurring during that time. > > This is the description from cassandra-cli > > Replication Strategy: org.apache.cassandra.locator.SimpleStrategy > Durable Writes: true > Options: [replication_factor:3] > Column Families: > ColumnFamily: weekly > Key Validation Class: org.apache.cassandra.db.marshal.BytesType > Default column value validator: > org.apache.cassandra.db.marshal.BytesType > Columns sorted by: org.apache.cassandra.db.marshal.BytesType > Row cache size / save period in seconds / keys to save : 0.0/0/all > Row Cache Provider: > org.apache.cassandra.cache.SerializingCacheProvider > Key cache size / save period in seconds: 200000.0/14400 > GC grace seconds: 3600 > Compaction min/max thresholds: 3/8 > Read repair chance: 1.0 > Replicate on write: true > Bloom Filter FP chance: default > Built indexes: [] > Compaction Strategy: > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy > > > Ross > > > **** > > On 23 March 2012 02:55, Viktor Jevdokimov <viktor.jevdoki...@adform.com> > wrote:**** > > Just tested 1.0.8 before upgrading from 1.0.7: tombstones created by TTL > or by delete operation are perfectly deleted after either compaction or > cleanup.**** > > Have no idea about any other settings than gc_grace_seconds, check you > schema from cassandra-cli.**** > > **** > > **** > > **** > > **** > > Best regards/ Pagarbiai**** > > **** > > *Viktor Jevdokimov***** > > Senior Developer**** > > **** > > Email: viktor.jevdoki...@adform.com**** > > Phone: +370 5 212 3063. Fax: +370 5 261 0453**** > > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania**** > > **** > > **** > > [image: Adform news] <http://www.adform.com/> > > [image: Visit us!]**** > > Follow:**** > > [image: twitter] <http://twitter.com/#%21/adforminsider>**** > > Visit our blog <http://www.adform.com/site/blog>**** > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies.**** > > ** ** > > *From:* Ross Black [mailto:ross.w.bl...@gmail.com] > *Sent:* Thursday, March 22, 2012 03:38 > *To:* user@cassandra.apache.org > *Subject:* tombstones problem with 1.0.8**** > > **** > > Hi, > > We recently moved from 0.8.2 to 1.0.8 and the behaviour seems to have > changed so that tombstones are now not being deleted. > > Our application continually adds and removes columns from Cassandra. We > have set a short gc_grace time (3600) since our application would > automatically delete zombies if they appear. > Under 0.8.2, the tombstones remained at a relatively constant number. > Under 1.0.8, the tombstones have been continually increasing so that they > exceed the size of our real data (at this stage we have over 100G of > tombstones). > Even after running a full compact the new compacted SSTable contains a > massive number of tombstones, many that are several weeks old. > > Have I missed some new configuration option to allow deletion of > tombstones? > > I also noticed that one of the changes between 0.8.2 and 1.0.8 was > https://issues.apache.org/jira/browse/CASSANDRA-2786 which changed code > to "avoid dropping tombstones when they might still be needed to shadow > data in another sstable". > Could this be having an impact since we continually add and remove columns > even while a major compact is executing? > > > Thanks, > Ross**** > > ** ** >
<<dm-exco4823.png>>
<<signature-logo29.png>>
<<image002.png>>
<<image003.png>>
<<tweet18be.png>>
<<image001.png>>
