It sounds like the problem is that the row is getting filled up with tombstones and becoming enormous? Another idea then, which might not be worth the added complexity, is to progressively use new rows. Depending on volume, this could mean having 5-minute-window rows, or 1 minute, or whatever works best.
Read: Assuming you're not falling behind, you only need to query the row that the current time falls in and the one immediately prior. If you do fall behind, you'll have to walk backwards in buckets until you find them empty. Write: Write column to the bucket (row) that corresponds to the correct time window. Delete: Delete the column from the row it was read from. When all columns in the row are deleted the row can GC. Again, cassandra might not be the correct datastore. On Wed, May 25, 2011 at 3:56 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > You're basically intentionally inflicting the worst case scenario on > the Cassandra storage engine: > http://wiki.apache.org/cassandra/DistributedDeletes > > You could play around with reducing gc_grace_seconds but a PQ with > "millions" of items is something you should probably just do in memory > these days. > > On Wed, May 25, 2011 at 10:43 AM, <dnalls...@taz.qinetiq.com> wrote: > > > > > > Hi all, > > > > I'm trying to implement a priority queue for holding a large number > (millions) > > of items that need to be processed in time order. My solution works - but > gets > > slower and slower until performance is unacceptable - even with a small > number > > of items. > > > > Each item essentially needs to be popped off the queue (some arbitrary > work is > > then done) and then the item is returned to the queue with a new > timestamp > > indicating when it should be processed again. We thus cycle through all > work > > items eventually, but some may come around more frequently than others. > > > > I am implementing this as a single Cassandra row, in a CF with a TimeUUID > > comparator. > > > > Each column name is a TimeUUID, with an arbitrary column value describing > the > > work item; the columns are thus sorted in time order. > > > > To pop items, I do a get() such as: > > > > cf.get(row_key, column_finish=now, column_start=yesterday, > column_count=1000) > > > > to get all the items at the head of the queue (if any) whose time exceeds > the > > current system time. > > > > For each item retrieved, I do a delete to remove the old column, then an > insert > > with a fresh TimeUUID column name (system time + arbitrary increment), > thus > > putting the item back somewhere in the queue (currently, the back of the > queue) > > > > I do a batch_mutate for all these deletes and inserts, with a queue size > of > > 2000. These are currently interleaved i.e. > delete1-insert1-delete2-insert2... > > > > This all appears to work correctly, but the performance starts at around > 8000 > > cycles/sec, falls to around 1800/sec over the first 250K cycles, and > continues > > to fall over time, down to about 150/sec, after a few million cycles. > This > > happens regardless of the overall size of the row (I have tried sizes > from 1000 > > to 100,000 items). My target performance is 1000 cycles/sec (but my data > store > > will need to handle other work concurrently). > > > > I am currently using just a single node running on localhost, using a > pycassa > > client. 4 core, 4GB machine, Fedora 14. > > > > Is this expected behaviour (is there just too much churn for a single row > to > > perform well), or am I doing something wrong? > > > > Would https://issues.apache.org/jira/browse/CASSANDRA-2583 in version > 0.8.1 fix > > this problem (I am using version 0.7.6)? > > > > Thanks! > > > > David. > > > > ---------------------------------------------------------------- > > This message was sent using IMP, the Internet Messaging Program. > > > > This email and any attachments to it may be confidential and are > > intended solely for the use of the individual to whom it is addressed. > > If you are not the intended recipient of this email, you must neither > > take any action based upon its contents, nor copy or show it to anyone. > > Please contact the sender if you believe you have received this email in > > error. QinetiQ may monitor email traffic data and also the content of > > email for the purposes of security. QinetiQ Limited (Registered in > > England & Wales: Company Number: 3796233) Registered office: Cody > Technology > > Park, Ively Road, Farnborough, Hampshire, GU14 0LX > http://www.qinetiq.com. > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
