Re: Using the commit log for external synchronization

2012-09-21 Thread Ben Hood
Brian, On Sep 22, 2012, at 1:46, "Brian O'Neill" wrote: >> IMHO it's a better design to multiplex the data stream at the application >> level. > +1, agreed. > > That is where we ended up. (and Storm is proving to be a solid > framework for that) Thanks for the heads up, I'll check it out. Che

Re: Using the commit log for external synchronization

2012-09-21 Thread Ben Hood
Rob, On Sep 22, 2012, at 0:39, Rob Coli wrote: > The above gets you most of the way there, but Aaron's point about the > commitlog not reflecting whether the app met its CL remains true. The > possibility that Cassandra might coalesce to a value that the > application does not know was successfu

Re: Using the commit log for external synchronization

2012-09-21 Thread Brian O'Neill
> IMHO it's a better design to multiplex the data stream at the application > level. +1, agreed. That is where we ended up. (and Storm is proving to be a solid framework for that) -brian On Fri, Sep 21, 2012 at 4:56 AM, aaron morton wrote: > The commit log is essentially internal implementation

Re: Using the commit log for external synchronization

2012-09-21 Thread Rob Coli
On Fri, Sep 21, 2012 at 4:31 AM, Ben Hood <0x6e6...@gmail.com> wrote: > So if I understand you correctly, one shouldn't code against what is > essentially an internal artefact that could be subject to change as > the Cassandra code base evolves and furthermore may not contain the > information an a

Re: Using the commit log for external synchronization

2012-09-21 Thread Ben Hood
Hi Aaron, Thanks for your input. On Fri, Sep 21, 2012 at 9:56 AM, aaron morton wrote: > The commit log is essentially internal implementation. The total size of the > commit log is restricted, and the multiple files used to represent segments > are recycled. So once all the memtables have been f

Re: Using the commit log for external synchronization

2012-09-21 Thread aaron morton
The commit log is essentially internal implementation. The total size of the commit log is restricted, and the multiple files used to represent segments are recycled. So once all the memtables have been flushed for segment it may be overwritten. To archive the segments see the conf/commitlog_a

Re: Using the commit log for external synchronization

2012-09-20 Thread Brian O'Neill
Along those lines... We sought to use triggers for external synchronization. If you read through this issue: https://issues.apache.org/jira/browse/CASSANDRA-1311 You'll see the idea of leveraging a commit log for synchronization, via triggers. We went ahead and implemented this concept in:

Re: Using the commit log for external synchronization

2012-09-20 Thread Michael Kjellman
+1. Would be a pretty cool feature Right now I write once to cassandra and once to kafka. On 9/20/12 4:13 PM, "Data Craftsman 木匠" wrote: >This will be a good new feature. I guess the development team don't >have time on this yet. ;) > > >On Thu, Sep 20, 2012 at 1:29 PM, Ben Hood <0x6e6...@gmai

Re: Using the commit log for external synchronization

2012-09-20 Thread Data Craftsman 木匠
This will be a good new feature. I guess the development team don't have time on this yet. ;) On Thu, Sep 20, 2012 at 1:29 PM, Ben Hood <0x6e6...@gmail.com> wrote: > Hi, > > I'd like to incrementally synchronize data written to Cassandra into > an external store without having to maintain an ind

Using the commit log for external synchronization

2012-09-20 Thread Ben Hood
Hi, I'd like to incrementally synchronize data written to Cassandra into an external store without having to maintain an index to do this, so I was wondering whether anybody is using the commit log to establish what updates have taken place since a given point in time? Cheers, Ben