Great work. Sorry for kinda hijacking this thread, but I though that we had built some-thing on mysql bin log event propagator and wanted to share it . You guys can also look into Aesop ( https://github.com/Flipkart/aesop). Its a change propagation frame-work. It has relays which listens to bin logs of Mysql, keeps track of SCNs and has consumers which can then (transform/map or interpret as is) the bin log-event to a destination. Consumers also keep track of SCNs and a slow consumer can go back to a previous SCN if it wants to re-listen to events ( similar to kafka's consumer view ).
All the producers/consumers are extensible and you can write your own custom consumer and feed off the data to it. Common use-cases: a) Archive mysql based data into say hbase b) Move mysql based data to say a search store for serving reads. It has a decent ( not an awesome :) ) console too which gives a nice human readable view of where the producers and consumers are. Current supported producers are mysql bin logs, hbase wall-edits. Further insights/reviews/feature reqs/pull reqs/advices are all welcome. -- Arya Arya On Tue, Mar 17, 2015 at 1:48 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > Really really nice! > > Thank you. > > On Mon, Mar 16, 2015 at 7:18 AM, Pierre-Yves Ritschard <p...@spootnik.org> > wrote: > > Hi kafka, > > > > I just wanted to mention I published a very simple project which can > > connect as MySQL replication client and stream replication events to > > kafka: https://github.com/pyr/sqlstream > > > > When you don't have control over an application, it can provide a simple > > way of consolidating SQL data in kafka. > > > > This is an early release and there are a few caveats (mentionned in the > > README), mostly the poor partitioning which I'm going to evolve quickly > > and the reconnection strategy which doesn't try to keep track of binlog > > position, other than that, it should work as advertised. > > > > Cheers, > > - pyr >
