This is a great set of projects! We should put this list of projects on a site somewhere so people can more easily see and refer to it. These aren't Kafka-specific, but most seem to be "MySQL CDC." Does anyone have a place where they can host a page? Preferably a wiki, so we can keep it up to date easily.
-James On Mar 17, 2015, at 8:21 AM, Hisham Mardam-Bey <hisham.mardam...@gmail.com> wrote: > Pretty much a hijack / plug as well (= > > https://github.com/mardambey/mypipe > > "MySQL binary log consumer with the ability to act on changed rows and > publish changes to different systems with emphasis on Apache Kafka." > > Mypipe currently encodes events using Avro before pushing them into Kafka > and is Avro schema repository aware. The project is young; and patches for > improvements are appreciated (= > > On Mon, Mar 16, 2015 at 10:35 PM, Arya Ketan <ketan.a...@gmail.com> wrote: > >> Great work. >> Sorry for kinda hijacking this thread, but I though that we had built >> some-thing on mysql bin log event propagator and wanted to share it . >> You guys can also look into Aesop ( https://github.com/Flipkart/aesop). >> Its >> a change propagation frame-work. It has relays which listens to bin logs of >> Mysql, keeps track of SCNs and has consumers which can then (transform/map >> or interpret as is) the bin log-event to a destination. Consumers also keep >> track of SCNs and a slow consumer can go back to a previous SCN if it wants >> to re-listen to events ( similar to kafka's consumer view ). >> >> All the producers/consumers are extensible and you can write your own >> custom consumer and feed off the data to it. >> >> Common use-cases: >> a) Archive mysql based data into say hbase >> b) Move mysql based data to say a search store for serving reads. >> >> It has a decent ( not an awesome :) ) console too which gives a nice human >> readable view of where the producers and consumers are. >> >> Current supported producers are mysql bin logs, hbase wall-edits. >> >> >> Further insights/reviews/feature reqs/pull reqs/advices are all welcome. >> >> -- >> Arya >> >> Arya >> >> On Tue, Mar 17, 2015 at 1:48 AM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> >>> Really really nice! >>> >>> Thank you. >>> >>> On Mon, Mar 16, 2015 at 7:18 AM, Pierre-Yves Ritschard <p...@spootnik.org >>> >>> wrote: >>>> Hi kafka, >>>> >>>> I just wanted to mention I published a very simple project which can >>>> connect as MySQL replication client and stream replication events to >>>> kafka: https://github.com/pyr/sqlstream >>>> >>>> When you don't have control over an application, it can provide a >> simple >>>> way of consolidating SQL data in kafka. >>>> >>>> This is an early release and there are a few caveats (mentionned in the >>>> README), mostly the poor partitioning which I'm going to evolve quickly >>>> and the reconnection strategy which doesn't try to keep track of binlog >>>> position, other than that, it should work as advertised. >>>> >>>> Cheers, >>>> - pyr >>> >> > > > > -- > Hisham Mardam-Bey > http://hisham.cc/
