Also, have you looked at Kafka Connect released in 0.9? It has a MySQL binlog connector implementation in progress, just thinking maybe you would be interested to check it out and see if there are any feedbacks that you want to give.
https://github.com/wushujames/kafka-mysql-connector Guozhang On Fri, Mar 18, 2016 at 7:38 AM, Ben Osheroff <b...@zendesk.com.invalid> wrote: > (lemme know if this belongs on the users email list, I'm not sure where > API questions fall) > > Hi, I'm Ben Osheroff, I wrote Maxwell > (http://github.com/zendesk/maxwell) and have been prototyping an engine > to do arbitrary denormalizations of Maxwell's CDC events based on the > kafka-streams library; the elevator pitch is that you can write SQL > joins which the engine compiles down to stream-joins and aggregations > and such. > > Maxwell partitions its stream by mysql database name, which means that > to do stream-joins I need to implement the same (custom) partitioning > algorithm somewhere in my stream processor. I'd prefer not drop down to > the lower level `addSink()` library calls if possible, and I can't > figure out how to mix and match the lower level alls with the higher > level DSL (map/filter/etc). > > So I guess I have two questions: > > 1. Is it somehow possible to add a custom `Sink` to an otherwise high > level stream topology? There's no obvious way to retrieve the topology > names that I can see. > > 2. If not, I'd like to make a feature request that the various stream > building functions (.to, .through) accept an optional > StreamPartitioner. > > 3. Any other ideas about how to pull this off? > > Thanks! > > > - Ben Osheroff > zendesk.com > -- -- Guozhang
