Re: synchronizing streams of different kafka topics

2015-11-17 Thread Cody Koeninger
Are you using the direct stream? Each batch should contain all of the unprocessed messages for each topic, unless you're doing some kind of rate limiting. On Tue, Nov 17, 2015 at 3:07 AM, Antony Mayi wrote: > Hi, > > I have two streams coming from two different kafka topics. the two topics > con

synchronizing streams of different kafka topics

2015-11-17 Thread Antony Mayi
Hi, I have two streams coming from two different kafka topics. the two topics contain time related events but are quite asymmetric in volume. I would obviously need to process them in sync to get the time related events together but with same processing rate if the heavier stream starts backlogg