Re: Data (re)processing with Kafka (new wiki page)

This page is really helpful.Thanks for putting this Some nice to have features can be (not sure for this wiki page) 1) Pause and resume without having to start and stop. It should drain all the inflight calculations before doing the actual pause and a notifier will be helpful that it is actually

Re: Kafka streaming changelog topic max.message.bytes exception

Hi Sachin Here is a possible general approach which might work for you in abscence of any broadcast variable and everything being local state and also if you can adjust you process to do everything before doing aggregation. Basically the idea here is to use a custom topology with custom processor

Re: Reopen KAFKA-4344 ?

Hi Srinivas I raised the issue and the way I got around this was to let kafka streams run on POJO way rather than some of the dependent instances being spring managed bean instances. If you create the instance of riakService and counterService in processor class instead of passing the spring manag

Re: Kafka Streams fails permanently when used with an unstable network

after fixing this JIRA: > https://issues.apache.org/jira/browse/KAFKA-3559 < > https://issues.apache.org/jira/browse/KAFKA-3559>. Feel free to give it a > go if rebalancing time continues to be a problem. > > Thanks > Eno > > > On 31 Oct 2016, at 19:44, saiprasad mis

Re: Kafka Streams fails permanently when used with an unstable network

Hey Guys I have noticed similar issues when network goes down on starting of kafka stream apps especially the store has initialized but the task initialization is not complete and when the network comes back the rebalance fails with the above error and I had to restart. as i run many partitions an

Re: issue with custom processor flush to rocksdb store

your valuable time on this Regards Sai On Wed, Oct 26, 2016 at 2:34 PM, Guozhang Wang wrote: > Is it a similar report as https://issues.apache.org/jira/browse/KAFKA-4344 > ? > > On Tue, Oct 25, 2016 at 2:43 PM, saiprasad mishra < > saiprasadmis...@gmail.com > > wrote: &

issue with custom processor flush to rocksdb store

Hi This is with version 10.1.0 kafka streams (server running in remote and streams app running local in my laptop). I have a kafka stream pipeline like this source topic(with 10 partitions) stream -> filter for null value ->map to make it keyed by id ->custom processor to mystore(persistent) I

Re: Exception when accessing partition, offset and timestamp in processor class

Just created the JIRA https://issues.apache.org/jira/browse/KAFKA-4344 Regards Sai On Tue, Oct 25, 2016 at 11:59 AM, saiprasad mishra < saiprasadmis...@gmail.com> wrote: > My JIRA id is saimishra > > Regards > Sai > > On Tue, Oct 25, 2016 at 10:55 AM, Matthias J. Sax

Re: Exception when accessing partition, offset and timestamp in processor class

ug to me; within > > Processor#process() accessing the context should work. Can you open > > a JIRA for it? > > > > -Matthias > > > > On 10/23/16 10:28 PM, saiprasad mishra wrote: > >>>> Sorry for the email again > >>>> > >>>>

Re: Exception when accessing partition, offset and timestamp in processor class

t; > - -Matthias > > On 10/23/16 10:28 PM, saiprasad mishra wrote: > > Sorry for the email again > > > > I was expecting it to work always when accessed from process() > > method as this corresponds to each kafka message/record processing. > > I understand il

Re: Exception when accessing partition, offset and timestamp in processor class

:18 PM, saiprasad mishra wrote: > Hi > > his is with my streaming app kafka 10.1.0. > > My flow looks something like below > > source topic stream -> filter for null value ->map to make it keyed by id > ->custom processor to mystore -> to another topic -> kt

Exception when accessing partition, offset and timestamp in processor class

Hi his is with my streaming app kafka 10.1.0. My flow looks something like below source topic stream -> filter for null value ->map to make it keyed by id ->custom processor to mystore -> to another topic -> ktable I am hitting the below type of exception in a custom processor class if I try to

Re: kafka streams metadata request fails on 0.10.0.1 broker/topic from 0.10.1.0 client

> cluster in this situation. Moving forward, we're planning to improve the > upgrade/compatibility story of Kafka so that you could, for example, run a > newer version of Kafka Streams (or any other Kafka client) against an older > version of Kafka. > > > > On T

kafka streams metadata request fails on 0.10.0.1 broker/topic from 0.10.1.0 client

Hi All Was testing with 0.10.1.0 rc3 build for my new streams app Seeing issues starting my kafk streams app( 0.10.1.0) on the old version broker 0.10.0.1. I dont know if it is supposed to work as is. Will upgrade the broker to same version and see whether it goes away client side issues ==

Re: Kafka consumer does not work from eclipse when kafka running inside VM

I have seen the zk consumers directory getting corrupted due to running your consumer again and again with some settings changed and causing this issue. I will do a removal of zk directory from kafka server's zk like below rmr /consumers/hello and exit zookeeper shell. Then run your consumer and