Micro-batching in Kafka streams - redux

2017-10-20 Thread Shrijeet Paliwal
Kafka version: 0.10.2.1 (can upgrade if needed) I wish to revive the discussion around micro batching in Kafka streams. Some past discussions are here & here . I am exploring ways to do at-least-once proces

Re: WorkerSinkTask not committing offsets (potential bug)

2017-09-08 Thread Shrijeet Paliwal
I found that this was already reported here https://issues.apache.org/jira/browse/KAFKA-5731 and fixed! -- Shrijeet On Thu, Sep 7, 2017 at 6:04 PM, Shrijeet Paliwal wrote: > > *Kafka Version: 0.10.2.1* > > Hi, > > I am running a custom connector (in distributed mode) and no

WorkerSinkTask not committing offsets (potential bug)

2017-09-07 Thread Shrijeet Paliwal
*Kafka Version: 0.10.2.1* Hi, I am running a custom connector (in distributed mode) and noticed one of the partition has its lag increasing consistently although it's assigned to a connect worker. Log messages in the connect log follow: [DEBUG] 2017-09-07 14:32:54,572 runtime.WorkerSinkTask onPa

Re: Consumer thread is waiting forever, not returning any objects

2016-03-29 Thread Shrijeet Paliwal
ort, and i believe thats correct because > consumer.sh script also working?) > #* ./kafka-console-consumer.sh --zookeeper xxx:2181 --topic mytopic* > > On 30 March 2016 at 10:29, Shrijeet Paliwal > wrote: > > > On Tue, Mar 29, 2016 at 4:19 PM, Ratha v wrote: > > &

Re: Consumer thread is waiting forever, not returning any objects

2016-03-29 Thread Shrijeet Paliwal
On Tue, Mar 29, 2016 at 4:19 PM, Ratha v wrote: > 9092 Are your brokers listening at this port? -- Shrijeet

Re: Consumer thread is waiting forever, not returning any objects

2016-03-29 Thread Shrijeet Paliwal
Can you share complete Consumer configuration? Also what does stack-trace show, is it stuck in poll? I think I have seen this before in one of my tests, in my case I had pinned it to side effect of https://issues.apache.org/jira/browse/KAFKA-1894. Essentially poll(timeout) doesn't always respect

KafkaConsumer#poll not returning records for all partitions of topic in single call

2016-03-10 Thread Shrijeet Paliwal
Version: 0.9.0.1 I have a test which creates two partitions in a topic, writes data to both partitions. Then a single consumer subscribes to the topic, verifies that it has got the assignment of both partitions in that topic & finally issues a poll. The firs poll always comes back with records of