Re: Process offsets from particular point

2020-04-11 Thread KhajaAsmath Mohammed
Thanks Senthil. This is helpful but I am worried about doing it with standalone process as our data is huge. Is there a way to do the same thing using kstream and utilize cluster resources instead of doing with standalone client process ? Sent from my iPhone > On Apr 11, 2020, at 7:27 PM, Se

Re: Process offsets from particular point

2020-04-11 Thread Matthias J. Sax
If cannot (or don't want to) modify your code, you can also stop the whole application and use `bin/kafka-consumer-groups.sh` to set a new start offset per partition. Afterward, you can just restart the application and it will pick up the corresponding start offsets. -Matthias On 4/11/20 6:56 AM,

Re: Kafka Streams endless rebalancing

2020-04-11 Thread Peter Levart
On 4/11/20 2:45 PM, Alex Craig wrote: Yep, max poll interval is 2147483647 and session timeout is 12 (2 minutes). I don't have anything set for heartbeat.interval.ms, so it must be using the default. (3 seconds I think?) Hmm, is it possible the heartbeat might not happen if the client ap

Re: Process offsets from particular point

2020-04-11 Thread SenthilKumar K
Hi, We can re-consume the data from particular point using consumer.seek() and consumer.assign() API [1]. Pls check out documentation. If you have used timestamp at the time producing records , You can use particular timestamp to consume records [2]. https://kafka.apache.org/24/javadoc/index.htm

Process offsets from particular point

2020-04-11 Thread KhajaAsmath Mohammed
Hi, We have lost some data while processing and would like to reprocess it. May I know the procedure to do it . I have offsets numbers that I need to process. Any suggestions please. Would be really helpful. Thanks, Asmath Sent from my iPhone

Re: Kafka Streams endless rebalancing

2020-04-11 Thread Alex Craig
Yep, max poll interval is 2147483647 and session timeout is 12 (2 minutes). I don't have anything set for heartbeat.interval.ms, so it must be using the default. (3 seconds I think?) Hmm, is it possible the heartbeat might not happen if the client app was so swamped with restoring data (and m