Re: Consumer commit logic

consumer offset commit can be controlled automatically or manually. What you are mentioning here is your own application problems. As I previously recommended, you should not hold your application in infinite loop just to process these failed messages. Also, I would certainly not recommend stoppin

Re: One Partition missing a node in ISR

hello, Could you please try to run kafka-reassign-partitions with your topic reassignment JSON? That doesn't require any restart, and should tell you if any issues with reassignment. The examples are provided in Confluence wiki. I would recommend that you do a "Describe" on your topic to ensure t

Apache Kafka training

Hi All, Can someone suggest a trainer for Apache Kafka who can train on all aspects of Kafka from beginner to advanced? Would be great, if there is some training in Mumbai. Thanks Sachin

Re: Apache Kafka training

If you like to study online then there's a series by Stephane Maarek on udemy , till now it's one of the best content I ever saw . Btw I am not promoting either of the 2 (udemy or Stephane ) in any way Regards On Thu, 26 Sep 2019, 3:59 pm Sachin Bal, wrote: > Hi All, > > Can someone suggest a

Re: Apache Kafka training

If you have access to Pluralsight, Apache Kafka has a good coverage there. On Thu, 26 Sep 2019 at 12:28, Arvind Sharma wrote: > If you like to study online then there's a series by Stephane Maarek on > udemy , till now it's one of the best content I ever saw . > > Btw I am not promoting either o

Re: Kafka Streams can't run normally after restart/redeployment

So I'm not exactly sure why supress() isn't working for you, because it should send out a final message once the window closes - assuming you're still getting new messages flowing through the topology. Have you tried using the count function in KGroupedTable? It should handle duplicates correctly.

Monitoring Broker/Prod/Cons on Kubernetes

Hello, Has anyone got any experience in using monitoring tool (e.g. Prometheus, DataDog, or custom) for Kafka on K8s? Cruise-control doesn't come with official docker image (Neither does Kafka Manager). I was just curious to know if people usually monitor this as a service through external provid

can't open site-docs index.html in Binary downloads package

I downloads a Binary package(kafka_2.12-2.3.0) and index.html in site-docs dir is can't be effective. like these pics.

Re: Consumer commit logic

I don't agree with you on that, 1) auto.offset.reset is used only when the consumer doesn't have an offset committed so the first time is consuming a partition 2) the fact that the consumer goes to the next message even if I don't commit the previous one was something it wasn't clear to me because

Re: can't open site-docs index.html in Binary downloads package

You need to setup a local web server as described in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/Setup+Kafka+Website+on+Local+Apache+Server -Matthias On 9/26/19 8:28 AM, 944126367 wrote: > I downloads a Binary package(kafka_2.12-2.3.0) and index.html in > site-docs dir is can't b

Re: Kafka Streams can't run normally after restart/redeployment

Thanks for the reply! I have two questions: 1) No output issue only happens when I restart/redeployment the application with the same application Id. But when I run the application first time, it works fine. Thus, I assume suppress() is working fine, at least fine for the first run. The thing I ca

Streams meetup at LinkedIn Sunnyvale, 6pm, Thursday, October 3, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infra team invites you to attend a Streams Processing meetup on Thursday, October 3, 2019 at LinkedIn's Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related streaming technologies

kafka streams application design on state stores

Hi, I am using Kafka Stream's RocksDB state stores in an application and the is a back ground job that sweeps the records in the store using range calls. During processing, we need to ensure that a record is not included in the next range call when that background jobs runs next time. I underst

Re: Kafka Streams can't run normally after restart/redeployment

1. Yeah I'm not sure why restarting is causing you problems. You shouldn't be changing your application ID just to get data flowing ,so something is wrong there I'm just not sure what. 2. Lag on the source topic? I guess that depends on how long your application is down and how quickly it can

Re: Kafka Streams can't run normally after restart/redeployment

Thanks Alex! Some updates: I tried to restart service with staging pool, which has far less traffic as production environment. And after restart, the application works fine without issues. I assume I can't restart the service in production, is caused by the huge lag in production? The lag is mostl

Re: Event time semantics across windows

I agree that there is overhead. It also happens when there are multiple applications producing into a single topic and if some application crashes/stops running for some time, then application consuming from this topic see a similar problem. I Though it might be application specific, are there

Kafka topic configuration to achieve highest data safety.

I'm using Kafka in a system where data safety should be at its notch. To set up a Kafka topic I need to guarantee that no message will be lost. What I found is, - if I use acks=1, we can't guarantee that messages will not be lost, - if I use acks= all, I will have a good data safety but uncl