Re: KafkaStreams backoff for non-existing topic

Hi Patrik, Are you referring to the source topic not available or state directory not available? For source topic not available case, as Murilo asked about, the main issue is of the "split brain" case: in very early versions of Kafka, Streams does check if topics are available at the beginning be

Re: KafkaStreams backoff for non-existing topic

Hi Guozhang Just a small question, why can't this be checked when trying to instantiate KafkaStreams? The Topology should know all topics and the existence of the topics could be verified with the AdminClient. This would allow to fail fast similar to when the state directory is not available. Or am

Re: KafkaStreams backoff for non-existing topic

Hello Murilo, Just to give some more background to John's message and KAFKA-7970 here. The main reason of trickiness is around the scenario of "topics being partially available", e.g. say your application is joining to topics A and B, while topicA exists but topicB does not (it is surprisingly com

Re: KafkaStreams backoff for non-existing topic

Hi, Murlio, I found https://issues.apache.org/jira/browse/KAFKA-7970, which sounds like the answer is currently "yes". Unfortunately, it is still tricky to handle this case, although the situation may improve soon. In the mean time, you can try to work around it with the StateListener. When Strea

Get the last committed message in a partition

Hello Kafka users, I am using exactly once semantics provided by kafka. Therefore, my producer writes a message within a transaction. While my producer was sending 100th message, right between send() and commitTransaction(); I killed the producer process. I read last few uncommitted messages in

Kafka SSL setup - producer timing out

I am trying to set up a three node Kafka v1.1 cluster with SSL. I can consume messages via the SSL port but I cannot produce. The command is timing out after 60 seconds with the following error message: ERROR Error when sending message to topic myTopic with key: null, value: 5 bytes with error:

Re: Proxying the Kafka protocol

Thank you all. We have in the past exposed message streams backed by Kafka via a HTTP/POST and Websocket service which worked very well. We were able to filter messages based on schema compliance and it was very simple for the teams that generate the data to use. It also had no trouble scaling to

Re: Tracking progress for messages generated by a batch process

Just add some metadata to the last message to indicate whether current batch is end. Agreed@Henn. If work is batch-oriented, Spark or other batch processing system is a more suitable solution. ---Original--- From: "Harper Henn" Date: Tue, Mar 26, 2019 00:36 AM To: "users"; Subject: Re: Trac

Re: Tracking progress for messages generated by a batch process

Assuming you know how many items are in a batch ahead of time, could you just add a batch ID and position of a message within a batch to each message you send to topic A? Then your end application (streaming app 3) could check if every message in that batch has been processed, and trigger events wh

Product solution

Hi Kafka support team If I want to use Kafka with API gateway such as Kong, Tyk. Do you have any solution to integrate between 2 system? Regards, Peerapong.O

Source connectors stops working after the broker crashes.

Configuration: 3 br 1 zk 1 connect worker(distributed mode) 8 connectors(sink and source) Description of connect topics: [image: Снимок экрана 2019-03-22 в 17.00.56.png] Environment: K8s, gcloud, confluent images After crash of first broker there are several messages that broker is not a

Re: FetchConsumer RemoteTime broker metric

On 2019/03/24 15:40:47, Nayanjyoti Deka wrote: > Hi, > > I have been trying to make sense of the various jmx metrics exposed by the > brokers. Currentl,y I was looking into the FetchConsumer request metrics. > Here are the values I got. > #mbean = > kafka.network:type=RequestMetrics,name=Requ