Re: RecordTooLargeException

Setting max.request.size does not prevent RecordTooLargeException from being thrown.   Thanks for the header info. 9. Jul 2018 09:20 by jiangtao@zuora.com .INVALID: > ​you can configure max.request.size > <> https://kafka.apache.org/documentation/#configura

Question - Partition Lag

Hello users. We have a 7 broker cluster running 0.11.0.2. We are seeing an issue with one of the partitions in a topic. We see lag on that partition even though there are many consumer threads available to process the messages. Any inputs/thoughts? Here is the metadata Topic:topic_name Partit

Unable to read topic from beginning after certain runs

Hi all, I am new to Kafka. I have a problem reading from a topic multiple times with kafka-console-consumer and a Java client. Any help is appreciated. I have used a tool called Kandalf (https://github.com/hellofresh/kandalf) to extract RabbitMQ messages to be pushed in to a Kafka topic `usage-re

RE: Kafka Connect: Increase Consumer Consumption

At this point of investigation, I have no idea De : Vishnu Manivannan Envoyé : vendredi 20 juillet 2018 21:28:24 À : users@kafka.apache.org Objet : Re: Kafka Connect: Increase Consumer Consumption No, I'm just running one consumer now and trying to increase i

Re: Kafka Connect: Increase Consumer Consumption

No, I'm just running one consumer now and trying to increase its consumption before I increase the number of consumers. Yeah I know it's weird, I thought I would see some changes in data consumption after tweaking these parameters. On 7/18/18, 11:48 PM, "adrien ruffie" wrote: Strange e

Re: UNKNOWN_PRODUCER_ID error when running Streams WordCount demo with processing.guarantee set to EXACTLY_ONCE

Created JIRA https://issues.apache.org/jira/browse/KAFKA-7190 -Bill On Fri, Jul 20, 2018 at 1:26 PM Guozhang Wang wrote: > Hello Shaobo, > > The situation you observed is a valid case: when a streams application has > very low traffic (like the quickstart example, where you manually type one >

Question - Partition Lag

Hello users. We have a 7 broker cluster running 0.11.0.2. We are seeing an issue with one of the partitions in a topic. We see lag on that partition even though there are many consumer threads available to process the messages. Any inputs/thoughts? Here is the metadata Topic:topic_name Partit

Re: UNKNOWN_PRODUCER_ID error when running Streams WordCount demo with processing.guarantee set to EXACTLY_ONCE

Hello Shaobo, The situation you observed is a valid case: when a streams application has very low traffic (like the quickstart example, where you manually type one message at a time), then it is possible that consumer purging would delete even the last message sent by this producer (i.e. all the m

Re: Use Kafka Streams for windowing data and processing each window at once

Hello Bruno, We've discussed about the callback approach before, but then we realized with the proposed API, this can still be achieved. In the "Final window results per key" section, John showed how to do that. Note the resulted stream will have exactly one record per each key, with the value rep

Re: Use Kafka Streams for windowing data and processing each window at once

Bill, After reading the documentation and sure it looks really close to our need however I had a doubt about it. One small question. I was expecting also a callback that Kafka would call after the whole period is passed and this callback would receive an iterable object that contains all the agg

Re: Measure latency from Source to Sink

Awesome thank you very much I will check an try with this.. I am pumping data with a simple Python producer script to send data to just 1 topic El vie., 20 jul. 2018 a las 11:04, Thakrar, Jayesh (< jthak...@conversantmedia.com>) escribió: > Below are some examples. > > They might be old, but the

Re: Measure latency from Source to Sink

Below are some examples. They might be old, but the concept is the same - essentially you are implementing an interface. You have to be conscious of not to do heavy or high latency processing to be non-impactful to the data flow. https://www.ashishpaliwal.com/blog/2013/06/flume-cookbook-implemen

Re: Measure latency from Source to Sink

Hi Actually I am running the app I a single node this is for a POC, I was not aware of the custom interceptors do you have an example of this? Best Regards El vie., 20 jul. 2018 a las 9:20, Thakrar, Jayesh (< jthak...@conversantmedia.com>) escribió: > See if you can use a custom interceptors fo

Admin Client

I'm new in Apache Kafwa world and start using Apache Kafka Client. So I start one docker container with Apache Kafka and Zookeeper, but I can't understand what of API key and URL I need to provide to this class https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/client

Re: Measure latency from Source to Sink

See if you can use a custom interceptors for this. The only fuzzy thing is that the clocks would be different so I would be a little skeptical of its accuracy. I have heard of some companies who have a special topic in which they insert test msgs and then read them back - using the same machine f

Re: Use Kafka Streams for windowing data and processing each window at once

Hello Bill, Thank you very much for the response. We are looking forward to the KIP feature. Best Regards, Bruno On gio, 2018-07-19 at 12:59 -0400, Bill Bejeck wrote: > Hi Bruno, > > What you are asking is a common request.  There is a KIP in the > works, > https://cwiki.apache.org/confluence/