Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-07-23 Thread Guozhang Wang
Hello Pieter, Thanks for your updates (and sorry I lost your email in my inbox for a month) ! In this release cycle (2.4) we are actively working on KIP-360 and and it should resolve the UKNOWN producer-id issue completely. At the mean time, I think I also agree with you that we do not need to pu

Re: Streams - Low level API

2019-07-23 Thread Matthias J. Sax
If you need to group-by an attribute/field in the value, you will need to extract the grouping attribute/field and set it as message key. Afterwards, you write it back into a topic to repartition the data accordingly and read it back to process it. -Matthias On 7/18/19 2:41 PM, Navneeth Krishnan

Re: SchemaException: Error reading field 'topics'

2019-07-23 Thread Kyla Palos
Hi Harper, For the producers, I am using 1.1.0. As for the brokers, I do not have idea as we are using Microsoft Event Hubs as our broker. Here's the error logs that I have when I run my producer and consumers: 19/07/12 15:22:20.354 ERROR reactor.kafka.receiver.internals.DefaultKafkaReceiver: Une

Re: SchemaException: Error reading field 'topics'

2019-07-23 Thread Harper Henn
Hi Kyla, What message format version are your producers and brokers using? That exception looks very similar to: https://stackoverflow.com/questions/39227213/why-kafka-0-10-console-producer-cannot-send-messages-to-kafka-0-9 On Mon, Jul 22, 2019 at 6:10 AM Kyla Palos wrote: > I have a producer

Kafka consumer is not reading some partition

2019-07-23 Thread Sergey Fedorov
Hello. I was using Kafka 2.1.1 and facing a problem where our consumers sometimes intermittently stop consuming from one or two of the partitions. My config

Re: Truncation

2019-07-23 Thread Jamie
Hi Guozhang,  Thanks very much, this was really useful. Jamie -Original Message- From: Guozhang Wang To: users Sent: Tue, 23 Jul 2019 2:40 Subject: Re: Truncation Hi Jamie, The most relevant materials I can think of would be in KIP-101: https://cwiki.apache.org/confluence/display/KAFK