Hello everyone, Do we have any update on the subject in hand? Every time we are trying to bump our Kafka streams version in production past 1.0.2 we are getting the UNKNOWN_PRODUCER_ID WARN (I've tried 1.1.0, 1.1.1, 2.0.0 & now 2.0.1). In our case, we are definitely NOT under low traffic, but I think this is already known after reading https://issues.apache.org/jira/browse/KAFKA-7190
On Mon, Jul 23, 2018 at 4:56 PM lambdaliu(刘少波) <lambda...@tencent.com> wrote: > Hello Guozhang, Bill > > Thanks much for your explanation. I assigned this JIRA issue to myself > and will > try to fix it. > > Regards, > Lambdaliu(Shaobo Liu) > > On 2018-07-21 at 02:24 Bill Bejeck <b...@confluent.io> wrote: > > >Created JIRA https://issues.apache.org/jira/browse/KAFKA-7190 > > > >-Bill > > > >On Fri, Jul 20, 2018 at 1:26 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > >> Hello Shaobo, > >> > >> The situation you observed is a valid case: when a streams application > has > >> very low traffic (like the quickstart example, where you manually type > one > >> message at a time), then it is possible that consumer purging would > delete > >> even the last message sent by this producer (i.e. all the messages sent > by > >> this producer have been consumed and committed), and as a result the > broker > >> would delete that producer's ID. The next time when this producer tries > to > >> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case > >> this error is retriable: the producer would just get a new producer id > and > >> retries, and then this time it will succeed. So the results you > observed is > >> still correct. > >> > >> > >> We can, probably, improve this situation either in broker side or > streams > >> client side: on broker side, we can consider delaying the deletion of > the > >> producer ID for a while; on streams client side, we can consider > purging in > >> a bit conservative manner (but it is still a bit tricky, since multiple > >> producers may be sending to the same inner topic, so just leaving the > last > >> N messages to not be purged may not be safe still). > >> > >> > >> Bill, could we create a JIRA for this? > >> > >> > >> Guozhang > >> > >> > >> On Thu, Jul 19, 2018 at 7:55 AM, Bill Bejeck <b...@confluent.io> wrote: > >> > >> > Hi > >> > > >> > Thanks for reporting this. Just off the top of my head, I'm thinking > it > >> > may have to do with using a console producer, but I'll have to take a > >> > deeper look. > >> > > >> > Thanks, > >> > Bill > >> > > >> > On Thu, Jul 19, 2018 at 9:59 AM lambdaliu(刘少波) <lambda...@tencent.com > > > >> > wrote: > >> > > >> > > Hi, > >> > > > >> > > I test the Kafka Streams WordCount demo follow the steps described > in > >> > > http://kafka.apache.org/11/documentation/streams/quickstart with > >> change > >> > > the processing.guarantee property to EXACTLY_ONCE. > >> > > > >> > > And seeing the following WARN message in streams demo app logs: > >> > > [2018-07-18 21:08:03,510] WARN The configuration 'admin.retries' was > >> > > supplied but isn't a known config. > >> > > (org.apache.kafka.clients.consumer.ConsumerConfig) > >> > > [2018-07-18 21:11:29,218] WARN [Producer > >> > > clientId=apache-wordcount-2a671de0-d2b7-404f-bfe8- > >> > 9e8cad5008d4-StreamThread-1-0_0-producer, > >> > > transactionalId=apache-wordcount-0_0] Got error produce response > with > >> > > correlation id 15 on topic-partition > >> > > > >> apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0, > >> > > retrying (2147483646 <(214)%20748-3646> attempts left). Error: > UNKNOWN_PRODUCER_ID > >> > > (org.apache.kafka.clients.producer.internals.Sender) > >> > > [2018-07-18 21:15:04,092] WARN [Producer > >> > > clientId=apache-wordcount-2a671de0-d2b7-404f-bfe8- > >> > 9e8cad5008d4-StreamThread-1-0_0-producer, > >> > > transactionalId=apache-wordcount-0_0] Got error produce response > with > >> > > correlation id 21 on topic-partition > >> > > > >> apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0, > >> > > retrying (2147483646 <(214)%20748-3646> attempts left). Error: > UNKNOWN_PRODUCER_ID > >> > > (org.apache.kafka.clients.producer.internals.Sender) > >> > > > >> > > There are also some ERROR message in the broker logs: > >> > > [2018-07-18 21:10:16,463] INFO Updated PartitionLeaderEpoch. New: > >> > > {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} for Partition: > >> > > > >> apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0. > >> > > Cache now contains 0 entries. > (kafka.server.epoch.LeaderEpochFileCache) > >> > > [2018-07-18 21:10:16,965] INFO [Log > >> > > partition=apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE- > >> > 0000000003-repartition-0, > >> > > dir=/tmp/kafka-logs0] Incrementing log start offset to 5 > >> (kafka.log.Log) > >> > > [2018-07-18 21:10:16,966] INFO Cleared earliest 0 entries from epoch > >> > cache > >> > > based on passed offset 5 leaving 1 in EpochFile for partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.epoch.LeaderEpochFileCache) > >> > > [2018-07-18 21:11:29,217] ERROR [ReplicaManager broker=0] Error > >> > processing > >> > > append operation on partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.ReplicaManager) > >> > > org.apache.kafka.common.errors.UnknownProducerIdException: Found no > >> > record > >> > > of producerId=5000 on the broker. It is possible that the last > message > >> > with > >> > > the producerId=5000 has been removed due to hitting the retention > >> limit. > >> > > [2018-07-18 21:11:29,331] INFO [Log > >> > > partition=apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE- > >> > 0000000003-repartition-0, > >> > > dir=/tmp/kafka-logs0] Incrementing log start offset to 9 > >> (kafka.log.Log) > >> > > [2018-07-18 21:11:29,332] INFO Cleared earliest 0 entries from epoch > >> > cache > >> > > based on passed offset 9 leaving 1 in EpochFile for partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.epoch.LeaderEpochFileCache) > >> > > [2018-07-18 21:15:04,091] ERROR [ReplicaManager broker=0] Error > >> > processing > >> > > append operation on partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.ReplicaManager) > >> > > org.apache.kafka.common.errors.UnknownProducerIdException: Found no > >> > record > >> > > of producerId=5000 on the broker. It is possible that the last > message > >> > with > >> > > the producerId=5000 has been removed due to hitting the retention > >> limit. > >> > > [2018-07-18 21:15:04,204] INFO [Log > >> > > partition=apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE- > >> > 0000000003-repartition-0, > >> > > dir=/tmp/kafka-logs0] Incrementing log start offset to 13 > >> (kafka.log.Log) > >> > > [2018-07-18 21:15:04,205] INFO Cleared earliest 0 entries from epoch > >> > cache > >> > > based on passed offset 13 leaving 1 in EpochFile for partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.epoch.LeaderEpochFileCache) > >> > > > >> > > I found the outputs of the WordCount app is correct. But each time I > >> send > >> > > a line to streams-wordcount-input, the Streams App throw a new > >> > > UNKNOWN_PRODUCER_ID error, and Broker also throw a new > >> > > UnknownProducerIdException. > >> > > The broker version I use is 1.1.0. Have anyone ecountered this > problem > >> > > before or can give me any hints about what might be causing this > >> > behaviour? > >> > > > >> > > Thanks, > >> > > lambdaliu > >> > > > >> > > >> > >> > >> > >> -- > >> -- Guozhang > >> > > >Created JIRA https://issues.apache.org/jira/browse/KAFKA-7190 > > > >-Bill > > > >On Fri, Jul 20, 2018 at 1:26 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > >> Hello Shaobo, > >> > >> The situation you observed is a valid case: when a streams application > has > >> very low traffic (like the quickstart example, where you manually type > one > >> message at a time), then it is possible that consumer purging would > delete > >> even the last message sent by this producer (i.e. all the messages sent > by > >> this producer have been consumed and committed), and as a result the > broker > >> would delete that producer's ID. The next time when this producer tries > to > >> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case > >> this error is retriable: the producer would just get a new producer id > and > >> retries, and then this time it will succeed. So the results you > observed is > >> still correct. > >> > >> > >> We can, probably, improve this situation either in broker side or > streams > >> client side: on broker side, we can consider delaying the deletion of > the > >> producer ID for a while; on streams client side, we can consider > purging in > >> a bit conservative manner (but it is still a bit tricky, since multiple > >> producers may be sending to the same inner topic, so just leaving the > last > >> N messages to not be purged may not be safe still). > >> > >> > >> Bill, could we create a JIRA for this? > >> > >> > >> Guozhang > >> > >> > >> On Thu, Jul 19, 2018 at 7:55 AM, Bill Bejeck <b...@confluent.io> wrote: > >> > >> > Hi > >> > > >> > Thanks for reporting this. Just off the top of my head, I'm thinking > it > >> > may have to do with using a console producer, but I'll have to take a > >> > deeper look. > >> > > >> > Thanks, > >> > Bill > >> > > >> > On Thu, Jul 19, 2018 at 9:59 AM lambdaliu(刘少波) <lambda...@tencent.com > > > >> > wrote: > >> > > >> > > Hi, > >> > > > >> > > I test the Kafka Streams WordCount demo follow the steps described > in > >> > > http://kafka.apache.org/11/documentation/streams/quickstart with > >> change > >> > > the processing.guarantee property to EXACTLY_ONCE. > >> > > > >> > > And seeing the following WARN message in streams demo app logs: > >> > > [2018-07-18 21:08:03,510] WARN The configuration 'admin.retries' was > >> > > supplied but isn't a known config. > >> > > (org.apache.kafka.clients.consumer.ConsumerConfig) > >> > > [2018-07-18 21:11:29,218] WARN [Producer > >> > > clientId=apache-wordcount-2a671de0-d2b7-404f-bfe8- > >> > 9e8cad5008d4-StreamThread-1-0_0-producer, > >> > > transactionalId=apache-wordcount-0_0] Got error produce response > with > >> > > correlation id 15 on topic-partition > >> > > > >> apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0, > >> > > retrying (2147483646 <(214)%20748-3646> attempts left). Error: > UNKNOWN_PRODUCER_ID > >> > > (org.apache.kafka.clients.producer.internals.Sender) > >> > > [2018-07-18 21:15:04,092] WARN [Producer > >> > > clientId=apache-wordcount-2a671de0-d2b7-404f-bfe8- > >> > 9e8cad5008d4-StreamThread-1-0_0-producer, > >> > > transactionalId=apache-wordcount-0_0] Got error produce response > with > >> > > correlation id 21 on topic-partition > >> > > > >> apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0, > >> > > retrying (2147483646 <(214)%20748-3646> attempts left). Error: > UNKNOWN_PRODUCER_ID > >> > > (org.apache.kafka.clients.producer.internals.Sender) > >> > > > >> > > There are also some ERROR message in the broker logs: > >> > > [2018-07-18 21:10:16,463] INFO Updated PartitionLeaderEpoch. New: > >> > > {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} for Partition: > >> > > > >> apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0. > >> > > Cache now contains 0 entries. > (kafka.server.epoch.LeaderEpochFileCache) > >> > > [2018-07-18 21:10:16,965] INFO [Log > >> > > partition=apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE- > >> > 0000000003-repartition-0, > >> > > dir=/tmp/kafka-logs0] Incrementing log start offset to 5 > >> (kafka.log.Log) > >> > > [2018-07-18 21:10:16,966] INFO Cleared earliest 0 entries from epoch > >> > cache > >> > > based on passed offset 5 leaving 1 in EpochFile for partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.epoch.LeaderEpochFileCache) > >> > > [2018-07-18 21:11:29,217] ERROR [ReplicaManager broker=0] Error > >> > processing > >> > > append operation on partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.ReplicaManager) > >> > > org.apache.kafka.common.errors.UnknownProducerIdException: Found no > >> > record > >> > > of producerId=5000 on the broker. It is possible that the last > message > >> > with > >> > > the producerId=5000 has been removed due to hitting the retention > >> limit. > >> > > [2018-07-18 21:11:29,331] INFO [Log > >> > > partition=apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE- > >> > 0000000003-repartition-0, > >> > > dir=/tmp/kafka-logs0] Incrementing log start offset to 9 > >> (kafka.log.Log) > >> > > [2018-07-18 21:11:29,332] INFO Cleared earliest 0 entries from epoch > >> > cache > >> > > based on passed offset 9 leaving 1 in EpochFile for partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.epoch.LeaderEpochFileCache) > >> > > [2018-07-18 21:15:04,091] ERROR [ReplicaManager broker=0] Error > >> > processing > >> > > append operation on partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.ReplicaManager) > >> > > org.apache.kafka.common.errors.UnknownProducerIdException: Found no > >> > record > >> > > of producerId=5000 on the broker. It is possible that the last > message > >> > with > >> > > the producerId=5000 has been removed due to hitting the retention > >> limit. > >> > > [2018-07-18 21:15:04,204] INFO [Log > >> > > partition=apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE- > >> > 0000000003-repartition-0, > >> > > dir=/tmp/kafka-logs0] Incrementing log start offset to 13 > >> (kafka.log.Log) > >> > > [2018-07-18 21:15:04,205] INFO Cleared earliest 0 entries from epoch > >> > cache > >> > > based on passed offset 13 leaving 1 in EpochFile for partition > >> > > > apache-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition-0 > >> > > (kafka.server.epoch.LeaderEpochFileCache) > >> > > > >> > > I found the outputs of the WordCount app is correct. But each time I > >> send > >> > > a line to streams-wordcount-input, the Streams App throw a new > >> > > UNKNOWN_PRODUCER_ID error, and Broker also throw a new > >> > > UnknownProducerIdException. > >> > > The broker version I use is 1.1.0. Have anyone ecountered this > problem > >> > > before or can give me any hints about what might be causing this > >> > behaviour? > >> > > > >> > > Thanks, > >> > > lambdaliu > >> > > > >> > > >> > >> > >> > >> -- > >> -- Guozhang > >> -- Nitay Kufert Backend Developer [image: ironSource] <http://www.ironsrc.com/> email nita...@ironsrc.com mobile +972-54-5480021 fax +972-77-5448273 skype nitay.kufert.ssa 9 Ehad Ha'am st. Tel- Aviv ironsrc.com <http://www.ironsrc.com/> [image: linkedin] <https://www.linkedin.com/company/ironsource>[image: twitter] <https://twitter.com/ironsource>[image: facebook] <https://www.facebook.com/ironSource>[image: googleplus] <https://plus.google.com/+ironsrc> This email (including any attachments) is for the sole use of the intended recipient and may contain confidential information which may be protected by legal privilege. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication and/or its content is strictly prohibited. If you are not the intended recipient, please immediately notify us by reply email or by telephone, delete this email and destroy any copies. Thank you.
