Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

And a question regarding transactions: what is the effect of a failing record on a transaction, do we rely on the application to abort the transaction as needed? This is a behavioural change since today an application can rely on the underlying idempotent producer to be sure that all messages in a

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hey Guozhang, I'm late to the game, again, but if it is not too late I'd recommend that instead of having a single error code for all failed records, instead provide per-record error codes by having an array of [relative_offset, error_code], where they're both varint-encoded for space-efficiency.

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hello folks, I've updated the wiki a bit more and here are the changes: 1. *InvalidRecordException* would inherit from ApiException (which is already non-retriable), not KafkaException. 2. Explained how to set the error codes if there are multiple error scenarios on the broker-side. I'm going

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hi Jun, Thanks for your comments. 1. Yeah I think APIException would not make a distinct call here anymore, and what really matters is the RetriableException. Updated the wiki. 2. Makes sense. Updated the wiki. 3. My current thoughts is to return the first ever hit error for that partition, and

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hi, Guozhang, Thanks for the KIP. A few comments below. 1. "If the error_records is not empty and the error code is not API exception and is not retriable, still retry by creating a new batch ". InvalidTimestampException is an ApiException. It seems we should still retry the non-error records in

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hi Kamal, I intend to scope KIP-467 for producer <--> brokers interaction only, so that we can have a small enough one to discuss and tackle. I've just read about KIP-334 as well and I feel it's still better to be addressed independently by itself. LMK what you think. Guozhang On Mon, Jun 10,

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

This KIP is inline with KIP-334 in which it's proposed that the consumer will throw FaultyRecordException with corrupt/in-operative record offset and topic-partition on encountering the invalid record. In this KIP proposed

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Bump up this thread again for whoever's interested. On Sat, May 11, 2019 at 12:34 PM Guozhang Wang wrote: > Hello everyone, > > I'd like to start a discussion thread on this newly created KIP to improve > error communication and handling for producer response: > > > https://cwiki.apache.org/conf

[DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hello everyone, I'd like to start a discussion thread on this newly created KIP to improve error communication and handling for producer response: https://cwiki.apache.org/confluence/display/KAFKA/KIP-467%3A+Augment+ProduceResponse+error+messaging+for+specific+culprit+records Thanks, -- -- Guoz