Hey Guozhang, I'm late to the game, again, but if it is not too late I'd recommend that instead of having a single error code for all failed records, instead provide per-record error codes by having an array of [relative_offset, error_code], where they're both varint-encoded for space-efficiency.
Regards, Magnus just a couple of months late. Den ons 17 juli 2019 kl 00:18 skrev Guozhang Wang <wangg...@gmail.com>: > Hello folks, > > I've updated the wiki a bit more and here are the changes: > > 1. *InvalidRecordException* would inherit from ApiException (which is > already non-retriable), not KafkaException. > > 2. Explained how to set the error codes if there are multiple error > scenarios on the broker-side. > > > I'm going to start the voting thread soon. > > Guozhang > > > On Thu, Jun 20, 2019 at 4:24 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Jun, > > > > Thanks for your comments. > > > > 1. Yeah I think APIException would not make a distinct call here anymore, > > and what really matters is the RetriableException. Updated the wiki. > > > > 2. Makes sense. Updated the wiki. > > > > 3. My current thoughts is to return the first ever hit error for that > > partition, and also the encoded error_records would only includes the > > relative offsets of records that hitting that exact error as well. > > > > > > Guozhang > > > > On Wed, Jun 12, 2019 at 3:38 PM Jun Rao <j...@confluent.io> wrote: > > > >> Hi, Guozhang, > >> > >> Thanks for the KIP. A few comments below. > >> > >> 1. "If the error_records is not empty and the error code is not API > >> exception and is not retriable, still retry by creating a new batch ". > >> InvalidTimestampException > >> is an ApiException. It seems we should still retry the non-error records > >> in > >> the batch. > >> > >> 2. error_records => [INT64] : Since we don't have more than 2 billion > >> messages per batch, we can just use INT32. It would also be useful to > >> describe what those numbers are. I guess they are the relative offset in > >> the batch? > >> > >> 3. It's possible that a batch of records hit more than one type of error > >> for different records, which error code and error message should the > >> server > >> return to the client? > >> > >> Jun > >> > >> On Sat, May 11, 2019 at 12:44 PM Guozhang Wang <wangg...@gmail.com> > >> wrote: > >> > >> > Hello everyone, > >> > > >> > I'd like to start a discussion thread on this newly created KIP to > >> improve > >> > error communication and handling for producer response: > >> > > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-467%3A+Augment+ProduceResponse+error+messaging+for+specific+culprit+records > >> > > >> > Thanks, > >> > -- > >> > -- Guozhang > >> > > >> > > > > > > -- > > -- Guozhang > > > > > -- > -- Guozhang >
