[
https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040449#comment-16040449
]
Magnus Reftel commented on KAFKA-5390:
--------------------------------------
Our records are typically small, but potentially huge. Doing manual batching
implicitly by ensuring that {{batch.size}} is greater than a fixed number of
worst-case records means tuning for the worst case instead of the expected
case. Also, the calculation still depends on protocol framing etc., which feels
wrong to assume things about outside of the client library. Setting
{{linger.ms}} to an arbitrary, large value in order to avoid batches being sent
too early also seems brittle - would adding a third way of deciding when to
send a batch in addition to the linger time and buffer size be regarded as an
API-breaking change?
I'd much rather do batching explicitly, e.g. by a sequence of calls to a
{{boolean addBatched(ProducerRecord, Callback}} method which indicates whether
there was buffer space for the records, followed by a {{void sendBatch()}}.
Would a patch adding that functionality be of interest?
> First records in batch rejected but others accepted when rolling log
> --------------------------------------------------------------------
>
> Key: KAFKA-5390
> URL: https://issues.apache.org/jira/browse/KAFKA-5390
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.2.1
> Reporter: Magnus Reftel
> Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz
>
>
> When sending a sequence of records in a batch right when the broker needs to
> roll a new segment, it's possible for the first few records to fail, while
> other records in the batch are accepted. If records have dependencies on
> earlier records, e.g. in the case of a sequence of events in an event-sourced
> system, then a producer cannot use the batching functionality, since it then
> risks consumers receiving a record without first receiving the records it
> depends on.
> See attached testcase (kafka-sequence.tar.gz).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
