Hi TaiJuWu, I find this adding lot's of complexity and I am still not convinced by the added value. IMO creating a producer instance per ack level is not problematic and the behavior is clear for developers. What would be the added value of the proposed change ?
Regards, On Wed, Nov 6, 2024 at 7:50 AM TaiJu Wu <tjwu1...@gmail.com> wrote: > Hi Fred and Greg, > > Thanks for your feedback and it really not straightforward but interesting! > There are some behavior I expect. > > The current producer uses the *RecordAccumulator* to gather records, and > the sender thread sends them in batches. We can track each record’s > acknowledgment setting as it appends to the *RecordAccumulator*, allowing > the *sender *to group batches by acknowledgment levels and topicPartition > when processing. > > Regarding the statement, "Callbacks for records being sent to the same > partition are guaranteed to execute in order," this is ensured when > *max.inflight.request > *is set to 1. We can send records with different acknowledgment levels in > the order of acks-0, acks=1, acks=-1. Since we need to send batches with > different acknowledgment levels batches to the broker, the callback will > execute after each request is completed. > > In response to, "If so, are low-acks records subject to head-of-line > blocking from high-acks records?," I believe an additional configuration is > necessary to control this behavior. We could allow records to be either > sync or async, though the callback would still execute after each batch > with varying acknowledgment levels completes. To measure behavior across > acknowledgment levels, we could also include acks in *ProducerIntercepor*. > > Furthermore, before this KIP, a producer could only include one acks level > so sequence is premised. However, with this change, we can *ONLY* guarantee > the sequence within records of the same acknowledgment level because we may > send up to three separate requests to brokers. > Best, > TaiJuWu > > > TaiJu Wu <tjwu1...@gmail.com> 於 2024年11月6日 週三 上午10:01寫道： > > > Hi Fred and Greg, > > > > Apologies for the delayed response. > > Yes, you’re correct. > > I’ll outline the behavior I expect. > > > > Thanks for your feedback! > > > > Best, > > TaiJuWu > > > > > > Greg Harris <greg.har...@aiven.io.invalid> 於 2024年11月6日 週三 上午9:48寫道： > > > >> Hi TaiJuWu, > >> > >> Thanks for the KIP! > >> > >> Can you explain in the KIP about the behavior when the number of acks is > >> different for individual records? I think the current description using > >> the > >> word "straightforward" does little to explain that, and may actually be > >> hiding some complexity. > >> > >> For example, the send() javadoc contains this: "Callbacks for records > >> being > >> sent to the same partition are guaranteed to execute in order." Is this > >> still true when acks vary for records within the same partition? > >> If so, are low-acks records subject to head-of-line-blocking from > >> high-acks > >> records? It seems to me that this feature is useful when acks is > specified > >> per-topic, but introduces a lot of edge cases that are underspecified. > >> > >> Thanks, > >> Greg > >> > >> > >> On Tue, Nov 5, 2024 at 4:52 PM TaiJu Wu <tjwu1...@gmail.com> wrote: > >> > >> > Hi Chia-Ping, > >> > > >> > Thanks for your feedback. > >> > I have updated KIP based on your suggestions. > >> > > >> > Best, > >> > Stanley > >> > > >> > Chia-Ping Tsai <chia7...@apache.org> 於 2024年11月5日 週二 下午4:41寫道： > >> > > >> > > hi TaiJuWu, > >> > > > >> > > Q0: Could you please add getter (Short acks()) to "public interface" > >> > > section? > >> > > > >> > > Q1: Could you please add RPC json reference to prove "been available > >> at > >> > > the RPC-level," > >> > > > >> > > Q2: Could you please add link to producer docs to prove "share a > >> single > >> > > producer instance across multiple threads" > >> > > > >> > > Thanks, > >> > > Chia-Ping > >> > > > >> > > On 2024/11/05 01:28:36 吳岱儒 wrote: > >> > > > Hi all, > >> > > > > >> > > > I open a KIP-1107: Adding record-level acks for producers > >> > > > < > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers > >> > > > > >> > > > to > >> > > > reduce the limitation associated with reusing KafkaProducer. > >> > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers > >> > > > > >> > > > Feedbacks and suggestions are welcome. > >> > > > > >> > > > Thanks, > >> > > > TaiJuWu > >> > > > > >> > > > >> > > >> > > >
