Hi TaiJuWu, Thanks for the KIP.
The motivation is not clear to me. Could you please elaborate a bit more on it? My concern is that it adds a lot of complexity and the added value seems to be low. Moreover, it will make reasoning about an application from the server side more difficult because we can no longer assume that it writes with the ack based on the config. Another issue is about the batching, how do you plan to handle batches mixing records with different acks? An alternative approach may be to define the ack per topic. We could even think about defining it on the server side as a topic config. I haven't really thought about it but it may be something to explore a bit more. Best, David On Wed, Nov 13, 2024 at 3:56 PM Frédérik Rouleau <froul...@confluent.io.invalid> wrote: > Hi TaiJuWu, > > I find this adding lot's of complexity and I am still not convinced by the > added value. IMO creating a producer instance per ack level is not > problematic and the behavior is clear for developers. What would be the > added value of the proposed change ? > > Regards, > > > On Wed, Nov 6, 2024 at 7:50 AM TaiJu Wu <tjwu1...@gmail.com> wrote: > > > Hi Fred and Greg, > > > > Thanks for your feedback and it really not straightforward but > interesting! > > There are some behavior I expect. > > > > The current producer uses the *RecordAccumulator* to gather records, and > > the sender thread sends them in batches. We can track each record’s > > acknowledgment setting as it appends to the *RecordAccumulator*, allowing > > the *sender *to group batches by acknowledgment levels and topicPartition > > when processing. > > > > Regarding the statement, "Callbacks for records being sent to the same > > partition are guaranteed to execute in order," this is ensured when > > *max.inflight.request > > *is set to 1. We can send records with different acknowledgment levels in > > the order of acks-0, acks=1, acks=-1. Since we need to send batches with > > different acknowledgment levels batches to the broker, the callback will > > execute after each request is completed. > > > > In response to, "If so, are low-acks records subject to head-of-line > > blocking from high-acks records?," I believe an additional configuration > is > > necessary to control this behavior. We could allow records to be either > > sync or async, though the callback would still execute after each batch > > with varying acknowledgment levels completes. To measure behavior across > > acknowledgment levels, we could also include acks in > *ProducerIntercepor*. > > > > Furthermore, before this KIP, a producer could only include one acks > level > > so sequence is premised. However, with this change, we can *ONLY* > guarantee > > the sequence within records of the same acknowledgment level because we > may > > send up to three separate requests to brokers. > > Best, > > TaiJuWu > > > > > > TaiJu Wu <tjwu1...@gmail.com> 於 2024年11月6日 週三 上午10:01寫道： > > > > > Hi Fred and Greg, > > > > > > Apologies for the delayed response. > > > Yes, you’re correct. > > > I’ll outline the behavior I expect. > > > > > > Thanks for your feedback! > > > > > > Best, > > > TaiJuWu > > > > > > > > > Greg Harris <greg.har...@aiven.io.invalid> 於 2024年11月6日 週三 上午9:48寫道： > > > > > >> Hi TaiJuWu, > > >> > > >> Thanks for the KIP! > > >> > > >> Can you explain in the KIP about the behavior when the number of acks > is > > >> different for individual records? I think the current description > using > > >> the > > >> word "straightforward" does little to explain that, and may actually > be > > >> hiding some complexity. > > >> > > >> For example, the send() javadoc contains this: "Callbacks for records > > >> being > > >> sent to the same partition are guaranteed to execute in order." Is > this > > >> still true when acks vary for records within the same partition? > > >> If so, are low-acks records subject to head-of-line-blocking from > > >> high-acks > > >> records? It seems to me that this feature is useful when acks is > > specified > > >> per-topic, but introduces a lot of edge cases that are underspecified. > > >> > > >> Thanks, > > >> Greg > > >> > > >> > > >> On Tue, Nov 5, 2024 at 4:52 PM TaiJu Wu <tjwu1...@gmail.com> wrote: > > >> > > >> > Hi Chia-Ping, > > >> > > > >> > Thanks for your feedback. > > >> > I have updated KIP based on your suggestions. > > >> > > > >> > Best, > > >> > Stanley > > >> > > > >> > Chia-Ping Tsai <chia7...@apache.org> 於 2024年11月5日 週二 下午4:41寫道： > > >> > > > >> > > hi TaiJuWu, > > >> > > > > >> > > Q0: Could you please add getter (Short acks()) to "public > interface" > > >> > > section? > > >> > > > > >> > > Q1: Could you please add RPC json reference to prove "been > available > > >> at > > >> > > the RPC-level," > > >> > > > > >> > > Q2: Could you please add link to producer docs to prove "share a > > >> single > > >> > > producer instance across multiple threads" > > >> > > > > >> > > Thanks, > > >> > > Chia-Ping > > >> > > > > >> > > On 2024/11/05 01:28:36 吳岱儒 wrote: > > >> > > > Hi all, > > >> > > > > > >> > > > I open a KIP-1107: Adding record-level acks for producers > > >> > > > < > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers > > >> > > > > > >> > > > to > > >> > > > reduce the limitation associated with reusing KafkaProducer. > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers > > >> > > > > > >> > > > Feedbacks and suggestions are welcome. > > >> > > > > > >> > > > Thanks, > > >> > > > TaiJuWu > > >> > > > > > >> > > > > >> > > > >> > > > > > >
