Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi Chia-Ping,

Thanks for your feedback.
I have updated KIP based on your suggestions.

Best,
Stanley

Chia-Ping Tsai  於 2024年11月5日 週二 下午4:41寫道：

> hi TaiJuWu,
>
> Q0: Could you please add getter (Short acks()) to "public interface"
> section?
>
> Q1: Could you please add RPC json reference to prove "been available at
> the RPC-level,"
>
> Q2: Could you please add link to producer docs to prove "share a single
> producer instance across multiple threads"
>
> Thanks,
> Chia-Ping
>
> On 2024/11/05 01:28:36 吳岱儒 wrote:
> > Hi all,
> >
> > I open a KIP-1107: Adding record-level acks for producers
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers
> >
> > to
> > reduce the limitation associated with reusing KafkaProducer.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers
> >
> > Feedbacks and suggestions are welcome.
> >
> > Thanks,
> > TaiJuWu
> >
>

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi  Fred and Greg,

Apologies for the delayed response.
Yes, you’re correct.
I’ll outline the behavior I expect.

Thanks for your feedback!

Best,
TaiJuWu


Greg Harris  於 2024年11月6日 週三 上午9:48寫道：

> Hi TaiJuWu,
>
> Thanks for the KIP!
>
> Can you explain in the KIP about the behavior when the number of acks is
> different for individual records? I think the current description using the
> word "straightforward" does little to explain that, and may actually be
> hiding some complexity.
>
> For example, the send() javadoc contains this: "Callbacks for records being
> sent to the same partition are guaranteed to execute in order." Is this
> still true when acks vary for records within the same partition?
> If so, are low-acks records subject to head-of-line-blocking from high-acks
> records? It seems to me that this feature is useful when acks is specified
> per-topic, but introduces a lot of edge cases that are underspecified.
>
> Thanks,
> Greg
>
>
> On Tue, Nov 5, 2024 at 4:52 PM TaiJu Wu  wrote:
>
> > Hi Chia-Ping,
> >
> > Thanks for your feedback.
> > I have updated KIP based on your suggestions.
> >
> > Best,
> > Stanley
> >
> > Chia-Ping Tsai  於 2024年11月5日 週二 下午4:41寫道：
> >
> > > hi TaiJuWu,
> > >
> > > Q0: Could you please add getter (Short acks()) to "public interface"
> > > section?
> > >
> > > Q1: Could you please add RPC json reference to prove "been available at
> > > the RPC-level,"
> > >
> > > Q2: Could you please add link to producer docs to prove "share a single
> > > producer instance across multiple threads"
> > >
> > > Thanks,
> > > Chia-Ping
> > >
> > > On 2024/11/05 01:28:36 吳岱儒 wrote:
> > > > Hi all,
> > > >
> > > > I open a KIP-1107: Adding record-level acks for producers
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers
> > > >
> > > > to
> > > > reduce the limitation associated with reusing KafkaProducer.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers
> > > >
> > > > Feedbacks and suggestions are welcome.
> > > >
> > > > Thanks,
> > > > TaiJuWu
> > > >
> > >
> >
>

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi Fred and Greg,

Thanks for your feedback and it really not straightforward but interesting!
There are some behavior I expect.

The current producer uses the *RecordAccumulator* to gather records, and
the sender thread sends them in batches. We can track each record’s
acknowledgment setting as it appends to the *RecordAccumulator*, allowing
the *sender *to group batches by acknowledgment levels and topicPartition
when processing.

Regarding the statement, "Callbacks for records being sent to the same
partition are guaranteed to execute in order," this is ensured when
*max.inflight.request
*is set to 1. We can send records with different acknowledgment levels in
the order of acks-0, acks=1, acks=-1. Since we need to send batches with
different acknowledgment levels batches to the broker, the callback will
execute after each request is completed.

In response to, "If so, are low-acks records subject to head-of-line
blocking from high-acks records?," I believe an additional configuration is
necessary to control this behavior. We could allow records to be either
sync or async, though the callback would still execute after each batch
with varying acknowledgment levels completes. To measure behavior across
acknowledgment levels, we could also include acks in *ProducerIntercepor*.

Furthermore, before this KIP, a producer could only include one acks level
so sequence is premised. However, with this change, we can *ONLY* guarantee
the sequence within records of the same acknowledgment level because we may
send up to three separate requests to brokers.
Best,
TaiJuWu


TaiJu Wu  於 2024年11月6日 週三 上午10:01寫道：

> Hi  Fred and Greg,
>
> Apologies for the delayed response.
> Yes, you’re correct.
> I’ll outline the behavior I expect.
>
> Thanks for your feedback!
>
> Best,
> TaiJuWu
>
>
> Greg Harris  於 2024年11月6日 週三 上午9:48寫道：
>
>> Hi TaiJuWu,
>>
>> Thanks for the KIP!
>>
>> Can you explain in the KIP about the behavior when the number of acks is
>> different for individual records? I think the current description using
>> the
>> word "straightforward" does little to explain that, and may actually be
>> hiding some complexity.
>>
>> For example, the send() javadoc contains this: "Callbacks for records
>> being
>> sent to the same partition are guaranteed to execute in order." Is this
>> still true when acks vary for records within the same partition?
>> If so, are low-acks records subject to head-of-line-blocking from
>> high-acks
>> records? It seems to me that this feature is useful when acks is specified
>> per-topic, but introduces a lot of edge cases that are underspecified.
>>
>> Thanks,
>> Greg
>>
>>
>> On Tue, Nov 5, 2024 at 4:52 PM TaiJu Wu  wrote:
>>
>> > Hi Chia-Ping,
>> >
>> > Thanks for your feedback.
>> > I have updated KIP based on your suggestions.
>> >
>> > Best,
>> > Stanley
>> >
>> > Chia-Ping Tsai  於 2024年11月5日 週二 下午4:41寫道：
>> >
>> > > hi TaiJuWu,
>> > >
>> > > Q0: Could you please add getter (Short acks()) to "public interface"
>> > > section?
>> > >
>> > > Q1: Could you please add RPC json reference to prove "been available
>> at
>> > > the RPC-level,"
>> > >
>> > > Q2: Could you please add link to producer docs to prove "share a
>> single
>> > > producer instance across multiple threads"
>> > >
>> > > Thanks,
>> > > Chia-Ping
>> > >
>> > > On 2024/11/05 01:28:36 吳岱儒 wrote:
>> > > > Hi all,
>> > > >
>> > > > I open a KIP-1107: Adding record-level acks for producers
>> > > > <
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers
>> > > >
>> > > > to
>> > > > reduce the limitation associated with reusing KafkaProducer.
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+record-level+acks+for+producers
>> > > >
>> > > > Feedbacks and suggestions are welcome.
>> > > >
>> > > > Thanks,
>> > > > TaiJuWu
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi Chia-Ping,

Thanks for your suggestions and feedback.

Q1: I have updated this according your suggestions.
Q2: This is necessary change since there is a assumption about
*RecourdAccumulator
*that all records have same acks(e.g. ProducerConfig.acks) so we need to a
method to distinguish which acks belong to each Batch.

Best,
TaiJuWu

Chia-Ping Tsai  於 2024年11月18日 週一 上午2:17寫道：

> hi TaiJuWu
>
> Q0:
>
> `Format: topic.acks`  the dot is acceptable character in topic naming, so
> maybe we should reverse the format to "acks.${topic}" to get the acks of
> topic easily
>
> Q1: `Return Map> when
> RecordAccumulator#drainBatchesForOneNode is called.`
>
> this is weird to me, as all we need to do is pass `Map to
> `Sender` and make sure `Sender#sendProduceRequest` add correct acks to
> ProduceRequest, right?
>
> Best,
> Chia-Ping
>
>
>
> On 2024/11/15 05:12:33 TaiJu Wu wrote:
> > Hi all,
> >
> > I have updated the contents of this KIP
> > Please take a look and let me know what you think.
> >
> > Thanks,
> > TaiJuWu
> >
> > On Thu, Nov 14, 2024 at 2:21 PM TaiJu Wu  wrote:
> >
> > > Hi all,
> > >
> > > Thanks for your feeback and @Chia-Ping's help.
> > > .
> > > I also agree topic-level acks config is more reasonable and it can
> simply
> > > the story.
> > > When I try implementing record-level acks, I notice I don't have good
> idea
> > > to avoid iterating batches for get partition information (need by
> > > *RecordAccumulator#partitionChanged*).
> > >
> > > Back to the init question how can I handle different acks for batches:
> > > First, we can attach *topic-level acks *to
> *RecordAccumulator#TopicInfo*.
> > > Second,  we can return *Map>* when
> *RecordAccumulator#drainBatchesForOneNode
> > > *is called. In this step, we can propagate acks to *sender*.
> > > Finally, we can get the acks info and group same acks into a
> > > *List>* for a node in *sender#sendProduceRequests*.
> > >
> > > If I missed something or there is any mistake, please let me know.
> > > I will update this KIP later, thank your feedback.
> > >
> > > Best,
> > > TaiJuWu
> > >
> > >
> > > Chia-Ping Tsai  於 2024年11月14日 週四 上午9:46寫道：
> > >
> > >> hi All
> > >>
> > >> This KIP is based on our use case where an edge application with many
> > >> sensors wants to use a single producer to deliver ‘few but varied’
> records
> > >> with different acks settings. The reason for using a single producer
> is to
> > >> minimize resource usage on edge devices with limited hardware
> capabilities.
> > >> Currently, we use a producer pool to handle different acks values,
> which
> > >> requires 3x producer instances. Additionally, this approach creates
> many
> > >> idle producers if a sensor with a specific acks setting has no data
> for a
> > >> while.
> > >>
> > >> I love David’s suggestion since the acks configuration is closely
> related
> > >> to the topic. Maybe we can introduce an optional configuration in the
> > >> producer to define topic-level acks, with the existing acks being the
> > >> default for all topics. This approach is not only simple but also
> easy to
> > >> understand and implement.
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> On 2024/11/13 16:04:24 Andrew Schofield wrote:
> > >> > Hi TaiJuWu,
> > >> > I've been thinking for a while about this KIP before jumping into
> the
> > >> discussion.
> > >> >
> > >> > I'm afraid that I don't think the approach in the KIP is the best,
> > >> given the design
> > >> > of the Kafka protocol in this area. Essentially, each Produce
> request
> > >> contains
> > >> > the acks value at the top level, and may contain records for many
> > >> topics or
> > >> > partitions. My point is that batching occurs at the level of a
> Produce
> > >> request,
> > >> > so changing the acks value between records will require a new
> Produce
> > >> request
> > >> > to be sent. There would likely be an efficiency penalty if this
> feature
> > >> was used
> > >> > heavily with the acks changing record by record.
> > >> >
> > >> > I can see that potentially 

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi all,

I have updated the contents of this KIP
Please take a look and let me know what you think.

Thanks,
TaiJuWu

On Thu, Nov 14, 2024 at 2:21 PM TaiJu Wu  wrote:

> Hi all,
>
> Thanks for your feeback and @Chia-Ping's help.
> .
> I also agree topic-level acks config is more reasonable and it can simply
> the story.
> When I try implementing record-level acks, I notice I don't have good idea
> to avoid iterating batches for get partition information (need by
> *RecordAccumulator#partitionChanged*).
>
> Back to the init question how can I handle different acks for batches:
> First, we can attach *topic-level acks *to *RecordAccumulator#TopicInfo*.
> Second,  we can return *Map>* when 
> *RecordAccumulator#drainBatchesForOneNode
> *is called. In this step, we can propagate acks to *sender*.
> Finally, we can get the acks info and group same acks into a
> *List>* for a node in *sender#sendProduceRequests*.
>
> If I missed something or there is any mistake, please let me know.
> I will update this KIP later, thank your feedback.
>
> Best,
> TaiJuWu
>
>
> Chia-Ping Tsai  於 2024年11月14日 週四 上午9:46寫道：
>
>> hi All
>>
>> This KIP is based on our use case where an edge application with many
>> sensors wants to use a single producer to deliver ‘few but varied’ records
>> with different acks settings. The reason for using a single producer is to
>> minimize resource usage on edge devices with limited hardware capabilities.
>> Currently, we use a producer pool to handle different acks values, which
>> requires 3x producer instances. Additionally, this approach creates many
>> idle producers if a sensor with a specific acks setting has no data for a
>> while.
>>
>> I love David’s suggestion since the acks configuration is closely related
>> to the topic. Maybe we can introduce an optional configuration in the
>> producer to define topic-level acks, with the existing acks being the
>> default for all topics. This approach is not only simple but also easy to
>> understand and implement.
>>
>> Best,
>> Chia-Ping
>>
>> On 2024/11/13 16:04:24 Andrew Schofield wrote:
>> > Hi TaiJuWu,
>> > I've been thinking for a while about this KIP before jumping into the
>> discussion.
>> >
>> > I'm afraid that I don't think the approach in the KIP is the best,
>> given the design
>> > of the Kafka protocol in this area. Essentially, each Produce request
>> contains
>> > the acks value at the top level, and may contain records for many
>> topics or
>> > partitions. My point is that batching occurs at the level of a Produce
>> request,
>> > so changing the acks value between records will require a new Produce
>> request
>> > to be sent. There would likely be an efficiency penalty if this feature
>> was used
>> > heavily with the acks changing record by record.
>> >
>> > I can see that potentially an application might want different ack
>> levels for
>> > different topics, but I would be surprised if they use different ack
>> levels within
>> > the same topic. Maybe David's suggestion of defining the acks per topic
>> > would be enough. What do you think?
>> >
>> > Thanks,
>> > Andrew
>> > 
>> > From: David Jacot 
>> > Sent: 13 November 2024 15:31
>> > To: dev@kafka.apache.org 
>> > Subject: Re: [DISCUSS]KIP-1107: Adding record-level acks for producers
>> >
>> > Hi TaiJuWu,
>> >
>> > Thanks for the KIP.
>> >
>> > The motivation is not clear to me. Could you please elaborate a bit
>> more on
>> > it?
>> >
>> > My concern is that it adds a lot of complexity and the added value
>> seems to
>> > be low. Moreover, it will make reasoning about an application from the
>> > server side more difficult because we can no longer assume that it
>> writes
>> > with the ack based on the config. Another issue is about the batching,
>> how
>> > do you plan to handle batches mixing records with different acks?
>> >
>> > An alternative approach may be to define the ack per topic. We could
>> even
>> > think about defining it on the server side as a topic config. I haven't
>> > really thought about it but it may be something to explore a bit more.
>> >
>> > Best,
>> > David
>> >
>> > On Wed, Nov 13, 2024 at 3:56 PM Frédérik Rouleau
>> >  wrote:
>> >
>> > > Hi TaiJuW

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi all,

Thanks for your feeback and @Chia-Ping's help.
.
I also agree topic-level acks config is more reasonable and it can simply
the story.
When I try implementing record-level acks, I notice I don't have good idea
to avoid iterating batches for get partition information (need by
*RecordAccumulator#partitionChanged*).

Back to the init question how can I handle different acks for batches:
First, we can attach *topic-level acks *to *RecordAccumulator#TopicInfo*.
Second,  we can return *Map>* when
*RecordAccumulator#drainBatchesForOneNode
*is called. In this step, we can propagate acks to *sender*.
Finally, we can get the acks info and group same acks into a
*List>* for a node in *sender#sendProduceRequests*.

If I missed something or there is any mistake, please let me know.
I will update this KIP later, thank your feedback.

Best,
TaiJuWu


Chia-Ping Tsai  於 2024年11月14日 週四 上午9:46寫道：

> hi All
>
> This KIP is based on our use case where an edge application with many
> sensors wants to use a single producer to deliver ‘few but varied’ records
> with different acks settings. The reason for using a single producer is to
> minimize resource usage on edge devices with limited hardware capabilities.
> Currently, we use a producer pool to handle different acks values, which
> requires 3x producer instances. Additionally, this approach creates many
> idle producers if a sensor with a specific acks setting has no data for a
> while.
>
> I love David’s suggestion since the acks configuration is closely related
> to the topic. Maybe we can introduce an optional configuration in the
> producer to define topic-level acks, with the existing acks being the
> default for all topics. This approach is not only simple but also easy to
> understand and implement.
>
> Best,
> Chia-Ping
>
> On 2024/11/13 16:04:24 Andrew Schofield wrote:
> > Hi TaiJuWu,
> > I've been thinking for a while about this KIP before jumping into the
> discussion.
> >
> > I'm afraid that I don't think the approach in the KIP is the best, given
> the design
> > of the Kafka protocol in this area. Essentially, each Produce request
> contains
> > the acks value at the top level, and may contain records for many topics
> or
> > partitions. My point is that batching occurs at the level of a Produce
> request,
> > so changing the acks value between records will require a new Produce
> request
> > to be sent. There would likely be an efficiency penalty if this feature
> was used
> > heavily with the acks changing record by record.
> >
> > I can see that potentially an application might want different ack
> levels for
> > different topics, but I would be surprised if they use different ack
> levels within
> > the same topic. Maybe David's suggestion of defining the acks per topic
> > would be enough. What do you think?
> >
> > Thanks,
> > Andrew
> > 
> > From: David Jacot 
> > Sent: 13 November 2024 15:31
> > To: dev@kafka.apache.org 
> > Subject: Re: [DISCUSS]KIP-1107: Adding record-level acks for producers
> >
> > Hi TaiJuWu,
> >
> > Thanks for the KIP.
> >
> > The motivation is not clear to me. Could you please elaborate a bit more
> on
> > it?
> >
> > My concern is that it adds a lot of complexity and the added value seems
> to
> > be low. Moreover, it will make reasoning about an application from the
> > server side more difficult because we can no longer assume that it writes
> > with the ack based on the config. Another issue is about the batching,
> how
> > do you plan to handle batches mixing records with different acks?
> >
> > An alternative approach may be to define the ack per topic. We could even
> > think about defining it on the server side as a topic config. I haven't
> > really thought about it but it may be something to explore a bit more.
> >
> > Best,
> > David
> >
> > On Wed, Nov 13, 2024 at 3:56 PM Frédérik Rouleau
> >  wrote:
> >
> > > Hi TaiJuWu,
> > >
> > > I find this adding lot's of complexity and I am still not convinced by
> the
> > > added value. IMO creating a producer instance per ack level is not
> > > problematic and the behavior is clear for developers. What would be the
> > > added value of the proposed change ?
> > >
> > > Regards,
> > >
> > >
> > > On Wed, Nov 6, 2024 at 7:50 AM TaiJu Wu  wrote:
> > >
> > > > Hi Fred and Greg,
> > > >
> > > > Thanks for your feedback and it really not straightforward but
> > &

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi Divij and Kirk,

Thanks for your response.
You are right, this change is not straightforward and I apologize for that.

> we haven't answered the question about protocol for ProduceRequest raised
above.
Sorry but which question did I miss, this KIP has been modified from
record-level to topic-level.

> Note that there are disadvantages of "vertically scaling" a producer i.e.
> reusing a producer with multiple threads.
This change is optional so users can choose to adopt it. If they don't want
to use this, it would not have any impact.

> making producer(s) cheap to create is a goal worth pursuing.
> I'd rather attack that in a more direct manner
Thanks for your suggestion, I will investigate this approach simultaneously.

Best,
TaiJuWu

Kirk True  於 2025年1月7日 週二 上午8:34寫道：

> Hi TaiJu!
>
> I will echo the concerns about the likelihood of gotchas arising in an
> effort to work around the existing API and protocol design.
>
> If the central concern is the performance impact and/or resource overhead
> of multiple client instances, I'd rather attack that in a more direct
> manner.
>
> Thanks,
> Kirk
>
> On Fri, Jan 3, 2025, at 8:03 AM, Divij Vaidya wrote:
> > Hey TaiJu
> >
> > I read the latest version of the KIP.
> >
> > I understand the problem you are trying to solve here. But the solution
> > needs more changes than you proposed and hence, is not straightforward.
> As
> > an example, we haven't answered the question about protocol for
> > ProduceRequest raised above. A `ProduceRequest` defines `ack` at a
> request
> > level where the payload consists of records belonging to multiple topics.
> > One way to solve it is to define topic-level `ack` at the server as
> > suggested above in this thread, but wouldn't that require us to
> > remove/deprecate this field?
> >
> > Alternatively, have you tried to explore the option of decreasing the
> > resource footprint of an idle producer so that it is not expensive to
> > create 3x producers?
> > Note that there are disadvantages of "vertically scaling" a producer i.e.
> > reusing a producer with multiple threads. One of the many disadvantages
> is
> > that all requests from the producer will be handled by the same network
> > thread on the broker. If that network thread is busy doing IO for some
> > reason (perhaps reading from disk is slow), then it will impact all other
> > requests from that producer. Hence, making producer(s) cheap to create
> is a
> > goal worth pursuing.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Fri, Jan 3, 2025 at 4:39 AM TaiJu Wu  wrote:
> >
> > > Hello folk,
> > >
> > > This thread is pending for a long time, I want to bump this thread and
> get
> > > more feedback.
> > > Any questions are welcome.
> > >
> > > Best,
> > > TaiJuWu
> > >
> > > TaiJu Wu  於 2024年11月23日 週六 下午9:15寫道：
> > >
> > > > Hi Chia-Ping,
> > > >
> > > > Sorry for late reply and thanks for your feedback to make this KIP
> more
> > > > valuable.
> > > > After initial verification, I think this can do without large
> changes.
> > > >
> > > > I have updated KIP, thanks a lot.
> > > >
> > > > Best,
> > > > TaiJuWu
> > > >
> > > >
> > > > Chia-Ping Tsai  於 2024年11月20日 週三 下午5:06寫道：
> > > >
> > > >> hi TaiJuWu
> > > >>
> > > >> Is there a possibility to extend this KIP to include topic-level
> > > >> compression for the producer? This is another issue that prevents us
> > > from
> > > >> sharing producers across different threads, as it's common to use
> > > different
> > > >> compression types for different topics (data).
> > > >>
> > > >> Best,
> > > >> Chia-Ping
> > > >>
> > > >> On 2024/11/18 08:36:25 TaiJu Wu wrote:
> > > >> > Hi Chia-Ping,
> > > >> >
> > > >> > Thanks for your suggestions and feedback.
> > > >> >
> > > >> > Q1: I have updated this according your suggestions.
> > > >> > Q2: This is necessary change since there is a assumption about
> > > >> > *RecourdAccumulator
> > > >> > *that all records have same acks(e.g. ProducerConfig.acks) so we
> need
> > > >> to a
> > > >> > method to distinguish which ac

Re: [VOTE] KIP-1101: Trigger rebalance on rack topology changes

[TaiJu Wu]

+1 （non-binding）

On Mon, Dec 16, 2024 at 5:01 PM PoAn Yang  wrote:

> Hi All,
>
> I would like to start vote on KIP-1101:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1101%3A+Trigger+rebalance+on+rack+topology+changes
>
> Discussion thread:
> https://lists.apache.org/thread/l8ko353v3nn1blgymsty895x6c98oxlx
>
> Thanks,
> PoAn

Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer Network Thread

[TaiJu Wu]

+1（non-binding)

On Mon, Dec 16, 2024 at 5:41 PM Chia-Ping Tsai  wrote:

> +1 (binding)
>
> 郭骏旺  於 2024年12月16日 週一 上午9:16寫道：
>
> > +1 (non-binding)
> >
> > On Mon, Dec 9, 2024 at 10:33 AM TengYao Chi  wrote:
> >
> > > Hi everyone,
> > >
> > > Based on our discussion
> > > 
> > > regarding
> > > KIP-1118
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1118%3A+Add+Deadlock+Protection+on+Producer+Network+Thread
> > > >,
> > > I believe this KIP is now ready for a vote.
> > >
> > > Sincerely,
> > > TengYao
> > >
> >
>

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hi Chia-Ping,

Sorry for late reply and thanks for your feedback to make this KIP more
valuable.
After initial verification, I think this can do without large changes.

I have updated KIP, thanks a lot.

Best,
TaiJuWu


Chia-Ping Tsai  於 2024年11月20日 週三 下午5:06寫道：

> hi TaiJuWu
>
> Is there a possibility to extend this KIP to include topic-level
> compression for the producer? This is another issue that prevents us from
> sharing producers across different threads, as it's common to use different
> compression types for different topics (data).
>
> Best,
> Chia-Ping
>
> On 2024/11/18 08:36:25 TaiJu Wu wrote:
> > Hi Chia-Ping,
> >
> > Thanks for your suggestions and feedback.
> >
> > Q1: I have updated this according your suggestions.
> > Q2: This is necessary change since there is a assumption about
> > *RecourdAccumulator
> > *that all records have same acks(e.g. ProducerConfig.acks) so we need to
> a
> > method to distinguish which acks belong to each Batch.
> >
> > Best,
> > TaiJuWu
> >
> > Chia-Ping Tsai  於 2024年11月18日 週一 上午2:17寫道：
> >
> > > hi TaiJuWu
> > >
> > > Q0:
> > >
> > > `Format: topic.acks`  the dot is acceptable character in topic naming,
> so
> > > maybe we should reverse the format to "acks.${topic}" to get the acks
> of
> > > topic easily
> > >
> > > Q1: `Return Map> when
> > > RecordAccumulator#drainBatchesForOneNode is called.`
> > >
> > > this is weird to me, as all we need to do is pass `Map to
> > > `Sender` and make sure `Sender#sendProduceRequest` add correct acks to
> > > ProduceRequest, right?
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > >
> > > On 2024/11/15 05:12:33 TaiJu Wu wrote:
> > > > Hi all,
> > > >
> > > > I have updated the contents of this KIP
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thanks,
> > > > TaiJuWu
> > > >
> > > > On Thu, Nov 14, 2024 at 2:21 PM TaiJu Wu  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks for your feeback and @Chia-Ping's help.
> > > > > .
> > > > > I also agree topic-level acks config is more reasonable and it can
> > > simply
> > > > > the story.
> > > > > When I try implementing record-level acks, I notice I don't have
> good
> > > idea
> > > > > to avoid iterating batches for get partition information (need by
> > > > > *RecordAccumulator#partitionChanged*).
> > > > >
> > > > > Back to the init question how can I handle different acks for
> batches:
> > > > > First, we can attach *topic-level acks *to
> > > *RecordAccumulator#TopicInfo*.
> > > > > Second,  we can return *Map>* when
> > > *RecordAccumulator#drainBatchesForOneNode
> > > > > *is called. In this step, we can propagate acks to *sender*.
> > > > > Finally, we can get the acks info and group same acks into a
> > > > > *List>* for a node in *sender#sendProduceRequests*.
> > > > >
> > > > > If I missed something or there is any mistake, please let me know.
> > > > > I will update this KIP later, thank your feedback.
> > > > >
> > > > > Best,
> > > > > TaiJuWu
> > > > >
> > > > >
> > > > > Chia-Ping Tsai  於 2024年11月14日 週四 上午9:46寫道：
> > > > >
> > > > >> hi All
> > > > >>
> > > > >> This KIP is based on our use case where an edge application with
> many
> > > > >> sensors wants to use a single producer to deliver ‘few but varied’
> > > records
> > > > >> with different acks settings. The reason for using a single
> producer
> > > is to
> > > > >> minimize resource usage on edge devices with limited hardware
> > > capabilities.
> > > > >> Currently, we use a producer pool to handle different acks values,
> > > which
> > > > >> requires 3x producer instances. Additionally, this approach
> creates
> > > many
> > > > >> idle producers if a sensor with a specific acks setting has no
> data
> > > for a
> > > > >> while.
> > > > >>
> > > > >> I love David’s suggestion since the a

[VOTE] KIP-1107: Adding topic-level acks and compressions for producers

[TaiJu Wu]

Hello everyone,

Happy New Year!
There is no response about KIP-1107 for a long time so I would like to vote.

Discussion thread:
https://lists.apache.org/thread/v1sm6c59j5wppg1w6t17bvkgqlt3orp7

KIP-1107
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+topic-level+acks+and+compressions+for+producers

Best,
TaiJuWu

Re: [VOTE] KIP-1107: Adding topic-level acks and compressions for producers

[TaiJu Wu]

Hi Chia-Ping

Thanks for your reply.
I will bump discussion thread and discuss it again.

Best,
TaiJuWu

Chia-Ping Tsai  於 2025年1月3日 週五 上午11:33寫道：

> hi TaiJuWu
>
> I propose that we engage in further discussions before initiating the
> voting process. This KIP involves substantial changes to the producer, and
> it's essential to ensure we have thoroughly considered all perspectives.
>
> Best,
> Chia-Ping
>
> On 2025/01/03 03:03:03 TaiJu Wu wrote:
> > Hello everyone,
> >
> > Happy New Year!
> > There is no response about KIP-1107 for a long time so I would like to
> vote.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/v1sm6c59j5wppg1w6t17bvkgqlt3orp7
> >
> > KIP-1107
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+topic-level+acks+and+compressions+for+producers
> >
> > Best,
> > TaiJuWu
> >
>

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hello folk,

This thread is pending for a long time, I want to bump this thread and get
more feedback.
Any questions are welcome.

Best,
TaiJuWu

TaiJu Wu  於 2024年11月23日 週六 下午9:15寫道：

> Hi Chia-Ping,
>
> Sorry for late reply and thanks for your feedback to make this KIP more
> valuable.
> After initial verification, I think this can do without large changes.
>
> I have updated KIP, thanks a lot.
>
> Best,
> TaiJuWu
>
>
> Chia-Ping Tsai  於 2024年11月20日 週三 下午5:06寫道：
>
>> hi TaiJuWu
>>
>> Is there a possibility to extend this KIP to include topic-level
>> compression for the producer? This is another issue that prevents us from
>> sharing producers across different threads, as it's common to use different
>> compression types for different topics (data).
>>
>> Best,
>> Chia-Ping
>>
>> On 2024/11/18 08:36:25 TaiJu Wu wrote:
>> > Hi Chia-Ping,
>> >
>> > Thanks for your suggestions and feedback.
>> >
>> > Q1: I have updated this according your suggestions.
>> > Q2: This is necessary change since there is a assumption about
>> > *RecourdAccumulator
>> > *that all records have same acks(e.g. ProducerConfig.acks) so we need
>> to a
>> > method to distinguish which acks belong to each Batch.
>> >
>> > Best,
>> > TaiJuWu
>> >
>> > Chia-Ping Tsai  於 2024年11月18日 週一 上午2:17寫道：
>> >
>> > > hi TaiJuWu
>> > >
>> > > Q0:
>> > >
>> > > `Format: topic.acks`  the dot is acceptable character in topic
>> naming, so
>> > > maybe we should reverse the format to "acks.${topic}" to get the acks
>> of
>> > > topic easily
>> > >
>> > > Q1: `Return Map> when
>> > > RecordAccumulator#drainBatchesForOneNode is called.`
>> > >
>> > > this is weird to me, as all we need to do is pass `Map
>> to
>> > > `Sender` and make sure `Sender#sendProduceRequest` add correct acks to
>> > > ProduceRequest, right?
>> > >
>> > > Best,
>> > > Chia-Ping
>> > >
>> > >
>> > >
>> > > On 2024/11/15 05:12:33 TaiJu Wu wrote:
>> > > > Hi all,
>> > > >
>> > > > I have updated the contents of this KIP
>> > > > Please take a look and let me know what you think.
>> > > >
>> > > > Thanks,
>> > > > TaiJuWu
>> > > >
>> > > > On Thu, Nov 14, 2024 at 2:21 PM TaiJu Wu 
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Thanks for your feeback and @Chia-Ping's help.
>> > > > > .
>> > > > > I also agree topic-level acks config is more reasonable and it can
>> > > simply
>> > > > > the story.
>> > > > > When I try implementing record-level acks, I notice I don't have
>> good
>> > > idea
>> > > > > to avoid iterating batches for get partition information (need by
>> > > > > *RecordAccumulator#partitionChanged*).
>> > > > >
>> > > > > Back to the init question how can I handle different acks for
>> batches:
>> > > > > First, we can attach *topic-level acks *to
>> > > *RecordAccumulator#TopicInfo*.
>> > > > > Second,  we can return *Map>* when
>> > > *RecordAccumulator#drainBatchesForOneNode
>> > > > > *is called. In this step, we can propagate acks to *sender*.
>> > > > > Finally, we can get the acks info and group same acks into a
>> > > > > *List>* for a node in *sender#sendProduceRequests*.
>> > > > >
>> > > > > If I missed something or there is any mistake, please let me know.
>> > > > > I will update this KIP later, thank your feedback.
>> > > > >
>> > > > > Best,
>> > > > > TaiJuWu
>> > > > >
>> > > > >
>> > > > > Chia-Ping Tsai  於 2024年11月14日 週四 上午9:46寫道：
>> > > > >
>> > > > >> hi All
>> > > > >>
>> > > > >> This KIP is based on our use case where an edge application with
>> many
>> > > > >> sensors wants to use a single producer to deliver ‘few but
>> varied’
>> > > records
>> > > > >> with different acks settings. 

Re: [DISCUSS] KIP-1133: AK Documentation and Website in Markdown

[TaiJu Wu]

Hi Harish,

Thanks for this KIP.

Is it possible to add dev website from trunk for developers?
It could help developer to browser the web without local build and check
recent documentation change quickly.
I think it can improve our documentation a lot since developers can easy to
check it.

Best,
TaiJuWu


Harish Vishwanath  於 2025年3月11日 週二 上午11:53寫道：

> Thanks Chia-Ping. I have updated
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-DevelopmentWorkflow
> >
> the KIP to include the above information.
>
> Regards,
> Harish
>
>
> On Sun, Mar 9, 2025 at 10:09 PM Chia-Ping Tsai  wrote:
>
> > hi Harish
> >
> > > Hugo supports live-reload. The docsy theme has dependencies on golang
> and
> > a
> > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it
> > will be simpler to use docker based workflows to avoid having to install
> > any dependencies locally for development (ref:
> >
> >
> https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server
> > ). We can set up CI pipelines to build and push production images serving
> > the site with nginx.
> >
> > Thank you for the explanation. Would you please consider including this
> > information in the KIP? Documenting the new workflow for reviewing
> > documentation would be very beneficial for future reference. In fact, we
> > recently resolved a bug where the local site's behavior differed from the
> > deployed version on Apache ...
> >
> > Best,
> > Chia-Ping
> >
> > Harish Vishwanath  於 2025年3月10日 週一 下午12:51寫道：
> >
> > > Thank you Chia-Ping.
> > >
> > > Conversion is mostly automated ( https://github.com/hvishwanath/ak2md
> )
> > > and
> > > as such the prototype
> > >  supports all
> > > previous versions. However, not having to support all previous versions
> > > will definitely be simpler.
> > >
> > > Hugo supports live-reload. The docsy theme has dependencies on golang
> > and a
> > > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it
> > > will be simpler to use docker based workflows to avoid having to
> install
> > > any dependencies locally for development (ref:
> > >
> > >
> >
> https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server
> > > ). We can set up CI pipelines to build and push production images
> serving
> > > the site with nginx.
> > >
> > > Regards,
> > > Harish
> > >
> > >
> > > On Sun, Mar 9, 2025 at 9:08 PM Chia-Ping Tsai 
> > wrote:
> > >
> > > > hi Harish
> > > >
> > > > I have to say, this KIP is one of the best ideas I've seen recently,
> > and
> > > I
> > > > agree with Matthias that we don't need to translate any older
> > versions. I
> > > > truly hope this KIP can be included in 4.0.
> > > >
> > > > I have a quick question: what is the recommended procedure for
> testing
> > > the
> > > > site locally after migrating to Markdown? Previously, we used a
> > container
> > > > to host the site locally for previewing changes. Could you please
> > provide
> > > > guidance on the updated workflow?
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > Harish Vishwanath  於 2025年3月10日 週一
> > 上午11:53寫道：
> > > >
> > > > > Hello all
> > > > >
> > > > > Sorry for my delayed response. Thanks for your feedback and
> comments.
> > > > > Please see below:
> > > > >
> > > > > > Do you plan to have a development version of the website online
> in
> > > > > parallel to the old HTML version?
> > > > > Yes. The current prototype which supports all versions until AK 3.9
> > is
> > > > > running at https://kafka-site-md-711970345036.us-central1.run.app/
> ,
> > > the
> > > > > prototype
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-Prototype
> > > > > >
> > > > > section of the KIP has more details. We should be able to host this
> > in
> > > > ASF
> > > > > infra. Further, as described in the KIP
> > > > >
> > > > > >I understand that locally with the markdown files and an markdown
> > > > editor,
> > > > > we see the rough structure of the docs (which is already an
> > > improvement),
> > > > > but we will not see the end product, because for that we again
> need a
> > > web
> > > > > server.
> > > > > Hugo supports live reload making local development easier (ref:
> > > > >
> https://github.com/hvishwanath/kafka-site-md/blob/main/Makefile#L25)
> > > > >
> > > > >
> > > > > [MM1] Yes., I have ensured that all existing auto generated
> > > > documentations
> > > > > (html files under javadoc, generated folders) are available at the
> > same
> > > > > relative location as they are in the current website. You can see
> > this
> > > in
> > > > > the prototype.
> > > > > For example:
> > > > >
> > > > >
> > > >
> > >
> >
> https://kafka.apache.org/39/javado

Re: [VOTE] KIP-1133: AK Documentation and Website in Markdown

[TaiJu Wu]

Thanks for this KIP,

+1(non-binding)

Best,
TaiJuWu

On Thu, Apr 3, 2025 at 9:42 PM Kuan Po Tseng  wrote:

> Hi Harish,
>
> Thank you for improving the documentation!
>
> +1 (non-binding)
>
> Best,
> Kuan-Po Tseng
>
> On 2025/03/27 09:30:12 Andrew Schofield wrote:
> > Hi Harish,
> > Thanks for this great KIP. I’m really looking forward to being able to
> write Kafka docs in Markdown instead of HTML.
> >
> > +1 (binding)
> >
> > Thanks,
> > Andrew
> >
> > From: Harish Vishwanath 
> > Date: Wednesday, 26 March 2025 at 23:38
> > To: dev@kafka.apache.org 
> > Subject: [VOTE] KIP-1133: AK Documentation and Website in Markdown
> > Hey all
> >
> > I would like to call a vote on KIP-1133: AK Documentation and Website in
> > Markdown.
> >
> > KIP:
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-1133%253A%2BAK%2BDocumentation%2Band%2BWebsite%2Bin%2BMarkdown&data=05%7C02%7C%7C940186c5945340391d7a08dd6cbf46dc%7C84df9e7fe9f640afb435%7C1%7C0%7C638786290955636991%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=YK70nZlRbBzQ6nRuYaGJdD4Au8oFKONv6d8EL%2BpuTu4%3D&reserved=0
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown
> >
> >
> > Discussion thread:
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fkc0kg0dxy6xdrwyv10k56k6g058ng2dw&data=05%7C02%7C%7C940186c5945340391d7a08dd6cbf46dc%7C84df9e7fe9f640afb435%7C1%7C0%7C638786290955681824%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2FtrIYkj5dW48vAn4V2RkYHZcJulIsnBxRcfeM8avDYI%3D&reserved=0
> 
> >
> > Regards
> > Harish
> >
>