Re: [VOTE] KIP-1090 Flaky Test Management

[TengYao Chi]

+1 (non-binding)

Thank you, David

Best,
TengYao

吳岱儒 於 2024年9月30日 週一，下午6:01寫道：

> +1 (non-binding)
> Thanks David, I do love this flaky tag.
>
> TJ
>

[jira] [Created] (KAFKA-17662) config.providers configuration missing from the docs

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-17662:
--

 Summary: config.providers configuration missing from the docs
 Key: KAFKA-17662
 URL: https://issues.apache.org/jira/browse/KAFKA-17662
 Project: Kafka
  Issue Type: Bug
Reporter: Mickael Maison


The config.providers configuration is only listed in the Connect configuration 
documentation. Since it's usable by all components, it should be listed in all 
the configuration sections.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.1 release

[Josep Prat]

Hi there,
I'll attempt to cut the first RC for 3.8.1 this Wednesday. If you have any
bug fix that you'd like to backport to the 3.8 branch and you'd need more
time, please let me know.

Best,

On Tue, Sep 24, 2024 at 1:15 PM Josep Prat  wrote:

> Hi folks!
> As promised, here you have the release plan for 3.8.1:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.1
>
> I'm aiming to have a release candidate by the first week of October unless
> somebody else finds a blocker before then.
>
> Best,
>
> On Fri, Sep 20, 2024 at 11:57 AM Chia-Ping Tsai 
> wrote:
>
>> +1
>>
>> Luke Chen  於 2024年9月20日 週五 下午4:22寫道：
>>
>> > Thanks Josep!
>> > +1
>> >
>> > Luke
>> >
>> > On Fri, Sep 20, 2024 at 3:36 PM Josep Prat > >
>> > wrote:
>> >
>> > > Hey folks,
>> > >
>> > > I'd like to volunteer to be the release manager for a bug fix release
>> of
>> > > the 3.8 series. This will be the first bug fix release and will be
>> > version
>> > > 3.8.1.
>> > >
>> > > If no one has any objections, I will send out a release plan on
>> > 2024/09/23
>> > > that includes a list of all of the fixes we are targeting for 3.8.1
>> along
>> > > with a timeline (aiming probably for a release happening at the
>> beginning
>> > > of October).
>> > >
>> > > Thanks!
>> > >
>> > > --
>> > > [image: Aiven] 
>> > >
>> > > *Josep Prat*
>> > > Open Source Engineering Director, *Aiven*
>> > > josep.p...@aiven.io   |   +491715557497
>> > > aiven.io    |   <
>> > https://www.facebook.com/aivencloud
>> > > >
>> > >      <
>> > > https://twitter.com/aiven_io>
>> > > *Aiven Deutschland GmbH*
>> > > Alexanderufer 3-7, 10117 Berlin
>> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
>> > > Anna Richardson, Kenneth Chen
>> > > Amtsgericht Charlottenburg, HRB 209739 B
>> > >
>> >
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-17660) Kafka not able to import partitioned data

[Adarsh Shukla (Jira)]

Adarsh Shukla created KAFKA-17660:
-

 Summary: Kafka not able to import partitioned data
 Key: KAFKA-17660
 URL: https://issues.apache.org/jira/browse/KAFKA-17660
 Project: Kafka
  Issue Type: Bug
  Components: clients, streams
Affects Versions: 3.6.1
 Environment: Linux, Zlinux, AIX
Reporter: Adarsh Shukla


Below are the logs after executing the import command

05:08:56.899 [pool-2-thread-1] INFO o.a.k.c.c.i.SubscriptionState - [Consumer 
clientId=consumer-NCOMS_nm.polldata_1727352530-1, 
groupId=NCOMS_nm.polldata_1727352530] Resetting offset for partition 
nm.polldata-0 to position FetchPosition\{offset=0, offsetEpoch=Optional.empty, 
currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9092 (id: 0 rack: 
null)], epoch=0}}. 05:17:52.808 [pool-2-thread-1] INFO o.a.k.c.NetworkClient - 
[Consumer clientId=consumer-NCOMS_nm.polldata_1727352530-1, 
groupId=NCOMS_nm.polldata_1727352530] Node -1 disconnected. 05:18:50.829 [main] 
INFO c.i.c.n.s.u.SingleThreadLoop - Stopping nm.polldata loop 05:18:50.836 
[main] WARN c.i.c.n.s.u.SingleThreadLoop - Failed to terminate nm.polldata 
loop. Calling end() explicitly 05:18:50.837 [main] INFO 
c.i.c.n.s.u.SingleThreadLoop - Stopped nm.polldata loop

 

This was executed based on the documentation available for troubleshooting this 
type of issues

After enabling 
"listeners=[PLAINTEXT://127.0.0.1:9092|plaintext://127.0.0.1:9092]" in 
server.properties we then executed below command

 
./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group 
127.0.0.1.NCOMS_nm.polldata_1727352530 --topic nm.polldata --reset-offsets 
--to-datetime 2024-09-27T12.50.39.306 --execute

Error: Executing consumer group command failed due to 
org.apache.kafka.common.errors.TimeoutException: 
Call(callName=describeGroups(api=FIND_COORDINATOR), deadlineMs=1727435855307, 
tries=305, nextAllowedTryMs=1727435855423) timed out at 1727435855323 after 305 
attempt(s)
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: 
Call(callName=describeGroups(api=FIND_COORDINATOR), deadlineMs=1727435855307, 
tries=305, nextAllowedTryMs=1727435855423) timed out at 1727435855323 after 305 
attempt(s)
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:368)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1919)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$resetOffsets$1(ConsumerGroupCommand.scala:434)
at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646)
at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642)
at scala.collection.AbstractIterable.foldLeft(Iterable.scala:919)
at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:432)
at kafka.admin.ConsumerGroupCommand$.run(ConsumerGroupCommand.scala:76)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:59)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: 
Call(callName=describeGroups(api=FIND_COORDINATOR), deadlineMs=1727435855307, 
tries=305, nextAllowedTryMs=1727435855423) timed out at 1727435855323 after 305 
attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled 
describeGroups(api=FIND_COORDINATOR) request with correlation id 307 due to 
node 0 being disconnected
 
we are using storm 2.5.0 and zookeeper 3.9.2 with kafka 3.6.1 the issue is also 
seen when we tried the latest version of kafka which is 3.6.5.
 
Kindly help us resolve this issue as the data is not getting imported from the 
DB



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17661) Fix flaky BufferPoolTest.testBlockTimeout

[Yu-Lin Chen (Jira)]

Yu-Lin Chen created KAFKA-17661:
---

 Summary: Fix flaky BufferPoolTest.testBlockTimeout
 Key: KAFKA-17661
 URL: https://issues.apache.org/jira/browse/KAFKA-17661
 Project: Kafka
  Issue Type: Bug
  Components: clients, unit tests
Reporter: Yu-Lin Chen
Assignee: Yu-Lin Chen
 Attachments: 
0001-reproduce-racing-issue-by-adding-delay-to-test-thread.patch

4 flaky out of 221 trunk build in the past 28 days. (github) ([Report 
Link|https://ge.apache.org/scans/tests?search.rootProjectNames=kafka&search.startTimeMax=1727681219558&search.startTimeMin=172520640&search.tags=github&search.timeZoneId=Asia%2FTaipei&tests.container=org.apache.kafka.clients.producer.internals.BufferPoolTest&tests.test=testBlockTimeout()])

([Sep 27 2024 at 
04:54:00|https://ge.apache.org/s/nh44u7tsm2lri/tests/task/:clients:test/details/org.apache.kafka.clients.producer.internals.BufferPoolTest/testBlockTimeout()?expanded-stacktrace=WyIwIl0&top-execution=1])
{code:java}
org.opentest4j.AssertionFailedError: The buffer allocated more memory than its 
maximum value 10 

at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:138)   
at 
org.apache.kafka.clients.producer.internals.BufferPoolTest.testBlockTimeout(BufferPoolTest.java:184)
 
at java.lang.reflect.Method.invoke(Method.java:566) 
at java.util.ArrayList.forEach(ArrayList.java:1541) 
at java.util.ArrayList.forEach(ArrayList.java:1541)
{code}
Root cause:
 # The test relies on 3 asynchronous threads being triggered in parallel with 
the test thread [1]. However, there is no guarantee of parallelism in test 
environment. The issue will happend if test thread didn't get CPU within 25 ms. 
We could reproduce this issue by adding 30 ms delay to test thread. Please 
check the attached patch.
 # Since a 25 ms delay is obviously unreliable in the test environment, we 
could consider rewriting the test or increasing the delay. (The maxBlockTimeMs 
was reduced from 2000ms to 10 ms in KAFKA-9852)

[1] 
[https://github.com/apache/kafka/blob/40360819bb97d6b05dfef6451888b4d908fc3bf4/clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java#L175-L179]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Francois Visconte]

Could we vote on this? This is causing a bunch of tiered storage read
issues as many consumers default to READ_COMMITTED (eg. librdkafka)

Thanks,
F.

On Mon, Sep 16, 2024 at 7:20 AM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Bumping this thread for vote. PTAL.
>
> On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to open voting for KIP-1058. This KIP improves the consumer
> > reading from remote storage when READ_COMMITTED isolation level is
> enabled.
> > PTAL.
> >
> > KIP-1058
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1058%3A+Txn+consumer+exerts+pressure+on+remote+storage+when+collecting+aborted+transactions
> >
> >
> > Thanks,
> > Kamal
> >
>

Re: [VOTE] KIP-1090 Flaky Test Management

[吳岱儒]

+1 (non-binding)
Thanks David, I do love this flaky tag.

TJ

[jira] [Resolved] (KAFKA-17560) testCurrentLag flaky if enabled for new consumer

[TengYao Chi (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

TengYao Chi resolved KAFKA-17560.
-
Resolution: Fixed

> testCurrentLag flaky if enabled for new consumer
> 
>
> Key: KAFKA-17560
> URL: https://issues.apache.org/jira/browse/KAFKA-17560
> Project: Kafka
>  Issue Type: Task
>  Components: clients, consumer
>Reporter: Lianet Magrans
>Assignee: TengYao Chi
>Priority: Minor
>  Labels: consumer, consumer-threading-refactor
>
> There was an initial attempt to enable testCurrentLag for the new consumer 
> with [https://github.com/apache/kafka/pull/16982.] Even though it included 
> several changes addressing flakiness, it seems not enough, given that the 
> test is still flaky when running for the new consumer. Fails with:
>  
> org.opentest4j.AssertionFailedError: 
> Expected :OptionalLong[40]
> Actual   :OptionalLong.empty
> 
> at 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testCurrentLag(KafkaConsumerTest.java:2529)
>  
> I wonder if this flakiness is because after receiving a response to a list 
> offset request, maybe a single call to currentLag may not be enough (offsets 
> received in the response not updated yet in the background). Note that after 
> receiving a response to the ListOffsets request, the test moves right away 
> into a single call to currentLag that it expects to be 40, but in the 
> background we only update the subscription state when we get to:
> [https://github.com/apache/kafka/blob/aaf3fc05f8e7cb6b8c79f8471f68faa3a994e1e5/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetsRequestManager.java#L565]
>  
> So we could end up with that single call to currentLag not finding the 
> endOffsets we got in the response (it will find them "eventually" with the 
> new consumer I expect)  
>  
> We should review this test to be able to enable it for the new consumer 
> without flakiness.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17663) Add Metadata caching in admin.internals.PartitionLeaderStrategy

[Nicolas Guyomar (Jira)]

Nicolas Guyomar created KAFKA-17663:
---

 Summary: Add Metadata caching in 
admin.internals.PartitionLeaderStrategy
 Key: KAFKA-17663
 URL: https://issues.apache.org/jira/browse/KAFKA-17663
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Reporter: Nicolas Guyomar


Hi team,

To the best of my knowledge, there is no Metadata caching in AdminClient 
admin.internals.PartitionLeaderStrategy, and for use cases that keep long live 
AdminClient running with scheduled invocation of method such as the 
listOffsets, we could keep reusing the same Metadata and not request fresh one 
on each invocation

This is particularly true when the listOffsets is invoked for monitoring 
purpose, often querying a high number of partitions at the same time, leading 
to rather "large" Metadata requests to be processed on the cluster side

Thank you



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17665) Kafka Tiered Storage retry back off is not used

[Jorge Machado (Jira)]

Jorge Machado created KAFKA-17665:
-

 Summary: Kafka Tiered Storage retry back off is not used
 Key: KAFKA-17665
 URL: https://issues.apache.org/jira/browse/KAFKA-17665
 Project: Kafka
  Issue Type: Bug
  Components: core, Tiered-Storage
Reporter: Jorge Machado


Hi, we are evaluating the Kafka Tiered Storage. We already see KAFKA-17062 
which under 3.9 we could have data loss. I was looking at the code and also the 
KIP-405. 

 

On the KIP it is mentioned that if a copy fails it will be retried with a 
backoff propagation defined on REMOTE_LOG_MANAGER_TASK_RETRY_BACK_OFF_MS_PROP.  
The issue is the method  remoteLogManagerTaskRetryBackoffMs() is not used in 
any place of the code. So I think this parameters make no effect currently. Is 
there any retry of failed segments ? Until now I just saw logging the error.

Question from me:  What happen to segments that have the state  
COPY_SEGMENT_STARTED between server restarts?

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Mickael Maison]

Congratulations Kamal!

On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
>
> Hi all,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal
> Chandraprakash.
>
> Kamal has been a Kafka contributor since May 2017. He has made significant
> contributions to the tiered storage feature (KIP-405). He authored KIP-1018
> and KIP-1075 which improved tiered storage operation. He also contributed
> to discussing and reviewing many KIPs.
>
> Congratulations, Kamal!
>
> Thanks,
> Luke (on behalf of the Apache Kafka PMC)

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[kafka]

Congrats Kamal!

-
Gaurav

> On 30 Sep 2024, at 18:10, Mickael Maison  wrote:
> 
> Congratulations Kamal!
> 
> On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
>> 
>> Hi all,
>> 
>> The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal
>> Chandraprakash.
>> 
>> Kamal has been a Kafka contributor since May 2017. He has made significant
>> contributions to the tiered storage feature (KIP-405). He authored KIP-1018
>> and KIP-1075 which improved tiered storage operation. He also contributed
>> to discussing and reviewing many KIPs.
>> 
>> Congratulations, Kamal!
>> 
>> Thanks,
>> Luke (on behalf of the Apache Kafka PMC)

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Josep Prat]

Congrats Kamal!

On Mon, Sep 30, 2024 at 2:46 PM  wrote:

> Congrats Kamal!
>
> -
> Gaurav
>
> > On 30 Sep 2024, at 18:10, Mickael Maison 
> wrote:
> >
> > Congratulations Kamal!
> >
> > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
> >>
> >> Hi all,
> >>
> >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> Kamal
> >> Chandraprakash.
> >>
> >> Kamal has been a Kafka contributor since May 2017. He has made
> significant
> >> contributions to the tiered storage feature (KIP-405). He authored
> KIP-1018
> >> and KIP-1075 which improved tiered storage operation. He also
> contributed
> >> to discussing and reviewing many KIPs.
> >>
> >> Congratulations, Kamal!
> >>
> >> Thanks,
> >> Luke (on behalf of the Apache Kafka PMC)
>
>

-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Murali Basani]

Congratulations Kamal.

On Mon, Sep 30, 2024 at 3:21 PM Chia-Ping Tsai  wrote:

> Congratulations Kamal!!!
>
> Satish Duggana  於 2024年9月30日 週一 下午9:01寫道：
>
> > Congratulations Kamal!
> >
> > On Mon, 30 Sept 2024 at 18:18, Josep Prat 
> > wrote:
> >
> > > Congrats Kamal!
> > >
> > > On Mon, Sep 30, 2024 at 2:46 PM  wrote:
> > >
> > > > Congrats Kamal!
> > > >
> > > > -
> > > > Gaurav
> > > >
> > > > > On 30 Sep 2024, at 18:10, Mickael Maison  >
> > > > wrote:
> > > > >
> > > > > Congratulations Kamal!
> > > > >
> > > > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen 
> wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> The PMC of Apache Kafka is pleased to announce a new Kafka
> > committer,
> > > > Kamal
> > > > >> Chandraprakash.
> > > > >>
> > > > >> Kamal has been a Kafka contributor since May 2017. He has made
> > > > significant
> > > > >> contributions to the tiered storage feature (KIP-405). He authored
> > > > KIP-1018
> > > > >> and KIP-1075 which improved tiered storage operation. He also
> > > > contributed
> > > > >> to discussing and reviewing many KIPs.
> > > > >>
> > > > >> Congratulations, Kamal!
> > > > >>
> > > > >> Thanks,
> > > > >> Luke (on behalf of the Apache Kafka PMC)
> > > >
> > > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > Anna Richardson, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Bruno Cadonna]

Congrats!

Best,
Bruno

On 9/30/24 3:23 PM, Murali Basani wrote:

Congratulations Kamal.

On Mon, Sep 30, 2024 at 3:21 PM Chia-Ping Tsai  wrote:


Congratulations Kamal!!!

Satish Duggana  於 2024年9月30日 週一 下午9:01寫道：


Congratulations Kamal!

On Mon, 30 Sept 2024 at 18:18, Josep Prat 
wrote:


Congrats Kamal!

On Mon, Sep 30, 2024 at 2:46 PM  wrote:


Congrats Kamal!

-
Gaurav


On 30 Sep 2024, at 18:10, Mickael Maison 


wrote:


Congratulations Kamal!

On Mon, Sep 30, 2024 at 2:37 PM Luke Chen 

wrote:


Hi all,

The PMC of Apache Kafka is pleased to announce a new Kafka

committer,

Kamal

Chandraprakash.

Kamal has been a Kafka contributor since May 2017. He has made

significant

contributions to the tiered storage feature (KIP-405). He authored

KIP-1018

and KIP-1075 which improved tiered storage operation. He also

contributed

to discussing and reviewing many KIPs.

Congratulations, Kamal!

Thanks,
Luke (on behalf of the Apache Kafka PMC)





--
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   <

https://www.facebook.com/aivencloud



      <
https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Matthias J. Sax]

I am also in favor of consistent APIs. That's very good point. I did not 
take `Admin` API into account, and I am not aware that consumer / 
producer would have config object classes?


Seems we are in a tricky situation here, because "consistent API" to me 
means producer, consumer, admin and KS.


The KS surface area might be the largest one (even if there is a long 
list of `XxxOption` classes for Admin API), and we do follow the pattern 
as described. It's of course not desirable to just change the whole 
Admin API (and neither the KS API), but it might be good to agree on a 
"gold standard" and do everything new accordingly?


Given that producer / consumer do not have any config object classes 
yet, it seems to be the question if they should follow the Admin pattern 
or the KS pattern. -- I tend to think, following the KS pattern might be 
better? But yes, I see your POV to say it should follow the Admin API 
pattern, however, it would imply that we "need" to change all the KS APIs.


Overall, the "static method builder" pattern seems better to me, and 
thus I would prefer to make it the "gold standard" and we can see what 
we can do for `Admin API` mid/long term?



Let's see what others think.


Btw: I was also wondering, if we should re-use the new consumer 
`CloseOption` class for `KafkaStreams#close()` and deprecate the KS 
`CloseOption` class? Not sure. Just another "random" idea.




-Matthias


On 9/29/24 12:08 PM, Chia-Ping Tsai wrote:

 From an API POV, I think the new `CloseOptions` class should not have

any "getters" and thus, it's irrelevant how we represent the different
cases in code internally (even if I believe using `Optional` might be a
good way to handle it).

If we choose to avoid using getters, consumers would have to access the
internal variables directly, similar to how it's done in the streams
CloseOptions. While this is a matter of preference, it's worth noting that
Admin options do provide both setters and getters. In my opinion, since all
of these options are part of the kafka-clients code, they should adhere to
a consistent coding style.


In KS, we use config objects like `CloseOption` all the time, with static

"factory" method (and private constructors), and an internal  sub-class
which would have the getters if needed. So I think we should have

I value the static factory methods for their convenience, allowing users to
quickly create an object when they want to set only one option. However,
while I don't want to sound repetitive, maintaining a consistent pattern
across the API is essential.

Best,
Chia-Ping


Matthias J. Sax  於 2024年9月30日 週一 上午2:26寫道：


I am not sure, but I get the impression that we are starting to talk too
much about implementation details now?

  From an API POV, I think the new `CloseOptions` class should not have
any "getters" and thus, it's irrelevant how we represent the different
cases in code internally (even if I believe using `Optional` might be a
good way to handle it).

In KS, we use config objects like `CloseOption` all the time, with
static "factory" method (and private constructors), and an internal
sub-class which would have the getters if needed. So I think we should have

public class CloseOptions {
  private CloseOptions(Optional, Optional);

  public static CloseOptions timeout(Duration);

  public static CloseOptions leaveGroup(boolean);

  public CloseOption withTimeout(Duration);

  public CloseOption withLeaveGroup(boolean);
}


This allows to call as example:

consumer.close(CloseOptions.leaveGroup(true));

or

consumer.close(
 CloseOptions.timeout(Duration.ofMinutes(5)
 .withLeaveGroup(false)
 );

We can still discuss naming and what overloads we want to add, but in
general, a well established and proven pattern is to have a few static
"entry" methods, with return the object itself to allow chaining with
non-static methods.



I am not sure, why KIP-812 did not follow this pattern... I think it got
it wrong and we should not repeat this "mistake". Maybe we could
actually piggy-pack a cleanup for the existing KS CloseOption object
into this KIP?



-Matthias

On 9/29/24 8:39 AM, TengYao Chi wrote:

Hi Chia-Ping,

Thanks for your feedback.
What I intended to express is that if `Optional.empty()` is passed to the
`timeout`, it will eventually be converted to `DEFAULT_CLOSE_TIMEOUT_MS`,
just as you mentioned.
Apologies for not expressing that clearly and for any confusion caused.

Best regards,
TengYao

Chia-Ping Tsai  於 2024年9月29日 週日 下午10:16寫道：


hi TengYao


I have reviewed the `close()` method implementation for both the

Classic

and Async Consumers. I believe the `timeout` parameter could have a

default

value, and this default should align with the existing

`Consumer#close()`

method, which internally calls the overloaded `Consumer#close(Duration)`
with a default of 30 seconds (`DEFAULT_CLOSE_TIMEOUT_MS`).

If you assign a default value (30s) to CloseOptions#t

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Matthias J. Sax]

Congrats!

On 9/30/24 6:59 AM, Yash Mayya wrote:

Congratulations Kamal!

On Mon, 30 Sept, 2024, 18:07 Luke Chen,  wrote:


Hi all,

The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal
Chandraprakash.

Kamal has been a Kafka contributor since May 2017. He has made significant
contributions to the tiered storage feature (KIP-405). He authored KIP-1018
and KIP-1075 which improved tiered storage operation. He also contributed
to discussing and reviewing many KIPs.

Congratulations, Kamal!

Thanks,
Luke (on behalf of the Apache Kafka PMC)

[jira] [Created] (KAFKA-17668) Rewrite LogCleaner#maxOverCleanerThreads and LogCleanerManager#maintainUncleanablePartitions

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17668:
--

 Summary: Rewrite LogCleaner#maxOverCleanerThreads and 
LogCleanerManager#maintainUncleanablePartitions
 Key: KAFKA-17668
 URL: https://issues.apache.org/jira/browse/KAFKA-17668
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Both methods can be rewrite by scala 2.13 methods



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1090 Flaky Tests 👻

[Jun Rao]

Hi, David,

Thanks for the report.

1. Where could we see the reports (quarantined tests, etc) mentioned in the
KIP?
2. The quarantine process seems to only apply to integration tests. What's
our recommendation for flaky unit tests?

Jun

On Thu, Sep 26, 2024 at 1:34 PM David Arthur  wrote:

> If there is no more feedback on this, I'll go ahead and move to a vote.
>
> -David
>
> On Sun, Sep 22, 2024 at 11:04 AM Chia-Ping Tsai 
> wrote:
>
> >
> >
> > > David Arthur  於 2024年9月22日 晚上10:07 寫道：
> > >
> > > Q2: Yes, I think we should run the quarantined tests on all CI builds,
> > PRs
> > > and trunk. We can achieve this with --rerun-tasks. This will let PR
> > authors
> > > gain feedback about their changes affect on the flaky tests. We could
> > even
> > > create a PR-specific report that shows if their changes improved or
> > > worsened the flakiness of the quarantined tests.
> >
> > I guess it will be an individual status like “Gradle Build Scan”? The
> > failure of quarantined tests does not obstruct us from merging PR unless
> > the target of PR is to fix specific flaky.
> >
> > Thanks,
> > Chia-Ping
>
>
>
> --
> David Arthur
>

[jira] [Resolved] (KAFKA-15862) Remove SecurityManager Support

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-15862.
-
Fix Version/s: 4.0.0
   Resolution: Fixed

> Remove SecurityManager Support
> --
>
> Key: KAFKA-15862
> URL: https://issues.apache.org/jira/browse/KAFKA-15862
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients, connect, Tiered-Storage
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Major
> Fix For: 4.0.0
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15862) Remove SecurityManager Support

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris reopened KAFKA-15862:
-

> Remove SecurityManager Support
> --
>
> Key: KAFKA-15862
> URL: https://issues.apache.org/jira/browse/KAFKA-15862
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients, connect, Tiered-Storage
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Major
> Fix For: 4.0.0
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17634) Tighten up wakeup handling for share consumer

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17634.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Tighten up wakeup handling for share consumer
> -
>
> Key: KAFKA-17634
> URL: https://issues.apache.org/jira/browse/KAFKA-17634
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1094 Add a new constructor method with nextOffsets to ConsumerRecords

[Alieh Saeedi]

Hi all,

thanks for the feedbacks.
Since there is consensus in deprecating the current constructor, we can
deprecate it for now and remove it later on (in 5.0 maybe).
The KIP is updated.

Thanks,
Alieh

On Sun, Sep 29, 2024 at 6:22 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> @sophie I meant AK tests. But you make a fair point. There will not be
> that many instances I'm sure, so if the consensus is to deprecate the
> old constructor, that's fine with me.
>
> 
> From: Sophie Blee-Goldman 
> Sent: 28 September 2024 22:56
> To: dev@kafka.apache.org 
> Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> nextOffsets to ConsumerRecords
>
> @Andrew do you mean user tests might create ConsumerRecords, or are you
> concerned about AK tests?
>
> I guess Alieh would be the one to implement this, so I'll leave it up to
> her to investigate the current usage of the existing constructor and
> whether it feels like too much trivial work to remove it from our own AK
> tests. Otherwise, on pure API design philosophy grounds, I would prefer to
> deprecate and remove the old one.
>
> However if we feel that users have reason to create ConsumerRecords for
> tests then it's probably not worth the upheaval to deprecate the old
> constructor.
>
> On Fri, Sep 27, 2024 at 2:38 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
> > Hi Matthias,
> > Tests will also create ConsumerRecords. I don't see the point in forcing
> a
> > bunch of trivial changes to tests, that's all.
> > Of course, it's not too hard to tweak the tests, but it's making work.
> >
> > Thanks,
> > Andrew
> >
> > 
> > From: Matthias J. Sax 
> > Sent: 27 September 2024 18:36
> > To: dev@kafka.apache.org 
> > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > nextOffsets to ConsumerRecords
> >
> > Thanks for the KIP Alieh.
> >
> > Deprecating the existing constructor won't break anybody atm, right? We
> > don't remove the constructor yet. Of course, in 5.0 in the future we
> > would apply this breaking change.
> >
> > While `ConsumerRecord` is a public API, it seems that actual user code
> > should never create a `ConsumerRecord` object? In general (of course
> > this is not in scope of this KIP) I am wondering why we have a public
> > constructor to begin with (I assume it's some Java limitations how
> > visibility is managed). Having said this, I would also tend to deprecate
> > the old constructor.
> >
> > @Andrew: Why do you prefer to keep the old constructor? To me it seems
> > to be an potential source of bugs, to keep the old one?
> >
> >
> > -Matthias
> >
> > On 9/27/24 2:19 AM, Andrew Schofield wrote:
> > > Hi Alieh,
> > > Thanks for the KIP. Looks like a nice addition to me. I prefer not
> > deprecating the existing
> > > constructor too.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > 
> > > From: Alieh Saeedi 
> > > Sent: 27 September 2024 09:09
> > > To: dev@kafka.apache.org 
> > > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > nextOffsets to ConsumerRecords
> > >
> > > Thank you, Bill and Sophie.
> > >
> > > @Bill: You are right. I updated the KIP.
> > > @Sophie: Yes, it was also our concern, but we thought we should keep
> the
> > > current constructor in order to not break the user's code who has
> called
> > > this constructor in their code.
> > >
> > > Thanks,
> > > Alieh
> > >
> > > On Fri, Sep 27, 2024 at 1:07 AM Sophie Blee-Goldman <
> > sop...@responsive.dev>
> > > wrote:
> > >
> > >> Should we deprecate the old constructor to make sure that all info
> gets
> > >> passed in when creating a ConsumerRecords instance?
> > >>
> > >> On Thu, Sep 26, 2024 at 3:37 PM Bill Bejeck 
> wrote:
> > >>
> > >>> Hi Alieh,
> > >>>
> > >>> Thanks for the KIP, it will be very useful to Kafka Streams.
> > >>> I have one comment.  In the "Proposed Changes" section, you mention
> the
> > >>> "The `nextOffsets` object contains the next offset and the last
> leader
> > >>> epoch per partition".
> > >>> If understand the KIP correctly, it should be something along the
> lines
> > >> of
> > >>> "The `nextOffsets` method returns a map of `TopicPartition` to
> > >>> `OffsetAndMetadata` objects and  `OffsetAndMetadata` contains the
> next
> > >>> offset and the last leader epoch per partition"
> > >>>
> > >>> Other than that, the KIP LGTM.
> > >>>
> > >>> Thanks,
> > >>> Bill
> > >>>
> > >>>
> > >>> On Wed, Sep 25, 2024 at 7:43 AM Alieh Saeedi
> > >>  > 
> > >>> wrote:
> > >>>
> >  Hi all,
> > 
> >  I would like to open a discussion for KIP-1094: Add a new
> constructor
> >  method with `nextOffsets` to `ConsumerRecords`.
> > 
> >  You can find the detailed proposal here:
> > 
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1094%3A+Add+a+new+constructor+method+with+nextOffsets+to+ConsumerRecord

[jira] [Resolved] (KAFKA-17669) Ivan_Test

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-17669.

Resolution: Invalid

> Ivan_Test
> -
>
> Key: KAFKA-17669
> URL: https://issues.apache.org/jira/browse/KAFKA-17669
> Project: Kafka
>  Issue Type: Task
>Reporter: Arseny
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] New committer: Kamal Chandraprakash

[Luke Chen]

Hi all,

The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal
Chandraprakash.

Kamal has been a Kafka contributor since May 2017. He has made significant
contributions to the tiered storage feature (KIP-405). He authored KIP-1018
and KIP-1075 which improved tiered storage operation. He also contributed
to discussing and reviewing many KIPs.

Congratulations, Kamal!

Thanks,
Luke (on behalf of the Apache Kafka PMC)

[jira] [Created] (KAFKA-17666) Kafka doesn't monitor disk space after it detects it is full

[Kevin Fletcher (Jira)]

Kevin Fletcher created KAFKA-17666:
--

 Summary: Kafka doesn't monitor disk space after it detects it is 
full
 Key: KAFKA-17666
 URL: https://issues.apache.org/jira/browse/KAFKA-17666
 Project: Kafka
  Issue Type: Bug
Reporter: Kevin Fletcher


Scenario: Kafka data volume becomes full (100%) but once freed up, Kafka 
ignores this disk until it is restarted. It does not auto-detect that space has 
been freed up, and begin to function again, without restarting.
{code:java}
Sep 12 17:40:39 kq-2b.was2.sd.com kafka-server-start[1696]: Stopping serving 
replicas in dir /opt/clusterone/kafka/disk4 (kafka.server.ReplicaManager)
...
Sep 12 17:40:39 kq-2b.was2.sd.com kafka-server-start[1696]: 
java.io.IOException: No space left on device {code}
Example showing disk4 became full:
{code:java}
[ops@us1-zpa-kq-2b.was2.moderate ~]$ df -h | grep kafka
/dev/nvme2n1    400G  256G  145G  64% /opt/clusterone/kafka/disk3
/dev/nvme4n1    400G  275G  126G  69% /opt/clusterone/kafka/disk1
/dev/nvme3n1    400G  229G  172G  58% /opt/clusterone/kafka/disk2
/dev/nvme1n1    400G  400G    0G 100% /opt/clusterone/kafka/disk4 {code}
This topic is 128 partitions, 2 replicas of each, spread across 8 KQ brokers.

Each broker has partitions spread across 4 dirs (4 disks) - here is disk4:
{code:java}
/usr/bin/kafka-log-dirs --bootstrap-server=localhost:9092 --describe 
--topic-list txn | tail -1 | jq
        {
          "logDir": "/opt/clusterone/kafka/disk4",
          "error": null,
          "partitions": [
            {
              "partition": "txn-49",
              "size": 5676453238,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-84",
              "size": 5616346237,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-52",
              "size": 5587352418,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-36",
              "size": 5559175359,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-116",
              "size": 5532912024,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-105",
              "size": 5525176032,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-76",
              "size": 5429519389,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "txn-119",
              "size": 5632860112,
              "offsetLag": 0,
              "isFuture": false
            }
          ]
        }, {code}
 

Issue 1: After freeing up disk space (or growing a volume in real-time), Kafka 
never reports any further log msgs about disk4, it just ignores it until it is 
restarted.

It would be ideal if Kafka could periodicially check back on this disk and see 
if it is freed up yet so it can continue.

Issue 2: When shrinking retention live (via retention.ms for example), Kafka 
does not begin to delete files from the 100% full disk, instead it continues to 
ignore all activity related to disk4. This forces us to have to manually delete 
files from the disk (less than ideal).

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1076: Metrics for client applications a KIP-714 extension

[Matthias J. Sax]

Thanks Bill.

Given that Admin client does currently not implement sending metric, the 
change to disable it by default for Admin client does make sense to me. 
It might be too much overhead to open/close connection if there are not 
metrics to be sent, and for developers using the new API to send custom 
metrics they can enable it.



-Matthias

On 9/27/24 3:32 PM, Bill Bejeck wrote:

Hi All,

Just a quick update on KIP-1076.  Due to the ad hoc connections made by the
`AdminClient` I've updated the KIP to state that the default value of the
`AdminClient.ENABLE_METRICS_PUSH_CONFIG` will be set to false.
I expect this minor change to be non-controversial, but it should be noted
in the discussion.

Thanks,
Bill

On Fri, Sep 6, 2024 at 12:59 PM Bill Bejeck  wrote:


Hi All,

The vote is now closed.
The KIP has been accepted with 4 binding votes from Lucas, Matthias,
Sophie, and Bruno and 2 non-binding votes from Andrew and Apoorv.
Thanks to everyone for participating!

Regards,
Bill

On Thu, Sep 5, 2024 at 10:58 AM Bruno Cadonna  wrote:


Thanks for the KIP!

+1 (binding)

Best,
Bruno

On 9/4/24 11:29 PM, Sophie Blee-Goldman wrote:

Thanks for the KIP Bill

+1 (binding)

On Tue, Sep 3, 2024 at 1:54 PM Apoorv Mittal 
wrote:


Thanks for the KIP, looking forward to it.

+1 (non-binding)

Regards,
Apoorv Mittal


On Tue, Sep 3, 2024 at 9:00 PM Matthias J. Sax 

wrote:



+1 (binding)

On 9/3/24 8:56 AM, Andrew Schofield wrote:

Thanks for the KIP. +1 (non-binding)

Andrew

From: Lucas Brutschy 
Sent: 03 September 2024 16:02
To: dev@kafka.apache.org 
Subject: Re: [VOTE] KIP-1076: Metrics for client applications a

KIP-714

extension


Hi Bill,

thanks for the KIP! +1 (binding)

Lucas

On Tue, Sep 3, 2024 at 4:49 PM Bill Bejeck 

wrote:


Hi All,

I'd like to call for a vote on KIP-1076
<





https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-1076%253A%2B%2BMetrics%2Bfor%2Bclient%2Bapplications%2BKIP-714%2Bextension&data=05%7C02%7C%7C0ba83dece6894a4126b508dccc29a7ed%7C84df9e7fe9f640afb435%7C1%7C0%7C638609726517807511%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=DqHjamMzaj4gwSvGwZv89CPCONXxNrB%2BNyhte2oJpKE%3D&reserved=0

<




https://cwiki.apache.org/confluence/display/KAFKA/KIP-1076%3A++Metrics+for+client+applications+KIP-714+extension


(discussion can be found here
<





https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F9yf4w9qy0zr7qy9ftjcft0s32w32b891&data=05%7C02%7C%7C0ba83dece6894a4126b508dccc29a7ed%7C84df9e7fe9f640afb435%7C1%7C0%7C638609726517819714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=hP3ykAMVOAb8WigLK92GDLl6pYCty4AgngZrsEDQLhQ%3D&reserved=0

>)


-Bill

[jira] [Created] (KAFKA-17664) Generate ShareAcknowledgeCommitCallbackEvent only when the callback is configured.

[Shivsundar R (Jira)]

Shivsundar R created KAFKA-17664:


 Summary: Generate ShareAcknowledgeCommitCallbackEvent only when 
the callback is configured.
 Key: KAFKA-17664
 URL: https://issues.apache.org/jira/browse/KAFKA-17664
 Project: Kafka
  Issue Type: Sub-task
Reporter: Shivsundar R


Currently, we prepare a ShareAcknowledgeCommitCallback event for every 
ShareAcknowledgeResponse and send it over to the application thread. 
In cases where the acknowledgement commit callback handler is not configured by 
the user, this event is not used in the application thread. So we can generate 
this event based on whether the callback was configured.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Satish Duggana]

Congratulations Kamal!

On Mon, 30 Sept 2024 at 18:18, Josep Prat 
wrote:

> Congrats Kamal!
>
> On Mon, Sep 30, 2024 at 2:46 PM  wrote:
>
> > Congrats Kamal!
> >
> > -
> > Gaurav
> >
> > > On 30 Sep 2024, at 18:10, Mickael Maison 
> > wrote:
> > >
> > > Congratulations Kamal!
> > >
> > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Kamal
> > >> Chandraprakash.
> > >>
> > >> Kamal has been a Kafka contributor since May 2017. He has made
> > significant
> > >> contributions to the tiered storage feature (KIP-405). He authored
> > KIP-1018
> > >> and KIP-1075 which improved tiered storage operation. He also
> > contributed
> > >> to discussing and reviewing many KIPs.
> > >>
> > >> Congratulations, Kamal!
> > >>
> > >> Thanks,
> > >> Luke (on behalf of the Apache Kafka PMC)
> >
> >
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Andrew Schofield]

Congratulations, Kamal.


From: Bruno Cadonna 
Sent: 30 September 2024 14:30
To: dev@kafka.apache.org 
Subject: Re: [ANNOUNCE] New committer: Kamal Chandraprakash

Congrats!

Best,
Bruno

On 9/30/24 3:23 PM, Murali Basani wrote:
> Congratulations Kamal.
>
> On Mon, Sep 30, 2024 at 3:21 PM Chia-Ping Tsai  wrote:
>
>> Congratulations Kamal!!!
>>
>> Satish Duggana  於 2024年9月30日 週一 下午9:01寫道：
>>
>>> Congratulations Kamal!
>>>
>>> On Mon, 30 Sept 2024 at 18:18, Josep Prat 
>>> wrote:
>>>
 Congrats Kamal!

 On Mon, Sep 30, 2024 at 2:46 PM  wrote:

> Congrats Kamal!
>
> -
> Gaurav
>
>> On 30 Sep 2024, at 18:10, Mickael Maison >>
> wrote:
>>
>> Congratulations Kamal!
>>
>> On Mon, Sep 30, 2024 at 2:37 PM Luke Chen 
>> wrote:
>>>
>>> Hi all,
>>>
>>> The PMC of Apache Kafka is pleased to announce a new Kafka
>>> committer,
> Kamal
>>> Chandraprakash.
>>>
>>> Kamal has been a Kafka contributor since May 2017. He has made
> significant
>>> contributions to the tiered storage feature (KIP-405). He authored
> KIP-1018
>>> and KIP-1075 which improved tiered storage operation. He also
> contributed
>>> to discussing and reviewing many KIPs.
>>>
>>> Congratulations, Kamal!
>>>
>>> Thanks,
>>> Luke (on behalf of the Apache Kafka PMC)
>
>

 --
 [image: Aiven] 

 *Josep Prat*
 Open Source Engineering Director, *Aiven*
 josep.p...@aiven.io   |   +491715557497
 aiven.io    |   <
>>> https://www.facebook.com/aivencloud
>
   <
 https://twitter.com/aiven_io>
 *Aiven Deutschland GmbH*
 Alexanderufer 3-7, 10117 Berlin
 Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
 Anna Richardson, Kenneth Chen
 Amtsgericht Charlottenburg, HRB 209739 B

>>>
>>
>

[jira] [Created] (KAFKA-17669) Ivan_Test

[Arseny (Jira)]

Arseny created KAFKA-17669:
--

 Summary: Ivan_Test
 Key: KAFKA-17669
 URL: https://issues.apache.org/jira/browse/KAFKA-17669
 Project: Kafka
  Issue Type: Task
Reporter: Arseny






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Kamal Chandraprakash]

Thank you all for your warm wishes!

--
Kamal

On Tue, Oct 1, 2024 at 12:11 AM Viktor Somogyi-Vass
 wrote:

> Congrats Kamal! :)
>
> On Mon, Sep 30, 2024, 19:21 Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 9/30/24 6:59 AM, Yash Mayya wrote:
> > > Congratulations Kamal!
> > >
> > > On Mon, 30 Sept, 2024, 18:07 Luke Chen,  wrote:
> > >
> > >> Hi all,
> > >>
> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Kamal
> > >> Chandraprakash.
> > >>
> > >> Kamal has been a Kafka contributor since May 2017. He has made
> > significant
> > >> contributions to the tiered storage feature (KIP-405). He authored
> > KIP-1018
> > >> and KIP-1075 which improved tiered storage operation. He also
> > contributed
> > >> to discussing and reviewing many KIPs.
> > >>
> > >> Congratulations, Kamal!
> > >>
> > >> Thanks,
> > >> Luke (on behalf of the Apache Kafka PMC)
> > >>
> > >
> >
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Chris Egerton]

Congrats!

-- Forwarded message -
From: Satish Duggana 
Date: Mon, Sep 30, 2024, 09:01
Subject: Re: [ANNOUNCE] New committer: Kamal Chandraprakash
To: 


Congratulations Kamal!

On Mon, 30 Sept 2024 at 18:18, Josep Prat 
wrote:

> Congrats Kamal!
>
> On Mon, Sep 30, 2024 at 2:46 PM  wrote:
>
> > Congrats Kamal!
> >
> > -
> > Gaurav
> >
> > > On 30 Sep 2024, at 18:10, Mickael Maison 
> > wrote:
> > >
> > > Congratulations Kamal!
> > >
> > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Kamal
> > >> Chandraprakash.
> > >>
> > >> Kamal has been a Kafka contributor since May 2017. He has made
> > significant
> > >> contributions to the tiered storage feature (KIP-405). He authored
> > KIP-1018
> > >> and KIP-1075 which improved tiered storage operation. He also
> > contributed
> > >> to discussing and reviewing many KIPs.
> > >>
> > >> Congratulations, Kamal!
> > >>
> > >> Thanks,
> > >> Luke (on behalf of the Apache Kafka PMC)
> >
> >
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Chia-Ping Tsai]

Congratulations Kamal!!!

Satish Duggana  於 2024年9月30日 週一 下午9:01寫道：

> Congratulations Kamal!
>
> On Mon, 30 Sept 2024 at 18:18, Josep Prat 
> wrote:
>
> > Congrats Kamal!
> >
> > On Mon, Sep 30, 2024 at 2:46 PM  wrote:
> >
> > > Congrats Kamal!
> > >
> > > -
> > > Gaurav
> > >
> > > > On 30 Sep 2024, at 18:10, Mickael Maison 
> > > wrote:
> > > >
> > > > Congratulations Kamal!
> > > >
> > > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
> > > >>
> > > >> Hi all,
> > > >>
> > > >> The PMC of Apache Kafka is pleased to announce a new Kafka
> committer,
> > > Kamal
> > > >> Chandraprakash.
> > > >>
> > > >> Kamal has been a Kafka contributor since May 2017. He has made
> > > significant
> > > >> contributions to the tiered storage feature (KIP-405). He authored
> > > KIP-1018
> > > >> and KIP-1075 which improved tiered storage operation. He also
> > > contributed
> > > >> to discussing and reviewing many KIPs.
> > > >>
> > > >> Congratulations, Kamal!
> > > >>
> > > >> Thanks,
> > > >> Luke (on behalf of the Apache Kafka PMC)
> > >
> > >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Lianet M.]

Congratulations Kamal!!!



On Mon, Sep 30, 2024, 9:22 a.m. Chia-Ping Tsai  wrote:

> Congratulations Kamal!!!
>
> Satish Duggana  於 2024年9月30日 週一 下午9:01寫道：
>
> > Congratulations Kamal!
> >
> > On Mon, 30 Sept 2024 at 18:18, Josep Prat 
> > wrote:
> >
> > > Congrats Kamal!
> > >
> > > On Mon, Sep 30, 2024 at 2:46 PM  wrote:
> > >
> > > > Congrats Kamal!
> > > >
> > > > -
> > > > Gaurav
> > > >
> > > > > On 30 Sep 2024, at 18:10, Mickael Maison  >
> > > > wrote:
> > > > >
> > > > > Congratulations Kamal!
> > > > >
> > > > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen 
> wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> The PMC of Apache Kafka is pleased to announce a new Kafka
> > committer,
> > > > Kamal
> > > > >> Chandraprakash.
> > > > >>
> > > > >> Kamal has been a Kafka contributor since May 2017. He has made
> > > > significant
> > > > >> contributions to the tiered storage feature (KIP-405). He authored
> > > > KIP-1018
> > > > >> and KIP-1075 which improved tiered storage operation. He also
> > > > contributed
> > > > >> to discussing and reviewing many KIPs.
> > > > >>
> > > > >> Congratulations, Kamal!
> > > > >>
> > > > >> Thanks,
> > > > >> Luke (on behalf of the Apache Kafka PMC)
> > > >
> > > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven
> 
> *
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > Anna Richardson, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Yash Mayya]

Congratulations Kamal!

On Mon, 30 Sept, 2024, 18:07 Luke Chen,  wrote:

> Hi all,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal
> Chandraprakash.
>
> Kamal has been a Kafka contributor since May 2017. He has made significant
> contributions to the tiered storage feature (KIP-405). He authored KIP-1018
> and KIP-1075 which improved tiered storage operation. He also contributed
> to discussing and reviewing many KIPs.
>
> Congratulations, Kamal!
>
> Thanks,
> Luke (on behalf of the Apache Kafka PMC)
>

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[TengYao Chi]

Hello everyone,

I'm also +1 on using the fluent API and having the `with` prefix in setter
method names.

Regarding Matthias' point, I agree with Sophie that we should keep the
`CloseOptions` classes separate.
These two `CloseOptions` serve different purposes, and while they may
occasionally share some similarities at the moment, keeping them separate
allows more flexibility for their own future changes. This way, each class
can evolve independently based on the specific needs of the Consumer and
Kafka Streams without introducing unnecessary complexity.

Best,
TengYao

Matthias J. Sax  於 2024年10月1日 週二 上午7:38寫道：

> Sophie, yes, that a fair summary, and yes, it was only an alternative
> idea for the case that people think, allowing to disable leave-group
> request for the plain consumer is not desirable. Seems we are actually
> on the same page.
>
> (And yes, it was meant for this thread, not KIP-1094...)
>
>
>
> On 9/30/24 4:32 PM, Matthias J. Sax wrote:
> > Kirk,
> >
> > I think good API design principle is to expose the minimum require API
> > to users, and users don't need getters, that's why we don't have any
> > getters in the KS config object classes. Getters are only needed
> > internally.
> >
> >  From an impl POV, the internal member can be either (1) package-private
> > to allow direct access within the same package, or (2) protected in
> > combination with an internal sub-class in an internal package to add the
> > necessary getters. As an example cf `Consumed` and `ConsumedInternal`
> >
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Consumed.java
> >
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/ConsumedInternal.java
> >
> > It provides a clean separation of user-facing public API, vs internal
> APIs.
> >
> >
> >
> > For `timeout()` given that it takes a `Duration` argument (not a
> > `long`), I believe the current name is correct?
> >
> >
> > The like the idea of using an enum for send-leave-group request flag.
> >
> >
> > For `Optional`, if we remove the public getters, the question goes away.
> >
> >
> >
> > About merging both `CloseOptions`: fine with me to keep them separated.
> > Was just an idea :)
> >
> >
> >
> > -Matthias
> >
> >
> > On 9/30/24 3:14 PM, Sophie Blee-Goldman wrote:
> >> +1 to using the fluent API and including "with" in the setter names.
> >>
> >> I also think Matthias raised a good point, that maybe it would be a good
> >> time to "fix" this issue in the CloseOptions for the KafkaStreams#close
> >> method, to conform to this API format. As for his other "random idea"
> >> about
> >> combining the two: Personally I would prefer to keep a separate
> >> CloseOptions class for the Consumer vs for Kafka Streams, since they
> will
> >> not necessarily always have the same exact semantics and parameters.
> Even
> >> if the API ends up looking more or less the same right now -- although
> >> consider that the "default" behavior is different for the consumer vs
> >> Kafka
> >> Streams, if leaveGroup is not specified in the CloseOptions then the
> >> consumer will opt to leave the group in some cases whereas Streams never
> >> will. So already there is some difference between the Consumer and
> >> Streams
> >> CloseOptions, so I'd rather not combine them
> >>
> >> On Mon, Sep 30, 2024 at 11:16 AM Chia-Ping Tsai 
> >> wrote:
> >>
>  Overall, the "static method builder" pattern seems better to me, and
> >>> thus I would prefer to make it the "gold standard" and we can see
> >>> what  we
> >>> can do for `Admin API` mid/long term?
> >>>
> >>> Since we want to avoid complicated compatibility issues, adding a
> static
> >>> method builder to Admin options seems more acceptable.
> >>>
> >>> However, the naming convention for the "setter" method might be a
> >>> concern.
> >>> Kafka Streams (KS) uses "withXXX," while Admin options do not include
> >>> the
> >>> "with" prefix. Additionally, the RPC requests follow the setXXX naming
> >>> convention.
> >>>
> >>> I’m unsure if this is the right time to align the fluent pattern naming
> >>> across the entire Kafka project, but it would be great if we could
> >>> reach a
> >>> consensus on "gold standard".
> >>>
> >>> I'm +1 to using the static method builder and `with` prefix due to
> >>> following advantages.
> >>> -
> >>>
> >>> 1. A static method builder allows making the constructor private,
> >>> which can
> >>> prevent unintended inheritance
> >>> -
> >>>
> >>> 2. The "with" prefix makes it easier to search for setters within the
> >>> option class
> >>>
> >>> Any feedback?
> >>>
>  Btw: I was also wondering, if we should re-use the new consumer
> >>> `CloseOption` class for `KafkaStreams#close()` and deprecate the KS
> >>> `CloseOption`
> >>> class? Not sure. Just another "random" idea.
> >>>
> >>> I guess it needs another KIP after this KIP gets merged.
> >>>
> >>>
> >>> Best,
> >>>
> >>> Chia-Ping
> >>

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Chia-Ping Tsai]

> Overall, the "static method builder" pattern seems better to me, and
thus I would prefer to make it the "gold standard" and we can see what  we
can do for `Admin API` mid/long term?

Since we want to avoid complicated compatibility issues, adding a static
method builder to Admin options seems more acceptable.

However, the naming convention for the "setter" method might be a concern.
Kafka Streams (KS) uses "withXXX," while Admin options do not include the
"with" prefix. Additionally, the RPC requests follow the setXXX naming
convention.

I’m unsure if this is the right time to align the fluent pattern naming
across the entire Kafka project, but it would be great if we could reach a
consensus on "gold standard".

I'm +1 to using the static method builder and `with` prefix due to
following advantages.
-

1. A static method builder allows making the constructor private, which can
prevent unintended inheritance
-

2. The "with" prefix makes it easier to search for setters within the
option class

Any feedback?

> Btw: I was also wondering, if we should re-use the new consumer
`CloseOption` class for `KafkaStreams#close()` and deprecate the KS
`CloseOption`
class? Not sure. Just another "random" idea.

I guess it needs another KIP after this KIP gets merged.


Best,

Chia-Ping





Matthias J. Sax  於 2024年10月1日 週二 上午1:16寫道：

> I am also in favor of consistent APIs. That's very good point. I did not
> take `Admin` API into account, and I am not aware that consumer /
> producer would have config object classes?
>
> Seems we are in a tricky situation here, because "consistent API" to me
> means producer, consumer, admin and KS.
>
> The KS surface area might be the largest one (even if there is a long
> list of `XxxOption` classes for Admin API), and we do follow the pattern
> as described. It's of course not desirable to just change the whole
> Admin API (and neither the KS API), but it might be good to agree on a
> "gold standard" and do everything new accordingly?
>
> Given that producer / consumer do not have any config object classes
> yet, it seems to be the question if they should follow the Admin pattern
> or the KS pattern. -- I tend to think, following the KS pattern might be
> better? But yes, I see your POV to say it should follow the Admin API
> pattern, however, it would imply that we "need" to change all the KS APIs.
>
> Overall, the "static method builder" pattern seems better to me, and
> thus I would prefer to make it the "gold standard" and we can see what
> we can do for `Admin API` mid/long term?
>
>
> Let's see what others think.
>
>
> Btw: I was also wondering, if we should re-use the new consumer
> `CloseOption` class for `KafkaStreams#close()` and deprecate the KS
> `CloseOption` class? Not sure. Just another "random" idea.
>
>
>
> -Matthias
>
>
> On 9/29/24 12:08 PM, Chia-Ping Tsai wrote:
> >>  From an API POV, I think the new `CloseOptions` class should not have
> > any "getters" and thus, it's irrelevant how we represent the different
> > cases in code internally (even if I believe using `Optional` might be a
> > good way to handle it).
> >
> > If we choose to avoid using getters, consumers would have to access the
> > internal variables directly, similar to how it's done in the streams
> > CloseOptions. While this is a matter of preference, it's worth noting
> that
> > Admin options do provide both setters and getters. In my opinion, since
> all
> > of these options are part of the kafka-clients code, they should adhere
> to
> > a consistent coding style.
> >
> >> In KS, we use config objects like `CloseOption` all the time, with
> static
> > "factory" method (and private constructors), and an internal  sub-class
> > which would have the getters if needed. So I think we should have
> >
> > I value the static factory methods for their convenience, allowing users
> to
> > quickly create an object when they want to set only one option. However,
> > while I don't want to sound repetitive, maintaining a consistent pattern
> > across the API is essential.
> >
> > Best,
> > Chia-Ping
> >
> >
> > Matthias J. Sax  於 2024年9月30日 週一 上午2:26寫道：
> >
> >> I am not sure, but I get the impression that we are starting to talk too
> >> much about implementation details now?
> >>
> >>   From an API POV, I think the new `CloseOptions` class should not have
> >> any "getters" and thus, it's irrelevant how we represent the different
> >> cases in code internally (even if I believe using `Optional` might be a
> >> good way to handle it).
> >>
> >> In KS, we use config objects like `CloseOption` all the time, with
> >> static "factory" method (and private constructors), and an internal
> >> sub-class which would have the getters if needed. So I think we should
> have
> >>
> >> public class CloseOptions {
> >>   private CloseOptions(Optional, Optional);
> >>
> >>   public static CloseOptions timeout(Duration);
> >>
> >>   public static CloseOptions leaveGroup(boolean);
> >>
> >>   public CloseO

[jira] [Resolved] (KAFKA-17670) stray partition deletion

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-17670.
--
Resolution: Duplicate

> stray partition deletion
> 
>
> Key: KAFKA-17670
> URL: https://issues.apache.org/jira/browse/KAFKA-17670
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin McCabe
>Priority: Major
>
> enable.stray.partition.deletion should be true by default on CP



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Viktor Somogyi-Vass]

Congrats Kamal! :)

On Mon, Sep 30, 2024, 19:21 Matthias J. Sax  wrote:

> Congrats!
>
> On 9/30/24 6:59 AM, Yash Mayya wrote:
> > Congratulations Kamal!
> >
> > On Mon, 30 Sept, 2024, 18:07 Luke Chen,  wrote:
> >
> >> Hi all,
> >>
> >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> Kamal
> >> Chandraprakash.
> >>
> >> Kamal has been a Kafka contributor since May 2017. He has made
> significant
> >> contributions to the tiered storage feature (KIP-405). He authored
> KIP-1018
> >> and KIP-1075 which improved tiered storage operation. He also
> contributed
> >> to discussing and reviewing many KIPs.
> >>
> >> Congratulations, Kamal!
> >>
> >> Thanks,
> >> Luke (on behalf of the Apache Kafka PMC)
> >>
> >
>

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Kirk True]

Hi TengYao,

Thanks for all the updates.

I took another look at the KIP and had some more questions:

KT4: Can we change the timeout() method to timeoutMs() to be more consistent?
KT5: Can we add the behavior when a null CloseOptions is provided?
KT6: Can we use "direct" values in CloseOptions vs. wrapping them in Optional? 
I understand the intent of using Optional to convey use of the default. 
However, I don't see that pattern used in the AdminClient Options classes which 
it seems like we're trying to emulate here. 
KT7: Have we considered using an enum to specify the behavior for the leave 
group? I'm thinking something like:

public enum GroupMembershipOptions {LEAVE_GROUP, REMAIN_IN_GROUP, DEFAULT}? 

KT8: Can we change the wording in the Test Plan section to specify that the 
default timeout should be 30 seconds and not Long.MAX_VALUE?

Thanks,
Kirk

On Mon, Sep 30, 2024, at 1:44 PM, Kirk True wrote:
> Hi all,
> 
> On Sun, Sep 29, 2024, at 12:08 PM, Chia-Ping Tsai wrote:
> > > From an API POV, I think the new `CloseOptions` class should not have
> > any "getters" and thus, it's irrelevant how we represent the different
> > cases in code internally (even if I believe using `Optional` might be a
> > good way to handle it).
> > 
> > If we choose to avoid using getters, consumers would have to access the
> > internal variables directly, similar to how it's done in the streams
> > CloseOptions. While this is a matter of preference, it's worth noting that
> > Admin options do provide both setters and getters. In my opinion, since all
> > of these options are part of the kafka-clients code, they should adhere to
> > a consistent coding style.
> 
> I'm in favor of adding getters to the CloseOptions class. I'm envisaging 
> CloseOptions as a Java Record/struct that is created by the user with the 
> desired values and inspected by the Consumer implementation.
> 
> Is there a downside to exposing getters?
> 
> Thanks,
> Kirk
> 
> > > In KS, we use config objects like `CloseOption` all the time, with static
> > "factory" method (and private constructors), and an internal  sub-class
> > which would have the getters if needed. So I think we should have
> > 
> > I value the static factory methods for their convenience, allowing users to
> > quickly create an object when they want to set only one option. However,
> > while I don't want to sound repetitive, maintaining a consistent pattern
> > across the API is essential.
> > 
> > Best,
> > Chia-Ping
> > 
> > 
> > Matthias J. Sax  於 2024年9月30日 週一 上午2:26寫道：
> > 
> > > I am not sure, but I get the impression that we are starting to talk too
> > > much about implementation details now?
> > >
> > >  From an API POV, I think the new `CloseOptions` class should not have
> > > any "getters" and thus, it's irrelevant how we represent the different
> > > cases in code internally (even if I believe using `Optional` might be a
> > > good way to handle it).
> > >
> > > In KS, we use config objects like `CloseOption` all the time, with
> > > static "factory" method (and private constructors), and an internal
> > > sub-class which would have the getters if needed. So I think we should 
> > > have
> > >
> > > public class CloseOptions {
> > >  private CloseOptions(Optional, Optional);
> > >
> > >  public static CloseOptions timeout(Duration);
> > >
> > >  public static CloseOptions leaveGroup(boolean);
> > >
> > >  public CloseOption withTimeout(Duration);
> > >
> > >  public CloseOption withLeaveGroup(boolean);
> > > }
> > >
> > >
> > > This allows to call as example:
> > >
> > >consumer.close(CloseOptions.leaveGroup(true));
> > >
> > > or
> > >
> > >consumer.close(
> > > CloseOptions.timeout(Duration.ofMinutes(5)
> > > .withLeaveGroup(false)
> > > );
> > >
> > > We can still discuss naming and what overloads we want to add, but in
> > > general, a well established and proven pattern is to have a few static
> > > "entry" methods, with return the object itself to allow chaining with
> > > non-static methods.
> > >
> > >
> > >
> > > I am not sure, why KIP-812 did not follow this pattern... I think it got
> > > it wrong and we should not repeat this "mistake". Maybe we could
> > > actually piggy-pack a cleanup for the existing KS CloseOption object
> > > into this KIP?
> > >
> > >
> > >
> > > -Matthias
> > >
> > > On 9/29/24 8:39 AM, TengYao Chi wrote:
> > > > Hi Chia-Ping,
> > > >
> > > > Thanks for your feedback.
> > > > What I intended to express is that if `Optional.empty()` is passed to 
> > > > the
> > > > `timeout`, it will eventually be converted to 
> > > > `DEFAULT_CLOSE_TIMEOUT_MS`,
> > > > just as you mentioned.
> > > > Apologies for not expressing that clearly and for any confusion caused.
> > > >
> > > > Best regards,
> > > > TengYao
> > > >
> > > > Chia-Ping Tsai  於 2024年9月29日 週日 下午10:16寫道：
> > > >
> > > >> hi TengYao
> > > >>
> > > >>> I have reviewed the `close()` method implementation for both the
>

Re: [DISCUSS] KIP-1094 Add a new constructor method with nextOffsets to ConsumerRecords

[Kirk True]

Hi Alieh,

Thanks for the KIP!

Questions:

KT1: In the second paragraph of the Motivation section, it mentions that in 
some cases the consumer does not return the correct next offset. Can you add a 
link to the bug here? I'm curious if that still happens with the "new" consumer.

KT2: There's a comment in the Fetch class' addRecords() method that I'm 
undecided if we need to be concerned about. The comment states that in some 
rare cases of partition leadership changing, we might issue fetches from 
multiple nodes for the same partition. In that rare case, it seems like the 
leader epoch wouldn't be the same for both fetch results, which means we 
wouldn't be able to represent the leader epoch as a single value. There's a 
really, really good chance I'm wringing my hands over this unnecessarily. And 
even if this does occur, it seems like a "last write wins" case that we'll end 
up returning the latest partition owner. Maybe that's all moot?

Thanks,
Kirk

On Mon, Sep 30, 2024, at 5:10 AM, Alieh Saeedi wrote:
> Hi all,
> 
> thanks for the feedbacks.
> Since there is consensus in deprecating the current constructor, we can
> deprecate it for now and remove it later on (in 5.0 maybe).
> The KIP is updated.
> 
> Thanks,
> Alieh
> 
> On Sun, Sep 29, 2024 at 6:22 AM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
> 
> > @sophie I meant AK tests. But you make a fair point. There will not be
> > that many instances I'm sure, so if the consensus is to deprecate the
> > old constructor, that's fine with me.
> >
> > 
> > From: Sophie Blee-Goldman 
> > Sent: 28 September 2024 22:56
> > To: dev@kafka.apache.org 
> > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > nextOffsets to ConsumerRecords
> >
> > @Andrew do you mean user tests might create ConsumerRecords, or are you
> > concerned about AK tests?
> >
> > I guess Alieh would be the one to implement this, so I'll leave it up to
> > her to investigate the current usage of the existing constructor and
> > whether it feels like too much trivial work to remove it from our own AK
> > tests. Otherwise, on pure API design philosophy grounds, I would prefer to
> > deprecate and remove the old one.
> >
> > However if we feel that users have reason to create ConsumerRecords for
> > tests then it's probably not worth the upheaval to deprecate the old
> > constructor.
> >
> > On Fri, Sep 27, 2024 at 2:38 PM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> > > Hi Matthias,
> > > Tests will also create ConsumerRecords. I don't see the point in forcing
> > a
> > > bunch of trivial changes to tests, that's all.
> > > Of course, it's not too hard to tweak the tests, but it's making work.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > 
> > > From: Matthias J. Sax 
> > > Sent: 27 September 2024 18:36
> > > To: dev@kafka.apache.org 
> > > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > > nextOffsets to ConsumerRecords
> > >
> > > Thanks for the KIP Alieh.
> > >
> > > Deprecating the existing constructor won't break anybody atm, right? We
> > > don't remove the constructor yet. Of course, in 5.0 in the future we
> > > would apply this breaking change.
> > >
> > > While `ConsumerRecord` is a public API, it seems that actual user code
> > > should never create a `ConsumerRecord` object? In general (of course
> > > this is not in scope of this KIP) I am wondering why we have a public
> > > constructor to begin with (I assume it's some Java limitations how
> > > visibility is managed). Having said this, I would also tend to deprecate
> > > the old constructor.
> > >
> > > @Andrew: Why do you prefer to keep the old constructor? To me it seems
> > > to be an potential source of bugs, to keep the old one?
> > >
> > >
> > > -Matthias
> > >
> > > On 9/27/24 2:19 AM, Andrew Schofield wrote:
> > > > Hi Alieh,
> > > > Thanks for the KIP. Looks like a nice addition to me. I prefer not
> > > deprecating the existing
> > > > constructor too.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > 
> > > > From: Alieh Saeedi 
> > > > Sent: 27 September 2024 09:09
> > > > To: dev@kafka.apache.org 
> > > > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > > nextOffsets to ConsumerRecords
> > > >
> > > > Thank you, Bill and Sophie.
> > > >
> > > > @Bill: You are right. I updated the KIP.
> > > > @Sophie: Yes, it was also our concern, but we thought we should keep
> > the
> > > > current constructor in order to not break the user's code who has
> > called
> > > > this constructor in their code.
> > > >
> > > > Thanks,
> > > > Alieh
> > > >
> > > > On Fri, Sep 27, 2024 at 1:07 AM Sophie Blee-Goldman <
> > > sop...@responsive.dev>
> > > > wrote:
> > > >
> > > >> Should we deprecate the old constructor to make sure that all info
> > gets
> > > >> passed in when creating a ConsumerRecords inst

[jira] [Created] (KAFKA-17670) enable.stray.partition.deletion should be true by default on CP

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-17670:


 Summary: enable.stray.partition.deletion should be true by default 
on CP
 Key: KAFKA-17670
 URL: https://issues.apache.org/jira/browse/KAFKA-17670
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe


enable.stray.partition.deletion should be true by default on CP



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Kirk True]

Hi all,

On Sun, Sep 29, 2024, at 12:08 PM, Chia-Ping Tsai wrote:
> > From an API POV, I think the new `CloseOptions` class should not have
> any "getters" and thus, it's irrelevant how we represent the different
> cases in code internally (even if I believe using `Optional` might be a
> good way to handle it).
> 
> If we choose to avoid using getters, consumers would have to access the
> internal variables directly, similar to how it's done in the streams
> CloseOptions. While this is a matter of preference, it's worth noting that
> Admin options do provide both setters and getters. In my opinion, since all
> of these options are part of the kafka-clients code, they should adhere to
> a consistent coding style.

I'm in favor of adding getters to the CloseOptions class. I'm envisaging 
CloseOptions as a Java Record/struct that is created by the user with the 
desired values and inspected by the Consumer implementation.

Is there a downside to exposing getters?

Thanks,
Kirk

> > In KS, we use config objects like `CloseOption` all the time, with static
> "factory" method (and private constructors), and an internal  sub-class
> which would have the getters if needed. So I think we should have
> 
> I value the static factory methods for their convenience, allowing users to
> quickly create an object when they want to set only one option. However,
> while I don't want to sound repetitive, maintaining a consistent pattern
> across the API is essential.
> 
> Best,
> Chia-Ping
> 
> 
> Matthias J. Sax  於 2024年9月30日 週一 上午2:26寫道：
> 
> > I am not sure, but I get the impression that we are starting to talk too
> > much about implementation details now?
> >
> >  From an API POV, I think the new `CloseOptions` class should not have
> > any "getters" and thus, it's irrelevant how we represent the different
> > cases in code internally (even if I believe using `Optional` might be a
> > good way to handle it).
> >
> > In KS, we use config objects like `CloseOption` all the time, with
> > static "factory" method (and private constructors), and an internal
> > sub-class which would have the getters if needed. So I think we should have
> >
> > public class CloseOptions {
> >  private CloseOptions(Optional, Optional);
> >
> >  public static CloseOptions timeout(Duration);
> >
> >  public static CloseOptions leaveGroup(boolean);
> >
> >  public CloseOption withTimeout(Duration);
> >
> >  public CloseOption withLeaveGroup(boolean);
> > }
> >
> >
> > This allows to call as example:
> >
> >consumer.close(CloseOptions.leaveGroup(true));
> >
> > or
> >
> >consumer.close(
> > CloseOptions.timeout(Duration.ofMinutes(5)
> > .withLeaveGroup(false)
> > );
> >
> > We can still discuss naming and what overloads we want to add, but in
> > general, a well established and proven pattern is to have a few static
> > "entry" methods, with return the object itself to allow chaining with
> > non-static methods.
> >
> >
> >
> > I am not sure, why KIP-812 did not follow this pattern... I think it got
> > it wrong and we should not repeat this "mistake". Maybe we could
> > actually piggy-pack a cleanup for the existing KS CloseOption object
> > into this KIP?
> >
> >
> >
> > -Matthias
> >
> > On 9/29/24 8:39 AM, TengYao Chi wrote:
> > > Hi Chia-Ping,
> > >
> > > Thanks for your feedback.
> > > What I intended to express is that if `Optional.empty()` is passed to the
> > > `timeout`, it will eventually be converted to `DEFAULT_CLOSE_TIMEOUT_MS`,
> > > just as you mentioned.
> > > Apologies for not expressing that clearly and for any confusion caused.
> > >
> > > Best regards,
> > > TengYao
> > >
> > > Chia-Ping Tsai  於 2024年9月29日 週日 下午10:16寫道：
> > >
> > >> hi TengYao
> > >>
> > >>> I have reviewed the `close()` method implementation for both the
> > Classic
> > >> and Async Consumers. I believe the `timeout` parameter could have a
> > default
> > >> value, and this default should align with the existing
> > `Consumer#close()`
> > >> method, which internally calls the overloaded `Consumer#close(Duration)`
> > >> with a default of 30 seconds (`DEFAULT_CLOSE_TIMEOUT_MS`).
> > >>
> > >> If you assign a default value (30s) to CloseOptions#timeout, the
> > consumer
> > >> won't be able to differentiate between the "default" and "user-defined"
> > >> behaviors.
> > >>
> > >> Therefore, I prefer to leave the timeout empty as the default value,
> > >> allowing the consumer to handle it in a way that reflects default
> > behavior.
> > >> Best,
> > >> Chia-Ping
> > >>
> > >>
> > >>
> > >> TengYao Chi  於 2024年9月29日 週日 下午2:23寫道：
> > >>
> > >>> Hi Sophie,
> > >>>
> > >>> Thanks for the suggestions.
> > >>>
> > >>> I have reviewed the `close()` method implementation for both the
> > Classic
> > >>> and Async Consumers. I believe the `timeout` parameter could have a
> > >> default
> > >>> value, and this default should align with the existing
> > `Consumer#close()`
> > >>> method, which internally calls the 

Re: [DISCUSS] KIP-1090 Flaky Tests 👻

[David Arthur]

Jun,

1) The reports mentioned in the KIP will need to be built. As a start, I
think we can use a cron-based GitHub Action that produces a markdown
report. Longer term, we can maybe look into some static site generator like
GitHub Pages for hosting weekly reports.

2) In my opinion, any test which could possibly be flaky should be an
integration test. For example, any test using a real network, system time,
threads, etc, should be marked as an integration test.

-David

On Mon, Sep 30, 2024 at 12:28 PM Jun Rao  wrote:

> Hi, David,
>
> Thanks for the report.
>
> 1. Where could we see the reports (quarantined tests, etc) mentioned in the
> KIP?
> 2. The quarantine process seems to only apply to integration tests. What's
> our recommendation for flaky unit tests?
>
> Jun
>
> On Thu, Sep 26, 2024 at 1:34 PM David Arthur  wrote:
>
> > If there is no more feedback on this, I'll go ahead and move to a vote.
> >
> > -David
> >
> > On Sun, Sep 22, 2024 at 11:04 AM Chia-Ping Tsai 
> > wrote:
> >
> > >
> > >
> > > > David Arthur  於 2024年9月22日 晚上10:07 寫道：
> > > >
> > > > Q2: Yes, I think we should run the quarantined tests on all CI
> builds,
> > > PRs
> > > > and trunk. We can achieve this with --rerun-tasks. This will let PR
> > > authors
> > > > gain feedback about their changes affect on the flaky tests. We could
> > > even
> > > > create a PR-specific report that shows if their changes improved or
> > > > worsened the flakiness of the quarantined tests.
> > >
> > > I guess it will be an individual status like “Gradle Build Scan”? The
> > > failure of quarantined tests does not obstruct us from merging PR
> unless
> > > the target of PR is to fix specific flaky.
> > >
> > > Thanks,
> > > Chia-Ping
> >
> >
> >
> > --
> > David Arthur
> >
>


-- 
David Arthur

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Matthias J. Sax]

Kirk,

I think good API design principle is to expose the minimum require API 
to users, and users don't need getters, that's why we don't have any 
getters in the KS config object classes. Getters are only needed internally.


From an impl POV, the internal member can be either (1) package-private 
to allow direct access within the same package, or (2) protected in 
combination with an internal sub-class in an internal package to add the 
necessary getters. As an example cf `Consumed` and `ConsumedInternal`


https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Consumed.java

https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/ConsumedInternal.java

It provides a clean separation of user-facing public API, vs internal APIs.



For `timeout()` given that it takes a `Duration` argument (not a 
`long`), I believe the current name is correct?



The like the idea of using an enum for send-leave-group request flag.


For `Optional`, if we remove the public getters, the question goes away.



About merging both `CloseOptions`: fine with me to keep them separated. 
Was just an idea :)




-Matthias


On 9/30/24 3:14 PM, Sophie Blee-Goldman wrote:

+1 to using the fluent API and including "with" in the setter names.

I also think Matthias raised a good point, that maybe it would be a good
time to "fix" this issue in the CloseOptions for the KafkaStreams#close
method, to conform to this API format. As for his other "random idea" about
combining the two: Personally I would prefer to keep a separate
CloseOptions class for the Consumer vs for Kafka Streams, since they will
not necessarily always have the same exact semantics and parameters. Even
if the API ends up looking more or less the same right now -- although
consider that the "default" behavior is different for the consumer vs Kafka
Streams, if leaveGroup is not specified in the CloseOptions then the
consumer will opt to leave the group in some cases whereas Streams never
will. So already there is some difference between the Consumer and Streams
CloseOptions, so I'd rather not combine them

On Mon, Sep 30, 2024 at 11:16 AM Chia-Ping Tsai  wrote:


Overall, the "static method builder" pattern seems better to me, and

thus I would prefer to make it the "gold standard" and we can see what  we
can do for `Admin API` mid/long term?

Since we want to avoid complicated compatibility issues, adding a static
method builder to Admin options seems more acceptable.

However, the naming convention for the "setter" method might be a concern.
Kafka Streams (KS) uses "withXXX," while Admin options do not include the
"with" prefix. Additionally, the RPC requests follow the setXXX naming
convention.

I’m unsure if this is the right time to align the fluent pattern naming
across the entire Kafka project, but it would be great if we could reach a
consensus on "gold standard".

I'm +1 to using the static method builder and `with` prefix due to
following advantages.
-

1. A static method builder allows making the constructor private, which can
prevent unintended inheritance
-

2. The "with" prefix makes it easier to search for setters within the
option class

Any feedback?


Btw: I was also wondering, if we should re-use the new consumer

`CloseOption` class for `KafkaStreams#close()` and deprecate the KS
`CloseOption`
class? Not sure. Just another "random" idea.

I guess it needs another KIP after this KIP gets merged.


Best,

Chia-Ping





Matthias J. Sax  於 2024年10月1日 週二 上午1:16寫道：


I am also in favor of consistent APIs. That's very good point. I did not
take `Admin` API into account, and I am not aware that consumer /
producer would have config object classes?

Seems we are in a tricky situation here, because "consistent API" to me
means producer, consumer, admin and KS.

The KS surface area might be the largest one (even if there is a long
list of `XxxOption` classes for Admin API), and we do follow the pattern
as described. It's of course not desirable to just change the whole
Admin API (and neither the KS API), but it might be good to agree on a
"gold standard" and do everything new accordingly?

Given that producer / consumer do not have any config object classes
yet, it seems to be the question if they should follow the Admin pattern
or the KS pattern. -- I tend to think, following the KS pattern might be
better? But yes, I see your POV to say it should follow the Admin API
pattern, however, it would imply that we "need" to change all the KS

APIs.


Overall, the "static method builder" pattern seems better to me, and
thus I would prefer to make it the "gold standard" and we can see what
we can do for `Admin API` mid/long term?


Let's see what others think.


Btw: I was also wondering, if we should re-use the new consumer
`CloseOption` class for `KafkaStreams#close()` and deprecate the KS
`CloseOption` class? Not sure. Just another "random" idea.



-Matthias


On 9/29/

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Matthias J. Sax]

Sophie, yes, that a fair summary, and yes, it was only an alternative 
idea for the case that people think, allowing to disable leave-group 
request for the plain consumer is not desirable. Seems we are actually 
on the same page.


(And yes, it was meant for this thread, not KIP-1094...)



On 9/30/24 4:32 PM, Matthias J. Sax wrote:

Kirk,

I think good API design principle is to expose the minimum require API 
to users, and users don't need getters, that's why we don't have any 
getters in the KS config object classes. Getters are only needed 
internally.


 From an impl POV, the internal member can be either (1) package-private 
to allow direct access within the same package, or (2) protected in 
combination with an internal sub-class in an internal package to add the 
necessary getters. As an example cf `Consumed` and `ConsumedInternal`


https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Consumed.java

https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/ConsumedInternal.java

It provides a clean separation of user-facing public API, vs internal APIs.



For `timeout()` given that it takes a `Duration` argument (not a 
`long`), I believe the current name is correct?



The like the idea of using an enum for send-leave-group request flag.


For `Optional`, if we remove the public getters, the question goes away.



About merging both `CloseOptions`: fine with me to keep them separated. 
Was just an idea :)




-Matthias


On 9/30/24 3:14 PM, Sophie Blee-Goldman wrote:

+1 to using the fluent API and including "with" in the setter names.

I also think Matthias raised a good point, that maybe it would be a good
time to "fix" this issue in the CloseOptions for the KafkaStreams#close
method, to conform to this API format. As for his other "random idea" 
about

combining the two: Personally I would prefer to keep a separate
CloseOptions class for the Consumer vs for Kafka Streams, since they will
not necessarily always have the same exact semantics and parameters. Even
if the API ends up looking more or less the same right now -- although
consider that the "default" behavior is different for the consumer vs 
Kafka

Streams, if leaveGroup is not specified in the CloseOptions then the
consumer will opt to leave the group in some cases whereas Streams never
will. So already there is some difference between the Consumer and 
Streams

CloseOptions, so I'd rather not combine them

On Mon, Sep 30, 2024 at 11:16 AM Chia-Ping Tsai  
wrote:



Overall, the "static method builder" pattern seems better to me, and
thus I would prefer to make it the "gold standard" and we can see 
what  we

can do for `Admin API` mid/long term?

Since we want to avoid complicated compatibility issues, adding a static
method builder to Admin options seems more acceptable.

However, the naming convention for the "setter" method might be a 
concern.
Kafka Streams (KS) uses "withXXX," while Admin options do not include 
the

"with" prefix. Additionally, the RPC requests follow the setXXX naming
convention.

I’m unsure if this is the right time to align the fluent pattern naming
across the entire Kafka project, but it would be great if we could 
reach a

consensus on "gold standard".

I'm +1 to using the static method builder and `with` prefix due to
following advantages.
-

1. A static method builder allows making the constructor private, 
which can

prevent unintended inheritance
-

2. The "with" prefix makes it easier to search for setters within the
option class

Any feedback?


Btw: I was also wondering, if we should re-use the new consumer

`CloseOption` class for `KafkaStreams#close()` and deprecate the KS
`CloseOption`
class? Not sure. Just another "random" idea.

I guess it needs another KIP after this KIP gets merged.


Best,

Chia-Ping





Matthias J. Sax  於 2024年10月1日 週二 上午1:16寫道：

I am also in favor of consistent APIs. That's very good point. I did 
not

take `Admin` API into account, and I am not aware that consumer /
producer would have config object classes?

Seems we are in a tricky situation here, because "consistent API" to me
means producer, consumer, admin and KS.

The KS surface area might be the largest one (even if there is a long
list of `XxxOption` classes for Admin API), and we do follow the 
pattern

as described. It's of course not desirable to just change the whole
Admin API (and neither the KS API), but it might be good to agree on a
"gold standard" and do everything new accordingly?

Given that producer / consumer do not have any config object classes
yet, it seems to be the question if they should follow the Admin 
pattern
or the KS pattern. -- I tend to think, following the KS pattern 
might be

better? But yes, I see your POV to say it should follow the Admin API
pattern, however, it would imply that we "need" to change all the KS

APIs.


Overall, the "static method builder" pattern seems better to me, and

Re: [DISCUSS] KIP-1094 Add a new constructor method with nextOffsets to ConsumerRecords

[Sophie Blee-Goldman]

@Matthias I suspect that last email was meant for KIP-1092? But glad we are
on the same page :P

Seems we're on the same page here as well. Not sure if there are any other
open questions, otherwise we can maybe move to a vote?


On Mon, Sep 30, 2024 at 5:11 AM Alieh Saeedi 
wrote:

> Hi all,
>
> thanks for the feedbacks.
> Since there is consensus in deprecating the current constructor, we can
> deprecate it for now and remove it later on (in 5.0 maybe).
> The KIP is updated.
>
> Thanks,
> Alieh
>
> On Sun, Sep 29, 2024 at 6:22 AM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
> > @sophie I meant AK tests. But you make a fair point. There will not be
> > that many instances I'm sure, so if the consensus is to deprecate the
> > old constructor, that's fine with me.
> >
> > 
> > From: Sophie Blee-Goldman 
> > Sent: 28 September 2024 22:56
> > To: dev@kafka.apache.org 
> > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > nextOffsets to ConsumerRecords
> >
> > @Andrew do you mean user tests might create ConsumerRecords, or are you
> > concerned about AK tests?
> >
> > I guess Alieh would be the one to implement this, so I'll leave it up to
> > her to investigate the current usage of the existing constructor and
> > whether it feels like too much trivial work to remove it from our own AK
> > tests. Otherwise, on pure API design philosophy grounds, I would prefer
> to
> > deprecate and remove the old one.
> >
> > However if we feel that users have reason to create ConsumerRecords for
> > tests then it's probably not worth the upheaval to deprecate the old
> > constructor.
> >
> > On Fri, Sep 27, 2024 at 2:38 PM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> > > Hi Matthias,
> > > Tests will also create ConsumerRecords. I don't see the point in
> forcing
> > a
> > > bunch of trivial changes to tests, that's all.
> > > Of course, it's not too hard to tweak the tests, but it's making work.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > 
> > > From: Matthias J. Sax 
> > > Sent: 27 September 2024 18:36
> > > To: dev@kafka.apache.org 
> > > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > > nextOffsets to ConsumerRecords
> > >
> > > Thanks for the KIP Alieh.
> > >
> > > Deprecating the existing constructor won't break anybody atm, right? We
> > > don't remove the constructor yet. Of course, in 5.0 in the future we
> > > would apply this breaking change.
> > >
> > > While `ConsumerRecord` is a public API, it seems that actual user code
> > > should never create a `ConsumerRecord` object? In general (of course
> > > this is not in scope of this KIP) I am wondering why we have a public
> > > constructor to begin with (I assume it's some Java limitations how
> > > visibility is managed). Having said this, I would also tend to
> deprecate
> > > the old constructor.
> > >
> > > @Andrew: Why do you prefer to keep the old constructor? To me it seems
> > > to be an potential source of bugs, to keep the old one?
> > >
> > >
> > > -Matthias
> > >
> > > On 9/27/24 2:19 AM, Andrew Schofield wrote:
> > > > Hi Alieh,
> > > > Thanks for the KIP. Looks like a nice addition to me. I prefer not
> > > deprecating the existing
> > > > constructor too.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > 
> > > > From: Alieh Saeedi 
> > > > Sent: 27 September 2024 09:09
> > > > To: dev@kafka.apache.org 
> > > > Subject: Re: [DISCUSS] KIP-1094 Add a new constructor method with
> > > nextOffsets to ConsumerRecords
> > > >
> > > > Thank you, Bill and Sophie.
> > > >
> > > > @Bill: You are right. I updated the KIP.
> > > > @Sophie: Yes, it was also our concern, but we thought we should keep
> > the
> > > > current constructor in order to not break the user's code who has
> > called
> > > > this constructor in their code.
> > > >
> > > > Thanks,
> > > > Alieh
> > > >
> > > > On Fri, Sep 27, 2024 at 1:07 AM Sophie Blee-Goldman <
> > > sop...@responsive.dev>
> > > > wrote:
> > > >
> > > >> Should we deprecate the old constructor to make sure that all info
> > gets
> > > >> passed in when creating a ConsumerRecords instance?
> > > >>
> > > >> On Thu, Sep 26, 2024 at 3:37 PM Bill Bejeck 
> > wrote:
> > > >>
> > > >>> Hi Alieh,
> > > >>>
> > > >>> Thanks for the KIP, it will be very useful to Kafka Streams.
> > > >>> I have one comment.  In the "Proposed Changes" section, you mention
> > the
> > > >>> "The `nextOffsets` object contains the next offset and the last
> > leader
> > > >>> epoch per partition".
> > > >>> If understand the KIP correctly, it should be something along the
> > lines
> > > >> of
> > > >>> "The `nextOffsets` method returns a map of `TopicPartition` to
> > > >>> `OffsetAndMetadata` objects and  `OffsetAndMetadata` contains the
> > next
> > > >>> offset and the last leader epoch per partition"
> > > >>>
> > > >>> Other than that

[jira] [Created] (KAFKA-17672) Run quarantined tests separately

[David Arthur (Jira)]

David Arthur created KAFKA-17672:


 Summary: Run quarantined tests separately
 Key: KAFKA-17672
 URL: https://issues.apache.org/jira/browse/KAFKA-17672
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Arthur
Assignee: David Arthur






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17671) Create better documentation for transactions

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-17671:
--

 Summary: Create better documentation for transactions
 Key: KAFKA-17671
 URL: https://issues.apache.org/jira/browse/KAFKA-17671
 Project: Kafka
  Issue Type: Task
Reporter: Justine Olshan


There is not a single source of truth that explains how transactions should 
work in kafka.

We should create a document that explains how they work, the guarantees, and 
tips for how to design applications that use transactions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Sophie Blee-Goldman]

+1 to using the fluent API and including "with" in the setter names.

I also think Matthias raised a good point, that maybe it would be a good
time to "fix" this issue in the CloseOptions for the KafkaStreams#close
method, to conform to this API format. As for his other "random idea" about
combining the two: Personally I would prefer to keep a separate
CloseOptions class for the Consumer vs for Kafka Streams, since they will
not necessarily always have the same exact semantics and parameters. Even
if the API ends up looking more or less the same right now -- although
consider that the "default" behavior is different for the consumer vs Kafka
Streams, if leaveGroup is not specified in the CloseOptions then the
consumer will opt to leave the group in some cases whereas Streams never
will. So already there is some difference between the Consumer and Streams
CloseOptions, so I'd rather not combine them

On Mon, Sep 30, 2024 at 11:16 AM Chia-Ping Tsai  wrote:

> > Overall, the "static method builder" pattern seems better to me, and
> thus I would prefer to make it the "gold standard" and we can see what  we
> can do for `Admin API` mid/long term?
>
> Since we want to avoid complicated compatibility issues, adding a static
> method builder to Admin options seems more acceptable.
>
> However, the naming convention for the "setter" method might be a concern.
> Kafka Streams (KS) uses "withXXX," while Admin options do not include the
> "with" prefix. Additionally, the RPC requests follow the setXXX naming
> convention.
>
> I’m unsure if this is the right time to align the fluent pattern naming
> across the entire Kafka project, but it would be great if we could reach a
> consensus on "gold standard".
>
> I'm +1 to using the static method builder and `with` prefix due to
> following advantages.
> -
>
> 1. A static method builder allows making the constructor private, which can
> prevent unintended inheritance
> -
>
> 2. The "with" prefix makes it easier to search for setters within the
> option class
>
> Any feedback?
>
> > Btw: I was also wondering, if we should re-use the new consumer
> `CloseOption` class for `KafkaStreams#close()` and deprecate the KS
> `CloseOption`
> class? Not sure. Just another "random" idea.
>
> I guess it needs another KIP after this KIP gets merged.
>
>
> Best,
>
> Chia-Ping
>
>
>
>
>
> Matthias J. Sax  於 2024年10月1日 週二 上午1:16寫道：
>
> > I am also in favor of consistent APIs. That's very good point. I did not
> > take `Admin` API into account, and I am not aware that consumer /
> > producer would have config object classes?
> >
> > Seems we are in a tricky situation here, because "consistent API" to me
> > means producer, consumer, admin and KS.
> >
> > The KS surface area might be the largest one (even if there is a long
> > list of `XxxOption` classes for Admin API), and we do follow the pattern
> > as described. It's of course not desirable to just change the whole
> > Admin API (and neither the KS API), but it might be good to agree on a
> > "gold standard" and do everything new accordingly?
> >
> > Given that producer / consumer do not have any config object classes
> > yet, it seems to be the question if they should follow the Admin pattern
> > or the KS pattern. -- I tend to think, following the KS pattern might be
> > better? But yes, I see your POV to say it should follow the Admin API
> > pattern, however, it would imply that we "need" to change all the KS
> APIs.
> >
> > Overall, the "static method builder" pattern seems better to me, and
> > thus I would prefer to make it the "gold standard" and we can see what
> > we can do for `Admin API` mid/long term?
> >
> >
> > Let's see what others think.
> >
> >
> > Btw: I was also wondering, if we should re-use the new consumer
> > `CloseOption` class for `KafkaStreams#close()` and deprecate the KS
> > `CloseOption` class? Not sure. Just another "random" idea.
> >
> >
> >
> > -Matthias
> >
> >
> > On 9/29/24 12:08 PM, Chia-Ping Tsai wrote:
> > >>  From an API POV, I think the new `CloseOptions` class should not have
> > > any "getters" and thus, it's irrelevant how we represent the different
> > > cases in code internally (even if I believe using `Optional` might be a
> > > good way to handle it).
> > >
> > > If we choose to avoid using getters, consumers would have to access the
> > > internal variables directly, similar to how it's done in the streams
> > > CloseOptions. While this is a matter of preference, it's worth noting
> > that
> > > Admin options do provide both setters and getters. In my opinion, since
> > all
> > > of these options are part of the kafka-clients code, they should adhere
> > to
> > > a consistent coding style.
> > >
> > >> In KS, we use config objects like `CloseOption` all the time, with
> > static
> > > "factory" method (and private constructors), and an internal  sub-class
> > > which would have the getters if needed. So I think we should have
> > >
> > > I value the static factory methods for their convenie

[jira] [Created] (KAFKA-17673) Gradle Build Scan PR check not showing

[David Arthur (Jira)]

David Arthur created KAFKA-17673:


 Summary: Gradle Build Scan PR check not showing
 Key: KAFKA-17673
 URL: https://issues.apache.org/jira/browse/KAFKA-17673
 Project: Kafka
  Issue Type: Bug
  Components: build
Reporter: David Arthur
 Attachments: image-2024-09-30-22-24-30-190.png

After merging [https://github.com/apache/kafka/pull/17299,] the build scans 
links are broken on PRs.

For example:

!image-2024-09-30-22-24-30-190.png|width=719,height=199!

 

This might have to do with the change from the statuses API to the checks API.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Andrew Schofield]

+1 from me on the fluent API using the `with` prefix too.

Thanks,
Andrew


From: TengYao Chi 
Sent: 01 October 2024 05:23
To: dev@kafka.apache.org 
Subject: Re: [DISCUSS] KIP-1092: Extend Consumer#close with an option to leave 
the group or not

Hello everyone,

I'm also +1 on using the fluent API and having the `with` prefix in setter
method names.

Regarding Matthias' point, I agree with Sophie that we should keep the
`CloseOptions` classes separate.
These two `CloseOptions` serve different purposes, and while they may
occasionally share some similarities at the moment, keeping them separate
allows more flexibility for their own future changes. This way, each class
can evolve independently based on the specific needs of the Consumer and
Kafka Streams without introducing unnecessary complexity.

Best,
TengYao

Matthias J. Sax  於 2024年10月1日 週二 上午7:38寫道：

> Sophie, yes, that a fair summary, and yes, it was only an alternative
> idea for the case that people think, allowing to disable leave-group
> request for the plain consumer is not desirable. Seems we are actually
> on the same page.
>
> (And yes, it was meant for this thread, not KIP-1094...)
>
>
>
> On 9/30/24 4:32 PM, Matthias J. Sax wrote:
> > Kirk,
> >
> > I think good API design principle is to expose the minimum require API
> > to users, and users don't need getters, that's why we don't have any
> > getters in the KS config object classes. Getters are only needed
> > internally.
> >
> >  From an impl POV, the internal member can be either (1) package-private
> > to allow direct access within the same package, or (2) protected in
> > combination with an internal sub-class in an internal package to add the
> > necessary getters. As an example cf `Consumed` and `ConsumedInternal`
> >
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Consumed.java
> >
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/ConsumedInternal.java
> >
> > It provides a clean separation of user-facing public API, vs internal
> APIs.
> >
> >
> >
> > For `timeout()` given that it takes a `Duration` argument (not a
> > `long`), I believe the current name is correct?
> >
> >
> > The like the idea of using an enum for send-leave-group request flag.
> >
> >
> > For `Optional`, if we remove the public getters, the question goes away.
> >
> >
> >
> > About merging both `CloseOptions`: fine with me to keep them separated.
> > Was just an idea :)
> >
> >
> >
> > -Matthias
> >
> >
> > On 9/30/24 3:14 PM, Sophie Blee-Goldman wrote:
> >> +1 to using the fluent API and including "with" in the setter names.
> >>
> >> I also think Matthias raised a good point, that maybe it would be a good
> >> time to "fix" this issue in the CloseOptions for the KafkaStreams#close
> >> method, to conform to this API format. As for his other "random idea"
> >> about
> >> combining the two: Personally I would prefer to keep a separate
> >> CloseOptions class for the Consumer vs for Kafka Streams, since they
> will
> >> not necessarily always have the same exact semantics and parameters.
> Even
> >> if the API ends up looking more or less the same right now -- although
> >> consider that the "default" behavior is different for the consumer vs
> >> Kafka
> >> Streams, if leaveGroup is not specified in the CloseOptions then the
> >> consumer will opt to leave the group in some cases whereas Streams never
> >> will. So already there is some difference between the Consumer and
> >> Streams
> >> CloseOptions, so I'd rather not combine them
> >>
> >> On Mon, Sep 30, 2024 at 11:16 AM Chia-Ping Tsai 
> >> wrote:
> >>
>  Overall, the "static method builder" pattern seems better to me, and
> >>> thus I would prefer to make it the "gold standard" and we can see
> >>> what  we
> >>> can do for `Admin API` mid/long term?
> >>>
> >>> Since we want to avoid complicated compatibility issues, adding a
> static
> >>> method builder to Admin options seems more acceptable.
> >>>
> >>> However, the naming convention for the "setter" method might be a
> >>> concern.
> >>> Kafka Streams (KS) uses "withXXX," while Admin options do not include
> >>> the
> >>> "with" prefix. Additionally, the RPC requests follow the setXXX naming
> >>> convention.
> >>>
> >>> I’m unsure if this is the right time to align the fluent pattern naming
> >>> across the entire Kafka project, but it would be great if we could
> >>> reach a
> >>> consensus on "gold standard".
> >>>
> >>> I'm +1 to using the static method builder and `with` prefix due to
> >>> following advantages.
> >>> -
> >>>
> >>> 1. A static method builder allows making the constructor private,
> >>> which can
> >>> prevent unintended inheritance
> >>> -
> >>>
> >>> 2. The "with" prefix makes it easier to search for setters within the
> >>> option class
> >>>
> >>> Any feedback?
> >>>
>  Btw: I was also wondering, if we should r