[jira] [Resolved] (KAFKA-14749) Re-enable 'spotlessScalaCheck' task (in Jenkinsfile)

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-14749.
-
  Assignee: Ismael Juma
Resolution: Fixed

We found a way to re-enable this when running with Java 11 or newer, see 
https://github.com/apache/kafka/pull/13311.

> Re-enable 'spotlessScalaCheck' task (in Jenkinsfile)
> 
>
> Key: KAFKA-14749
> URL: https://issues.apache.org/jira/browse/KAFKA-14749
> Project: Kafka
>  Issue Type: Task
>  Components: build
>Reporter: Dejan Stojadinović
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 3.5.0
>
>
> {*}Description{*}:
> We were forced to remove 'spotlessScalaCheck' (see KAFKA-14728) but we should 
> bring it back when circumstances change (i.e. when Apache Kafka 4.0 drops 
> support for Java 8).
> Related github issue comment: 
> [https://github.com/apache/kafka/pull/13263#issuecomment-1441825913]
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14680) Gradle version upgrade 7 -->> 8

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-14680.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Gradle version upgrade 7 -->> 8
> ---
>
> Key: KAFKA-14680
> URL: https://issues.apache.org/jira/browse/KAFKA-14680
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Reporter: Dejan Stojadinović
>Assignee: Dejan Stojadinović
>Priority: Major
> Fix For: 3.5.0
>
>
> +*Gradle 8 release notes:*+
>  * *8.0*
>  ** [https://github.com/gradle/gradle/releases/tag/v8.0.0]
>  ** 
> [https://docs.gradle.org/8.0/release-notes.html|https://docs.gradle.org/8.0-rc-3/release-notes.html]
>  *  *8.0.1:*
>  ** [https://github.com/gradle/gradle/releases/tag/v8.0.1]
>  ** [https://docs.gradle.org/8.0.1/release-notes.html]
>  ** [https://github.com/gradle/gradle/milestone/229?closed=1]
> *Upgrade notes:* 
> [https://docs.gradle.org/8.0/userguide/upgrading_version_7.html#changes_8.0|https://docs.gradle.org/8.0/userguide/upgrading_version_7.html#changes_8.0]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14206) Upgrade zookeeper to 3.7.1 to address security vulnerabilities

[Valeriy Kassenbayev (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valeriy Kassenbayev reopened KAFKA-14206:
-

Still have the same CVEs reported:
{code:java}
  ✗ Denial of Service (DoS) [High 
Severity][https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584063] in 
io.netty:netty-codec@4.1.63.Final
    introduced by org.apache.kafka:kafka_2.13@3.4.0 > 
org.apache.zookeeper:zookeeper@3.6.3 > io.netty:netty-handler@4.1.63.Final > 
io.netty:netty-codec@4.1.63.Final
  This issue was fixed in versions: 4.1.68.Final
  ✗ Denial of Service (DoS) [High 
Severity][https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584064] in 
io.netty:netty-codec@4.1.63.Final
    introduced by org.apache.kafka:kafka_2.13@3.4.0 > 
org.apache.zookeeper:zookeeper@3.6.3 > io.netty:netty-handler@4.1.63.Final > 
io.netty:netty-codec@4.1.63.Final
  This issue was fixed in versions: 4.1.68.Final {code}
ZooKeeper does not seem to have been upgraded:
{code:java}
[mac /tmp]# tar tzf kafka_2.13-3.4.0.tgz | grep -i libs/zookeeper
kafka_2.13-3.4.0/libs/zookeeper-3.6.3.jar
kafka_2.13-3.4.0/libs/zookeeper-jute-3.6.3.jar
[mac /tmp]# {code}

> Upgrade zookeeper to 3.7.1 to address security vulnerabilities
> --
>
> Key: KAFKA-14206
> URL: https://issues.apache.org/jira/browse/KAFKA-14206
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 3.2.1
>Reporter: Valeriy Kassenbayev
>Assignee: Luke Chen
>Priority: Blocker
> Fix For: 3.4.0
>
>
> Kafka 3.2.1 is using ZooKeeper, which is affected by 
> [CVE-2021-37136|https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584064] and 
> [CVE-2021-37137:|https://www.cve.org/CVERecord?id=CVE-2021-37137]
> {code:java}
>   ✗ Denial of Service (DoS) [High 
> Severity][https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584063] in 
> io.netty:netty-codec@4.1.63.Final
>     introduced by org.apache.kafka:kafka_2.13@3.2.1 > 
> org.apache.zookeeper:zookeeper@3.6.3 > io.netty:netty-handler@4.1.63.Final > 
> io.netty:netty-codec@4.1.63.Final
>   This issue was fixed in versions: 4.1.68.Final
>   ✗ Denial of Service (DoS) [High 
> Severity][https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584064] in 
> io.netty:netty-codec@4.1.63.Final
>     introduced by org.apache.kafka:kafka_2.13@3.2.1 > 
> org.apache.zookeeper:zookeeper@3.6.3 > io.netty:netty-handler@4.1.63.Final > 
> io.netty:netty-codec@4.1.63.Final
>   This issue was fixed in versions: 4.1.68.Final {code}
> The issues were fixed in the next versions of ZooKeeper (starting from 
> 3.6.4). ZooKeeper 3.7.1 is the next stable 
> [release|https://zookeeper.apache.org/releases.html] at the moment.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14763) Add integration test for DelegationTokenCommand tool

[Gantigmaa Selenge (Jira)]

Gantigmaa Selenge created KAFKA-14763:
-

 Summary: Add integration test for DelegationTokenCommand tool
 Key: KAFKA-14763
 URL: https://issues.apache.org/jira/browse/KAFKA-14763
 Project: Kafka
  Issue Type: Task
Reporter: Gantigmaa Selenge


When moving DelegationTokenCommand from core to tools module in 
[https://github.com/apache/kafka/pull/13172], the existing integration test 
could not be migrated because there is no {{BaseRequestTest}} or {{SaslSetup}} 
to help setup integration tests in the tools module. We will need to create 
similar setup in the tools module and create an integration test for the 
command tool. 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Gantigmaa Selenge]

The KIP is accepted with 3 binding votes (Chris, Luke, Mickael) and 1
non-binding vote (Federico).

Thank you all.
Regards,
Tina

Re: [VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors

[Mickael Maison]

Thanks for the KIP

+1 (binding)

On Thu, Jan 26, 2023 at 4:36 PM Jorge Esteban Quilcate Otoya
 wrote:
>
> Hi all,
>
> I'd like to call for a vote on KIP-864, which proposes to add metrics to
> measure end-to-end latency in source and sink connectors.
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-864%3A+Add+End-To-End+Latency+Metrics+to+Connectors
>
> Discussion thread:
> https://lists.apache.org/thread/k6rh2mr7pg94935fgpqw8b5fj308f2n7
>
> Many thanks,
> Jorge.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1622

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 541816 lines...]
[2023-02-27T12:00:39.044Z] 
[2023-02-27T12:00:39.044Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange() STARTED
[2023-02-27T12:00:49.085Z] 
[2023-02-27T12:00:49.085Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange() PASSED
[2023-02-27T12:00:49.085Z] 
[2023-02-27T12:00:49.085Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards() STARTED
[2023-02-27T12:02:02.482Z] 
[2023-02-27T12:02:02.482Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards() PASSED
[2023-02-27T12:02:02.482Z] 
[2023-02-27T12:02:02.482Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow() STARTED
[2023-02-27T12:02:02.482Z] 
[2023-02-27T12:02:02.482Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow() PASSED
[2023-02-27T12:02:02.482Z] 
[2023-02-27T12:02:02.482Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldNotAllowDivergentLogs() STARTED
[2023-02-27T12:02:02.482Z] 
[2023-02-27T12:02:02.482Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldNotAllowDivergentLogs() PASSED
[2023-02-27T12:02:02.482Z] 
[2023-02-27T12:02:02.482Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
logsShouldNotDivergeOnUncleanLeaderElections() STARTED
[2023-02-27T12:02:10.202Z] 
[2023-02-27T12:02:10.202Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > EpochDrivenReplicationProtocolAcceptanceTest > 
logsShouldNotDivergeOnUncleanLeaderElections() PASSED
[2023-02-27T12:02:10.202Z] 
[2023-02-27T12:02:10.202Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts() STARTED
[2023-02-27T12:02:11.310Z] 
[2023-02-27T12:02:11.310Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts() PASSED
[2023-02-27T12:02:11.310Z] 
[2023-02-27T12:02:11.310Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader() STARTED
[2023-02-27T12:02:15.446Z] 
[2023-02-27T12:02:15.446Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader() PASSED
[2023-02-27T12:02:15.446Z] 
[2023-02-27T12:02:15.446Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse() STARTED
[2023-02-27T12:02:16.477Z] 
[2023-02-27T12:02:16.477Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse() PASSED
[2023-02-27T12:02:16.477Z] 
[2023-02-27T12:02:16.477Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ReplicationUtilsTest > testUpdateLeaderAndIsr() STARTED
[2023-02-27T12:02:16.477Z] 
[2023-02-27T12:02:16.477Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ReplicationUtilsTest > testUpdateLeaderAndIsr() PASSED
[2023-02-27T12:02:16.477Z] 
[2023-02-27T12:02:16.477Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
STARTED
[2023-02-27T12:02:17.507Z] 
[2023-02-27T12:02:17.507Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
PASSED
[2023-02-27T12:02:17.507Z] 
[2023-02-27T12:02:17.507Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() STARTED
[2023-02-27T12:02:17.507Z] 
[2023-02-27T12:02:17.507Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() PASSED
[2023-02-27T12:02:17.507Z] 
[2023-02-27T12:02:17.507Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
STARTED
[2023-02-27T12:02:17.507Z] 
[2023-02-27T12:02:17.507Z] Gradle Test Run :core:integrationTest 

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[David Jacot]

Thanks for the update.

- The new interface looks good to me. Note that the javadoc does not
reflect the new interface though.
- Could we precise how errors will be handled? For instance, say that the
iterator could translate the input stream to a record. Would calling the
next method on the iterator throw an exception?

Thanks,
David

On Sat, Feb 25, 2023 at 10:43 PM Chia-Ping Tsai  wrote:

>
>
> On 2023/02/25 08:26:28 David Jacot wrote:
> > Hi Chia-Ping,
> >
> > Thanks for the KIP.
> >
> > I find the configure method in the proposed interface a bit weird for a
> few
> > reasons. First, it has a default implementation which suggests that it is
> > optional but it is not because the InputStream is required. Second, it
>
> oh, my bad. I forgot to remove the default impl after adding the input
> stream to config method.
>
>
> >
> > Did we consider using two methods instead of one? We could have configure
> > coming from Configurable et setInputStream to set the InputStream.
> Another
> > option would be to have a method which takes the input stream and returns
> > an iterator to consume the records.
>
> I prefer to set input stream only once. Also, if Configurable interface is
> required for all plugins in kafka code base, the option.2 is suitable - we
> can change the returned type of `readRecords(InputStream)` from single
> record to an iterator of records. Thus, the new interface not only extends
> Configurable but also take input stream only once.
>
>
> >
> > Cheers,
> > David
> >
> > Le mer. 22 févr. 2023 à 11:53, Chia-Ping Tsai  a
> > écrit :
> >
> > >
> > >
> > > On 2023/02/22 10:01:29 Alexandre Dupriez wrote:
> > > > Hi Chia-Ping,
> > > >
> > > > Thanks for your answer. Apologies I should have been clearer in the
> > > > previous message. What I meant is, is there a plan to use the SPI
> more
> > > > broadly inside the Kafka codebase?
> > >
> > > no, there is no plan to reuse the SPI.
> > >
> > > > The question arises because the interface exposes a close() method
> > > > which is never invoked by the ConsoleProducer. Hence, although we
> need
> > > > to keep this method to maintain compatibility of the SPI with its
> > > > current implementations
> > >
> > > yep, you are right. the close() method is never executed by the
> > > ConsoleProducer. The ConsolerConsumer has similar issue (
> > > https://github.com/apache/kafka/pull/8978). I will fix it.
> > >
> > > > we should perhaps clarify that this method is
> > > > not used/deprecated, unless it is intended to be used in the future.
> > >
> > >  I prefer to keep the close() since the new interface is similar to
> > > Deserializer. The close() method can be used to notify/release
> something
> > > when the console is going to be down.
> > >
> > >
> >
>

[jira] [Resolved] (KAFKA-14060) Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14060.
---
Resolution: Done

> Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest
> ---
>
> Key: KAFKA-14060
> URL: https://issues.apache.org/jira/browse/KAFKA-14060
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Hector Geraldino
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1623

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 449191 lines...]
[2023-02-27T14:35:09.782Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldGetToRunningWithOnlyGlobalTopology() STARTED
[2023-02-27T14:35:10.724Z] 
[2023-02-27T14:35:10.724Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldGetToRunningWithOnlyGlobalTopology() PASSED
[2023-02-27T14:35:10.724Z] 
[2023-02-27T14:35:10.724Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin() STARTED
[2023-02-27T14:35:13.369Z] 
[2023-02-27T14:35:13.369Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin() PASSED
[2023-02-27T14:35:13.369Z] 
[2023-02-27T14:35:13.369Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart() STARTED
[2023-02-27T14:35:16.012Z] 
[2023-02-27T14:35:16.012Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart() PASSED
[2023-02-27T14:35:16.012Z] 
[2023-02-27T14:35:16.012Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin() STARTED
[2023-02-27T14:35:18.823Z] 
[2023-02-27T14:35:18.824Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin() PASSED
[2023-02-27T14:35:19.765Z] 
[2023-02-27T14:35:19.765Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() STARTED
[2023-02-27T14:35:25.536Z] 
[2023-02-27T14:35:25.536Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() PASSED
[2023-02-27T14:35:26.491Z] 
[2023-02-27T14:35:26.491Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFailStopped() STARTED
[2023-02-27T14:35:26.491Z] 
[2023-02-27T14:35:26.491Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFailStopped() PASSED
[2023-02-27T14:35:26.491Z] 
[2023-02-27T14:35:26.491Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > 
shouldNotRequireQueryHandler(TestInfo) STARTED
[2023-02-27T14:35:28.252Z] 
[2023-02-27T14:35:28.252Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > 
shouldNotRequireQueryHandler(TestInfo) PASSED
[2023-02-27T14:35:28.252Z] 
[2023-02-27T14:35:28.252Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFailNotStarted() STARTED
[2023-02-27T14:35:28.252Z] 
[2023-02-27T14:35:28.252Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFailNotStarted() PASSED
[2023-02-27T14:35:28.252Z] 
[2023-02-27T14:35:28.252Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFetchFromPartition() STARTED
[2023-02-27T14:35:30.895Z] 
[2023-02-27T14:35:30.895Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFetchFromPartition() PASSED
[2023-02-27T14:35:30.895Z] 
[2023-02-27T14:35:30.895Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > 
shouldFetchExplicitlyFromAllPartitions() STARTED
[2023-02-27T14:35:32.660Z] 
[2023-02-27T14:35:32.660Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > 
shouldFetchExplicitlyFromAllPartitions() PASSED
[2023-02-27T14:35:32.660Z] 
[2023-02-27T14:35:32.660Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFailUnknownStore() STARTED
[2023-02-27T14:35:32.660Z] 
[2023-02-27T14:35:32.660Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldFailUnknownStore() PASSED
[2023-02-27T14:35:32.660Z] 
[2023-02-27T14:35:32.660Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldRejectNonRunningActive() STARTED
[2023-02-27T14:35:34.424Z] 
[2023-02-27T14:35:34.424Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > IQv2IntegrationTest > shouldRejectNonRunningActive() PASSED
[2023-02-27T14:35:36.344Z] 
[2023-02-27T14:35:36.344Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 182 > InternalTopicIntegrationTest > 
shouldCompactTopicsForKeyValueStoreChangelogs() STA

[jira] [Resolved] (KAFKA-14738) Topic disappears from kafka_topic.sh --list after modifying it with kafka_acl.sh

[Andras Katona (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Katona resolved KAFKA-14738.
---
Resolution: Not A Bug

> Topic disappears from kafka_topic.sh --list after modifying it with 
> kafka_acl.sh
> 
>
> Key: KAFKA-14738
> URL: https://issues.apache.org/jira/browse/KAFKA-14738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 3.2.3
>Reporter: Gabriel Lukacs
>Priority: Major
>
> Topic is not listed via kafka-topics.sh --list after modifying it with 
> kafka-acls.sh (-add --allow-principal User:CN=test --operation Read):
> $ /opt/kafka/bin/kafka-topics.sh --create --bootstrap-server kafka:9092 
> --topic test2 --replication-factor 1 --partitions 50
> Created topic test2.
> $ /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka:9092 --topic 
> test2
> test2
> $ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka:9092 --topic test2 
> --add --allow-principal User:CN=test --operation Read
> Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=test2, 
> patternType=LITERAL)`:
>         (principal=User:CN=test, host=*, operation=READ, permissionType=ALLOW)
> Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=test2, 
> patternType=LITERAL)`:
>         (principal=User:CN=test, host=*, operation=READ, permissionType=ALLOW)
> $ /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka:9092 --topic 
> test2                                   
> $ /opt/kafka/bin/kafka-topics.sh --create --bootstrap-server kafka:9092 
> --topic test2
> Error while executing topic command : Topic 'test2' already exists.
> [2023-02-21 16:37:39,185] ERROR 
> org.apache.kafka.common.errors.TopicExistsException: Topic 'test2' already 
> exists.
>  (kafka.admin.TopicCommand$)
> $ /opt/kafka/bin/kafka-topics.sh --delete --bootstrap-server kafka:9092 
> --topic test2
> Error while executing topic command : Topic 'test2' does not exist as expected
> [2023-02-21 16:37:49,485] ERROR java.lang.IllegalArgumentException: Topic 
> 'test2' does not exist as expected
>         at 
> kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:401)
>         at 
> kafka.admin.TopicCommand$TopicService.deleteTopic(TopicCommand.scala:361)
>         at kafka.admin.TopicCommand$.main(TopicCommand.scala:63)
>         at kafka.admin.TopicCommand.main(TopicCommand.scala)
>  (kafka.admin.TopicCommand$)
> $ /opt/kafka/bin/kafka-topics.sh --version
> 3.2.3 (Commit:50029d3ed8ba576f)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14387) kafka.common.KafkaException | kafka_2.12-3.3.1.jar

[Andras Katona (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Katona resolved KAFKA-14387.
---
Resolution: Information Provided

> kafka.common.KafkaException  | kafka_2.12-3.3.1.jar
> ---
>
> Key: KAFKA-14387
> URL: https://issues.apache.org/jira/browse/KAFKA-14387
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 3.3.1
>Reporter: masood
>Priority: Major
>
> It appears, Kafka.common.KafkaException is deprecated in 
> kafka_2.12-3.3.1.jar. 
> Please let me know which exception should be used.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14764) Metadata API ignores topic names if at least one topic is provided with an id

[David Jacot (Jira)]

David Jacot created KAFKA-14764:
---

 Summary: Metadata API ignores topic names if at least one topic is 
provided with an id
 Key: KAFKA-14764
 URL: https://issues.apache.org/jira/browse/KAFKA-14764
 Project: Kafka
  Issue Type: Bug
Reporter: David Jacot


The Metadata API accepts both topic names and topic ids in the request. This 
suggests that a single request could mix them in. At least, we have no logic on 
the server side to prevent this. The issue is that the server just ignores any 
topic specified with a name if there is at least one topic specified with an id 
in the request. In other words, if a request contains topic-id-1, topic-id-2, 
topic-name-1 and topic-name-2, the response will only have metadata for 
topic-id-1 and topic-id-2.

This does not hurt us today because the clients does not use topic ids in the 
request at all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14765) Support SCRAM for brokers at bootstrap

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-14765:
-

 Summary: Support SCRAM for brokers at bootstrap
 Key: KAFKA-14765
 URL: https://issues.apache.org/jira/browse/KAFKA-14765
 Project: Kafka
  Issue Type: Improvement
  Components: kraft
Reporter: Proven Provenzano


We want to add SCRAM support for brokers at bootstrap.

We will support bootstrap as described in 
[KIP-900|https://cwiki.apache.org/confluence/display/KAFKA/KIP-900%3A+KRaft+kafka-storage.sh+API+additions+to+support+SCRAM+for+Kafka+Brokers]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chia-Ping Tsai]

Hi David

> 
> - The new interface looks good to me. Note that the javadoc does not
> reflect the new interface though.
> - Could we precise how errors will be handled? For instance, say that the
> iterator could translate the input stream to a record. Would calling the
> next method on the iterator throw an exception?

Thanks for kind reminder. I have updated the KIP. Please take a look. Thanks



> David Jacot  於 2023年2月27日 下午8:26 寫道：
> 
> Thanks for the update.
> 
> - The new interface looks good to me. Note that the javadoc does not
> reflect the new interface though.
> - Could we precise how errors will be handled? For instance, say that the
> iterator could translate the input stream to a record. Would calling the
> next method on the iterator throw an exception?
> 
> Thanks,
> David
> 
> On Sat, Feb 25, 2023 at 10:43 PM Chia-Ping Tsai  wrote:
> 
>> 
>> 
>> On 2023/02/25 08:26:28 David Jacot wrote:
>>> Hi Chia-Ping,
>>> 
>>> Thanks for the KIP.
>>> 
>>> I find the configure method in the proposed interface a bit weird for a
>> few
>>> reasons. First, it has a default implementation which suggests that it is
>>> optional but it is not because the InputStream is required. Second, it
>> 
>> oh, my bad. I forgot to remove the default impl after adding the input
>> stream to config method.
>> 
>> 
>>> 
>>> Did we consider using two methods instead of one? We could have configure
>>> coming from Configurable et setInputStream to set the InputStream.
>> Another
>>> option would be to have a method which takes the input stream and returns
>>> an iterator to consume the records.
>> 
>> I prefer to set input stream only once. Also, if Configurable interface is
>> required for all plugins in kafka code base, the option.2 is suitable - we
>> can change the returned type of `readRecords(InputStream)` from single
>> record to an iterator of records. Thus, the new interface not only extends
>> Configurable but also take input stream only once.
>> 
>> 
>>> 
>>> Cheers,
>>> David
>>> 
>>> Le mer. 22 févr. 2023 à 11:53, Chia-Ping Tsai  a
>>> écrit :
>>> 
 
 
 On 2023/02/22 10:01:29 Alexandre Dupriez wrote:
> Hi Chia-Ping,
> 
> Thanks for your answer. Apologies I should have been clearer in the
> previous message. What I meant is, is there a plan to use the SPI
>> more
> broadly inside the Kafka codebase?
 
 no, there is no plan to reuse the SPI.
 
> The question arises because the interface exposes a close() method
> which is never invoked by the ConsoleProducer. Hence, although we
>> need
> to keep this method to maintain compatibility of the SPI with its
> current implementations
 
 yep, you are right. the close() method is never executed by the
 ConsoleProducer. The ConsolerConsumer has similar issue (
 https://github.com/apache/kafka/pull/8978). I will fix it.
 
> we should perhaps clarify that this method is
> not used/deprecated, unless it is intended to be used in the future.
 
 I prefer to keep the close() since the new interface is similar to
 Deserializer. The close() method can be used to notify/release
>> something
 when the console is going to be down.
 
 
>>> 
>> 

Re: [VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors

[Chris Egerton]

Hi all,

I could have sworn I +1'd this but I can't seem to find a record of that.

In the hopes that this action is idempotent, +1 (binding). Thanks for the
KIP!

Cheers,

Chris

On Mon, Feb 27, 2023 at 6:28 AM Mickael Maison 
wrote:

> Thanks for the KIP
>
> +1 (binding)
>
> On Thu, Jan 26, 2023 at 4:36 PM Jorge Esteban Quilcate Otoya
>  wrote:
> >
> > Hi all,
> >
> > I'd like to call for a vote on KIP-864, which proposes to add metrics to
> > measure end-to-end latency in source and sink connectors.
> >
> > KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-864%3A+Add+End-To-End+Latency+Metrics+to+Connectors
> >
> > Discussion thread:
> > https://lists.apache.org/thread/k6rh2mr7pg94935fgpqw8b5fj308f2n7
> >
> > Many thanks,
> > Jorge.
>

Re: [VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors

[Knowles Atchison Jr]

+1 (non binding)

On Mon, Feb 27, 2023 at 11:21 AM Chris Egerton 
wrote:

> Hi all,
>
> I could have sworn I +1'd this but I can't seem to find a record of that.
>
> In the hopes that this action is idempotent, +1 (binding). Thanks for the
> KIP!
>
> Cheers,
>
> Chris
>
> On Mon, Feb 27, 2023 at 6:28 AM Mickael Maison 
> wrote:
>
> > Thanks for the KIP
> >
> > +1 (binding)
> >
> > On Thu, Jan 26, 2023 at 4:36 PM Jorge Esteban Quilcate Otoya
> >  wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-864, which proposes to add metrics
> to
> > > measure end-to-end latency in source and sink connectors.
> > >
> > > KIP:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-864%3A+Add+End-To-End+Latency+Metrics+to+Connectors
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/k6rh2mr7pg94935fgpqw8b5fj308f2n7
> > >
> > > Many thanks,
> > > Jorge.
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1624

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 535704 lines...]
[2023-02-27T17:11:10.578Z] 
[2023-02-27T17:11:10.578Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testGetAclExistingZNode() PASSED
[2023-02-27T17:11:10.578Z] 
[2023-02-27T17:11:10.578Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testSessionExpiryDuringClose() STARTED
[2023-02-27T17:11:10.578Z] 
[2023-02-27T17:11:10.578Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testSessionExpiryDuringClose() PASSED
[2023-02-27T17:11:10.578Z] 
[2023-02-27T17:11:10.578Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testReinitializeAfterAuthFailure() STARTED
[2023-02-27T17:11:13.624Z] 
[2023-02-27T17:11:13.624Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testReinitializeAfterAuthFailure() PASSED
[2023-02-27T17:11:13.624Z] 
[2023-02-27T17:11:13.624Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testSetAclNonExistentZNode() STARTED
[2023-02-27T17:11:13.624Z] 
[2023-02-27T17:11:13.624Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testSetAclNonExistentZNode() PASSED
[2023-02-27T17:11:13.624Z] 
[2023-02-27T17:11:13.624Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testConnectionLossRequestTermination() 
STARTED
[2023-02-27T17:11:23.568Z] 
[2023-02-27T17:11:23.568Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testConnectionLossRequestTermination() 
PASSED
[2023-02-27T17:11:23.568Z] 
[2023-02-27T17:11:23.568Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testExistsNonExistentZNode() STARTED
[2023-02-27T17:11:23.568Z] 
[2023-02-27T17:11:23.568Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testExistsNonExistentZNode() PASSED
[2023-02-27T17:11:23.568Z] 
[2023-02-27T17:11:23.568Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testGetDataNonExistentZNode() STARTED
[2023-02-27T17:11:24.572Z] 
[2023-02-27T17:11:24.572Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testGetDataNonExistentZNode() PASSED
[2023-02-27T17:11:24.572Z] 
[2023-02-27T17:11:24.572Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testConnectionTimeout() STARTED
[2023-02-27T17:11:26.991Z] 
[2023-02-27T17:11:26.991Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testConnectionTimeout() PASSED
[2023-02-27T17:11:26.991Z] 
[2023-02-27T17:11:26.991Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler() STARTED
[2023-02-27T17:11:27.911Z] 
[2023-02-27T17:11:27.911Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler() PASSED
[2023-02-27T17:11:27.911Z] 
[2023-02-27T17:11:27.911Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testUnresolvableConnectString() STARTED
[2023-02-27T17:11:27.911Z] 
[2023-02-27T17:11:27.911Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testUnresolvableConnectString() PASSED
[2023-02-27T17:11:27.911Z] 
[2023-02-27T17:11:27.911Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testGetChildrenNonExistentZNode() STARTED
[2023-02-27T17:11:28.917Z] 
[2023-02-27T17:11:28.917Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testGetChildrenNonExistentZNode() PASSED
[2023-02-27T17:11:28.917Z] 
[2023-02-27T17:11:28.917Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testPipelinedGetData() STARTED
[2023-02-27T17:11:28.917Z] 
[2023-02-27T17:11:28.917Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testPipelinedGetData() PASSED
[2023-02-27T17:11:28.917Z] 
[2023-02-27T17:11:28.917Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > 
testZNodeChildChangeHandlerForChildChange() STARTED
[2023-02-27T17:11:28.917Z] 
[2023-02-27T17:11:28.917Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > 
testZNodeChildChangeHandlerForChildChange() PASSED
[2023-02-27T17:11:28.917Z] 
[2023-02-27T17:11:28.917Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > ZooKeeperClientTest > testGetChildrenExistingZNodeWi

[jira] [Created] (KAFKA-14766) Improve performance of VarInt encoding/decoding

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-14766:


 Summary: Improve performance of VarInt encoding/decoding
 Key: KAFKA-14766
 URL: https://issues.apache.org/jira/browse/KAFKA-14766
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Divij Vaidya
Assignee: Divij Vaidya
 Fix For: 3.5.0


Our current implementation in ByteUtils could be improved via loop unrolling 
and short circuiting scenarios such as 0. 

With the changes in attached PR, we see upto 83% improvement in throughput for  
writingVarInt.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Chris Egerton]

Hi Philip,

Yeah,  "DNS resolution should occur..." seems like a better fit. 👍

One other question I have is whether we should expose some kind of public
API for performing preflight validation of the bootstrap URLs. If we change
the behavior of a client configured with a silly typo (e.g.,
"loclahost instead of localhost") from failing in the constructor to
failing with a retriable exception, this might lead some client
applications to handle that failure by, well, retrying. For reference, this
is exactly what we do in Kafka Connect right now; see [1] and [2]. IMO it'd
be nice to be able to opt into keeping the current behavior so that
projects like Connect could still do preflight checks of the
bootstrap.servers property for connectors before starting them, and report
any issues by failing fast instead of continuously writing warning/error
messages to their logs.

I'm not sure about where this new API could go, but a few options might be:

- Expose a public variant of the existing ClientUtils class
- Add static methods to the ConsumerConfig, ProducerConfig, and
AdminClientConfig classes
- Add those same static methods to the KafkaConsumer, KafkaProducer, and
KafkaAdminClient classes

If this seems reasonable, we should probably also specify in the KIP that
Kafka Connect will leverage this preflight validation logic before
instantiating any Kafka clients for use by connectors or tasks, and
continue to fail fast if there are typos in the bootstrap.servers property,
or if temporary DNS resolution issues come up.

[1] -
https://github.com/apache/kafka/blob/5f9d01668cae64b2cacd7872d82964fa78862aaf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L606
[2] -
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L439

Cheers,

Chris

On Fri, Feb 24, 2023 at 4:59 PM Philip Nee  wrote:

> Hey Chris,
>
> Thanks for the quick response, and I apologize for the unclear wording
> there, I guess "DNS lookup" would be a more appropriate wording here. So
> what I meant there was, to delegate the DNS lookup in the constructor to
> the network client poll, and it will happen on the very first poll.  I
> guess the logic could look like this:
>
> - if the client has been bootstrapped, do nothing.
> - Otherwise, perform DNS lookup, and acquire the bootstrap server address.
>
> Thanks for the comment there, I'll change up the wording.  Maybe revise it
> as "DNS resolution should occur in the poll" ?
>
> P
>
> On Fri, Feb 24, 2023 at 1:47 PM Chris Egerton 
> wrote:
>
> > Hi Philip,
> >
> > Thanks for the KIP!
> >
> > QQ: In the "Proposed Changes" section, the KIP states that "Bootstrapping
> > should now occur in the poll method before attempting to update the
> > metadata. This includes resolving the addresses and bootstrapping the
> > metadata.". By "bootstrapping the metadata" do we mean actually
> contacting
> > the bootstrap servers, or just setting some internal state related to the
> > current set of servers that can be contacted for metadata? I ask because
> it
> > seems like the language here implies the former, but if that's the case,
> > this is already happening in poll (or at least, the first invocation of
> > it), and if it's the latter, it's probably not necessary to mention in
> the
> > KIP since it doesn't really impact user-facing behavior. It also seems
> like
> > that detail might impact how intertwined this and KIP-899 are, though the
> > similarity could still be superficial either way.
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Feb 23, 2023 at 9:21 PM Philip Nee  wrote:
> >
> > > Hey Ismael,
> > >
> > > Thanks for the feedback! The proposal is not to retry automatically but
> > > relies on the user polling the NetworkClient (basically, consumer.poll)
> > to
> > > reattempt the bootstrap. If bootstrapping fails, a NetworkException
> > > (retriable) will be thrown.
> > >
> > > Thanks!
> > > P
> > >
> > >
> > >
> > > On Thu, Feb 23, 2023 at 1:34 PM Ismael Juma  wrote:
> > >
> > > > Thanks for the KIP. Not sure if I missed it, but how long will we
> retry
> > > for
> > > > and when do we give up and propagate the failure to the user?
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Feb 23, 2023 at 9:30 AM Philip Nee 
> > wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > I want to start a discussion thread about how we can handle client
> > > > > bootstrap failure due DNS lookup.  This requires a bit of
> behavioral
> > > > > change, so a KIP is proposed and attached to this email. Let me
> know
> > > what
> > > > > you think!
> > > > >
> > > > >
> > > > > *A small remark here*: *As the title of this KIP might sound
> > > > > familiar/similar to KIP-899, it is not the same.*
> > > > >
> > > > > *In Summary:* I want to propose a KIP to change the existing
> > bootstrap
> > > > > (upon instantiation) strategy because it is reasonable to allow
> > clients
> > > > to
> > > > > retry
> > > > >
>

Re: [VOTE] KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

[Calvin Liu]

Hi Jason,
Updated the fields accordingly. Also, rename the BrokerState to
ReplicaState.
Thanks.

On Wed, Feb 22, 2023 at 4:38 PM Jason Gustafson 
wrote:

> Hi Calvin,
>
> The `BrokerState` struct I suggested would replace the `BrokerId` field in
> older versions.
>
> { "name": "ReplicaId", "type": "int32", "versions": "0-13",
> "entityType": "brokerId",
>   "about": "The broker ID of the follower, of -1 if this request is
> from a consumer." },
> { "name": "BrokerState", "type": "BrokerState", "taggedVersions":
> "14+", "tag": 1, "fields": [
>   { "name": "BrokerId", "type": "int32", "versions": "14+",
> "entityType": "brokerId",
> "about": "The broker ID of the follower, of -1 if this request is
> from a consumer." },
>   { "name": "BrokerEpoch", "type": "int64", "versions": "14+", "about":
> "The epoch of this follower." }
> ]},
>
> Note that the version range of `ReplicaId` is set to 0-13. Version 14
> onward would not include it.
>
> -Jason
>
> On Wed, Feb 22, 2023 at 12:07 PM Calvin Liu 
> wrote:
>
> > To Jose:
> > 1. Actually I have a second thoughts about the naming ReplicaEpoch. The
> > BrokerEpoch only applies to the replication protocol between the brokers.
> > For others like the KRaft controller, this field can be ignored. So can
> we
> > change the name to ReplicaEpoch when we really use it in other scenarios?
> >
> > On Wed, Feb 22, 2023 at 11:08 AM Calvin Liu  wrote:
> >
> > > To Jason:
> > > 1. Related to the Fetch Request fields change, previously you suggested
> > > deprecating the ReplicaId and moving it into a BrokerState field. How
> > about
> > > we just make the BrokerEpoch a tag field?
> > > - The ReplicaId is currently in use and is filled every time. So that
> we
> > > can keep the way simple.
> > > - We can still make the optional BrokerEpoch out of the request when it
> > is
> > > not needed.
> > >
> > > On Tue, Feb 21, 2023 at 10:39 PM Calvin Liu 
> wrote:
> > >
> > >> To Jason:
> > >> 1. We can make the BrokerEpoch a tagged field. But I am not sure about
> > >> your proposed metadata structure. In the BrokerState, do we need to
> > store
> > >> the BrokerId again? It would duplicate with ReplicaId.
> > >> 2. Considering that the broker reboot data loss case is rare and Kraft
> > is
> > >> coming soon. Plus we need extra effort to
> > >> - Simply asking the controller to compare the epoch with its best
> > >> knowledge is not enough, because the ZK controller may not know the
> > latest
> > >> broker epoch,
> > >> - The current design only helps with the delayed AlterPartition issue
> > >> when the broker reboots. In ZK mode, we also need to cover the broker
> > >> reboot + controller reboot scenario. If the reboot broker is in ISR
> > >> already, the controller also crashes during the broker reboot, the new
> > >> controller can be completely unaware of the bounced broker and select
> > this
> > >> broker as the leader.
> > >> - Create a test framework to simulate the event sequence of broker
> > reboot
> > >> and registration, delayed AlterPartition request.
> > >>
> > >> To Jose:
> > >> 1. Thanks for the renaming advice. I will update the KIP later.
> > >> 2. Ack, will update.
> > >>
> > >>
> > >> On Tue, Feb 21, 2023 at 2:49 PM José Armando García Sancio
> > >>  wrote:
> > >>
> > >>> Hi Calvin,
> > >>>
> > >>> Thanks for the improvement.
> > >>>
> > >>> 1. In the KIP, you suggest changing the Fetch request to "Rename the
> > >>> ReplicaId to BrokerId" and "Add a new Field BrokerEpoch". The Fetch
> > >>> RPC is used by replicas that are not brokers, for example controllers
> > >>> in KRaft.
> > >>> Can we keep the name "ReplicaId" and use "ReplicaEpoch". Both KRaft
> > >>> and ISR partitions have the concept of replica id and replica epoch
> > >>> but not necessarily the concept of a broker.
> > >>>
> > >>> 2. Since the new field "BrokerEpoch '' is ignorable, should it also
> > >>> have a default value? How about -1 since that is what you use in
> > >>> AlterPartittion RPC.
> > >>>
> > >>> --
> > >>> -José
> > >>>
> > >>
> >
>

[VOTE] KIP-898: Modernize Connect plugin discovery

[Greg Harris]

Hi,

I'd like to call a vote for KIP-898 which aims to improve the performance
of Connect startup by allowing discovery of plugins via the ServiceLoader.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-898
%3A+Modernize+Connect+plugin+discovery

Discussion thread:
https://lists.apache.org/thread/wxh0r343w86s91py0876njzbyn5qxd8s

Thanks!

Re: [VOTE] KIP-898: Modernize Connect plugin discovery

[Chris Egerton]

+1 (binding). Thanks for the KIP!

On Mon, Feb 27, 2023 at 12:51 PM Greg Harris 
wrote:

> Hi,
>
> I'd like to call a vote for KIP-898 which aims to improve the performance
> of Connect startup by allowing discovery of plugins via the ServiceLoader.
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-898
> %3A+Modernize+Connect+plugin+discovery
>
> Discussion thread:
> https://lists.apache.org/thread/wxh0r343w86s91py0876njzbyn5qxd8s
>
> Thanks!
>

Re: [VOTE] KIP-904: Kafka Streams - Guarantee subtractor is called before adder if key has not changed

[Guozhang Wang]

+1.

On Sun, Feb 26, 2023 at 4:27 PM Fq Public  wrote:
>
> Hi everyone,
>
> I'd like to start the vote on KIP-904: Kafka Streams - Guarantee subtractor
> is called before adder if key has not changed.
> The KIP is available here: https://cwiki.apache.org/confluence/x/P5VbDg
> The easiest way to view the entire discussion thread is via this search
> link: https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:KIP-904
> Please take a look and vote.
>
> Thank you,
> Farooq

[jira] [Resolved] (KAFKA-14264) Refactor coordinator code

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-14264.
---
Resolution: Fixed

[~pnee] can you fill in the details on the fixed versions? Thanks!

> Refactor coordinator code
> -
>
> Key: KAFKA-14264
> URL: https://issues.apache.org/jira/browse/KAFKA-14264
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Major
>
> To refactor the consumer, we changed how the coordinator is called.  However, 
> there will be a time period where the old and new implementation need to 
> coexist, so we will need to override some of the methods and create a new 
> implementation of the coordinator.  In particular:
>  # ensureCoordinatorReady needs to be non-blocking or we could just use the 
> sendFindCoordinatorRequest.
>  # joinGroupIfNeeded needs to be broken up into more find grain stages for 
> the new implementation to work.
> We also need to create the coordinator state machine.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14468) Refactor Commit Logic

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-14468.
---
Resolution: Fixed

[~pnee] can you fill in the details on the fixed versions? Thanks!

> Refactor Commit Logic
> -
>
> Key: KAFKA-14468
> URL: https://issues.apache.org/jira/browse/KAFKA-14468
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Major
>
> Refactor commit logic using the new multi-threaded coordinator construct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-907: Add Boolean Serde to public interface

[Guozhang Wang]

+1, thanks!

On Fri, Feb 24, 2023 at 5:43 AM SpacRocket 
wrote:

> Hi Everyone,
>
> I'd like to call for a vote on KIP-907, which proposes new public classes
> to the package org.apache.kafka.common.serialization:
> - BooleanSerde
> - BooleanSerializer
> - BooleanDeserializer
>
> KIP:
> lists.apache.org
> 
> 
> 
> 
> 
>
> - Jakub
>

RE: [VOTE] KIP-907: Add Boolean Serde to public interface

[Chia-Ping Tsai]

+1 (binding)

[jira] [Created] (KAFKA-14767) Gradle build fails with missing commitId after git gc

[Greg Harris (Jira)]

Greg Harris created KAFKA-14767:
---

 Summary: Gradle build fails with missing commitId after git gc
 Key: KAFKA-14767
 URL: https://issues.apache.org/jira/browse/KAFKA-14767
 Project: Kafka
  Issue Type: Bug
  Components: build
Reporter: Greg Harris


Reproduction steps:
1. `git gc`
2. `./gradlew jar`

Expected behavior: build completes successfully (or shows other build errors)
Actual behavior:
{noformat}
Task failed with an exception.
---
* What went wrong:
A problem was found with the configuration of task ':storage:createVersionFile' 
(type 'DefaultTask').
  - Property 'commitId' doesn't have a configured value.
    
    Reason: This property isn't marked as optional and no value has been 
configured.
    
    Possible solutions:
      1. Assign a value to 'commitId'.
      2. Mark property 'commitId' as optional.
    
    Please refer to 
https://docs.gradle.org/7.6/userguide/validation_problems.html#value_not_set 
for more details about this problem.{noformat}
This appears to be due to the fact that the build.gradle determineCommitId() 
function is unable to read the git commit hash for the current HEAD. This 
appears to happen after a `git gc` takes place, which causes the 
`.git/refs/heads/*` files to be moved to `.git/packed-refs`.

The determineCommitId() should be patched to also try reading from the 
packed-refs to determine the commit hash.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1625

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Philip Nee]

Hey Chris,

Thanks again for the feedback!


For the preflight DNS check (are we basically trying to resolve the DNS
there?): Maybe it makes more sense to add it to the Config modules? I would
like to hear what the community says as I'm not familiar with the Connect
use case.

A "slower failing" alternative - I wonder if it makes sense for us to
extend the NetworkException so that clients can be smarter at handling
these exceptions. Of course, it is still retriable and requires polling the
consumer, but then we can distinguish the DNS resolution error from other
network errors.

Thanks!
P





On Mon, Feb 27, 2023 at 9:36 AM Chris Egerton 
wrote:

> Hi Philip,
>
> Yeah,  "DNS resolution should occur..." seems like a better fit. 👍
>
> One other question I have is whether we should expose some kind of public
> API for performing preflight validation of the bootstrap URLs. If we change
> the behavior of a client configured with a silly typo (e.g.,
> "loclahost instead of localhost") from failing in the constructor to
> failing with a retriable exception, this might lead some client
> applications to handle that failure by, well, retrying. For reference, this
> is exactly what we do in Kafka Connect right now; see [1] and [2]. IMO it'd
> be nice to be able to opt into keeping the current behavior so that
> projects like Connect could still do preflight checks of the
> bootstrap.servers property for connectors before starting them, and report
> any issues by failing fast instead of continuously writing warning/error
> messages to their logs.
>
> I'm not sure about where this new API could go, but a few options might be:
>
> - Expose a public variant of the existing ClientUtils class
> - Add static methods to the ConsumerConfig, ProducerConfig, and
> AdminClientConfig classes
> - Add those same static methods to the KafkaConsumer, KafkaProducer, and
> KafkaAdminClient classes
>
> If this seems reasonable, we should probably also specify in the KIP that
> Kafka Connect will leverage this preflight validation logic before
> instantiating any Kafka clients for use by connectors or tasks, and
> continue to fail fast if there are typos in the bootstrap.servers property,
> or if temporary DNS resolution issues come up.
>
> [1] -
>
> https://github.com/apache/kafka/blob/5f9d01668cae64b2cacd7872d82964fa78862aaf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L606
> [2] -
>
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L439
>
> Cheers,
>
> Chris
>
> On Fri, Feb 24, 2023 at 4:59 PM Philip Nee  wrote:
>
> > Hey Chris,
> >
> > Thanks for the quick response, and I apologize for the unclear wording
> > there, I guess "DNS lookup" would be a more appropriate wording here. So
> > what I meant there was, to delegate the DNS lookup in the constructor to
> > the network client poll, and it will happen on the very first poll.  I
> > guess the logic could look like this:
> >
> > - if the client has been bootstrapped, do nothing.
> > - Otherwise, perform DNS lookup, and acquire the bootstrap server
> address.
> >
> > Thanks for the comment there, I'll change up the wording.  Maybe revise
> it
> > as "DNS resolution should occur in the poll" ?
> >
> > P
> >
> > On Fri, Feb 24, 2023 at 1:47 PM Chris Egerton 
> > wrote:
> >
> > > Hi Philip,
> > >
> > > Thanks for the KIP!
> > >
> > > QQ: In the "Proposed Changes" section, the KIP states that
> "Bootstrapping
> > > should now occur in the poll method before attempting to update the
> > > metadata. This includes resolving the addresses and bootstrapping the
> > > metadata.". By "bootstrapping the metadata" do we mean actually
> > contacting
> > > the bootstrap servers, or just setting some internal state related to
> the
> > > current set of servers that can be contacted for metadata? I ask
> because
> > it
> > > seems like the language here implies the former, but if that's the
> case,
> > > this is already happening in poll (or at least, the first invocation of
> > > it), and if it's the latter, it's probably not necessary to mention in
> > the
> > > KIP since it doesn't really impact user-facing behavior. It also seems
> > like
> > > that detail might impact how intertwined this and KIP-899 are, though
> the
> > > similarity could still be superficial either way.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Feb 23, 2023 at 9:21 PM Philip Nee 
> wrote:
> > >
> > > > Hey Ismael,
> > > >
> > > > Thanks for the feedback! The proposal is not to retry automatically
> but
> > > > relies on the user polling the NetworkClient (basically,
> consumer.poll)
> > > to
> > > > reattempt the bootstrap. If bootstrapping fails, a NetworkException
> > > > (retriable) will be thrown.
> > > >
> > > > Thanks!
> > > > P
> > > >
> > > >
> > > >
> > > > On Thu, Feb 23, 2023 at 1:34 PM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks for the KIP. Not sure if I missed it, 

Kafka back-pressure | Proposal for dynamic config

[Vipul Goyal]

Hi Team,

I am looking for an effective back pressure solution in Kafka Please find
below my use case and detail.

Use case:
I need to run “some execution”  when receiving a Kafka record.  Some
execution could be an external API call, And it is possible that sometimes
this API may not perform well. Considering we already have API timeout
configured.
Now, let’s say there is some degradation in external API, and batch
processing will start taking much time and overall it may breach
max.poll.internal.ms and hence rebalancing. This would have a ripple effect
on other consumers in the same group.

As-is:
Currently, Kafka consumer provide the capability to pause/resume, but it
may not be very effective to back-pressure the flow.
1. Ideally We may need to just slow down, not exactly pause the consumption
itself.
2. If we pause the consumer, then we would have to remember to resume it.


*Proposal?*
To be able to auto recover from this, we can have some control algorithm
(ex PID) in place, which will adjust *max.poll.records* dynamically and we
can avoid rebalancing. And I see that PID (proportional integral
derivative) is being referenced in many other places as well like:
PIDRateEstimator

,
tuning-spark-back-pressure-by-simulation


kafka-spark-consumer .

Currently, the Kafka consumer doesn't allow dynamic change of the max poll
records setting, But the proposal is to allow passing in this config
dynamically.

I am not an expert and may be thinking in an orthogonal direction. Please
advice.

Regards,
Vipul

[jira] [Created] (KAFKA-14768) proposal to reduce the first message's send time cost and max block time for safety

[fujian (Jira)]

fujian created KAFKA-14768:
--

 Summary: proposal to reduce the first message's send time cost and 
max block time for safety 
 Key: KAFKA-14768
 URL: https://issues.apache.org/jira/browse/KAFKA-14768
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.3.1
Reporter: fujian
Assignee: hzh0425


{*}Background{*}:
In [KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software 
Foundation|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
  kafka introduced the feature of hierarchical storage.
Also, [KAFKA-9555] Topic-based implementation for the RemoteLogMetadataManager 
- ASF JIRA (apache.org) implements the default RLMM - 'TopicBased-RLMM'.

{*}Problem{*}:
TopicBased-RLMM will only subscribe to the Partitions where the current Broker 
is Leader or Follower. If the current Broker is not the Leader or Follower, 
then RLMM will directly skip the metadata records related to these Partitions.

When reassign user-partitions occurs, rlmm will subscribe to new 
user-partitions, assuming that the metadata-partition to which the new 
user-partition belongs is 'metadata-partition0', and RLMM has consumed 
'metadata-partition0' *to offset = 100* before the reassign partition occurs, 
then {*}after reassign{*}, RMLM will *not* consume 'metadata-partition0' 
\{*}from the beginning{*}, and finally cause the metadata records related to 
the new user-partition to *be lost with offset < 100.*

*Solution*

Let RLMM subscribe to all user-patitions, instead of only subscribing to 
partitions where the current broker is leader or follower.
In this way, when reassign partition occurs, RLMM will have new partition's 
metadata records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)