Re: [VOTE] 3.5.2 RC1

[Jakub Scholz]

+1 non-binding. I used the staged Scala 2.13 binaries and the staged Maven
repo to run my tests and all seems to work fine.

Thanks & Regards
Jakub

On Tue, Nov 21, 2023 at 11:09 AM Luke Chen  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.5.2.
>
> This is a bugfix release with several fixes since the release of 3.5.1,
> including dependency version bumps for CVEs.
>
> Release notes for the 3.5.2 release:
> https://home.apache.org/~showuon/kafka-3.5.2-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Nov. 28.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~showuon/kafka-3.5.2-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~showuon/kafka-3.5.2-rc1/javadoc/
>
> * Tag to be voted upon (off 3.5 branch) is the 3.5.2 tag:
> https://github.com/apache/kafka/releases/tag/3.5.2-rc1
>
> * Documentation:
> https://kafka.apache.org/35/documentation.html
>
> * Protocol:
> https://kafka.apache.org/35/protocol.html
>
> * Successful Jenkins builds for the 3.5 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/98/
> There are some falky tests, including the testSingleIP test failure. It
> failed because of some infra change and we fixed it
>  recently.
>
> System tests: running, will update the results later.
>
>
>
> Thank you.
> Luke
>

Re: [DISCUSS] Road to Kafka 4.0

[Anton Agestam]

Hi again Colin,

I don't know at what point the discussion skewed from talking about
migrations paths to talking about "production readiness". The concern I
raised originally was about the former rather than the latter.

In your last message you wrote:

> But, on the KRaft side, I still maintain that nothing is missing except
JBOD, which we already have a plan for.

But earlier in this thread you mentioned an issue with "torn writes",
possibly missing tests, as well as the fact that the recommended method of
replacing controller nodes is undocumented. Would you mind clarifying what
your stance is on these three issues? Do you think that they are important
enablers of upgrade paths or not?

---

> Well, the line was drawn in KIP-833. If we redraw it, what is to stop us
from redrawing it again and again?

I'm fairly new to the Kafka community so please forgive me if I'm missing
things that have been said in earlier discussions, but reading up on that
KIP I see it has language like "Note: this timeline is very rough and
subject to change." in the section of versions, but it also says "As
outlined above, we expect to close these gaps soon" with relation to the
outstanding features. From my perspective this doesn't really look like an
agreement that dynamic quorum membership changes shall not be a blocker for
4.0.

To answer the specific question you pose here, "what is to stop us from
redrawing it again and again?", wouldn't the suggestion of parallel work
lanes brought up by Josep address this concern?

BR,
Anton

Den tors 23 nov. 2023 kl 05:48 skrev Colin McCabe :

> On Tue, Nov 21, 2023, at 19:30, Luke Chen wrote:
> > Yes, KIP-853 and disk failure support are both very important missing
> > features. For the disk failure support, I don't think this is a
> > "good-to-have-feature", it should be a "must-have" IMO. We can't announce
> > the 4.0 release without a good solution for disk failure in KRaft.
>
> Hi Luke,
>
> Thanks for the reply.
>
> Controller disk failure support is not missing from KRaft. I described how
> to handle controller disk failures earlier in this thread.
>
> I should note here that the broker in ZooKeeper mode also requires manual
> handling of disk failures. Restarting a broker with the same ID, but an
> empty disk, breaks the invariants of replication when in ZK mode. Consider:
>
> 1. Broker 1 goes down. A ZK state change notification for /brokers fires
> and goes on the controller queue.
>
> 2. Broker 1 comes back up with an empty disk.
>
> 3. The controller processes the zk state change notification for /brokers.
> Since broker 1 is up no action is taken.
>
> 4. Now broker 1 is in the ISR for any partitions it was previously, but
> has no data. If it is or becomes leader for any partitions, irreversable
> data loss will occur.
>
> This problem is more than theoretical. We at Confluent have observed it in
> production and put in place special workarounds for the ZK clusters we
> still have.
>
> KRaft has never had this problem because brokers are removed from ISRs
> when a new incarnation of the broker registers.
>
> So perhaps ZK mode is not ready for production for Aiven? Since disk
> failures do in fact require special handling there. (And/or bringing up new
> nodes with empty disks, which seems to be their main concern.)
>
> >
> > It’s also worth thinking about how Apache Kafka users who depend on JBOD
> > might look at the risks of not having a 3.8 release. JBOD support on
> KRaft
> > is planned to be added in 3.7, and is still in progress so far. So it’s
> > hard to say it’s a blocker or not. But in practice, even if the feature
> is
> > made into 3.7 in time, a lot of new code for this feature is unlikely to
> be
> > entirely bug free. We need to maintain the confidence of those users, and
> > forcing them to migrate through 3.7 where this new code is hardly
> > battle-tested doesn’t appear to do that.
> >
>
> As Ismael said, if there are JBOD bugs in 3.7, we will do follow-on point
> releases to address them.
>
> > Our goal for 4.0 should be that all the “main” features in KRaft are in
> > production ready state. To reach the goal, I think having one more
> release
> > makes sense. We can have different opinions about what the “main
> features”
> > in KRaft are, but we should all agree, JBOD is one of them.
>
> The current plan is for JBOD to be production-ready in the 3.7 branch.
>
> The other features of KRaft have been in production-ready state since the
> 3.3 release. (Well, except for delegation tokens and SCRAM, which were
> implemented in 3.5 and 3.6)
>
> > I totally agree with you. We can keep delaying the 4.0 release forever.
> I'd
> > also like to draw a line to it. So, in my opinion, the 3.8 release is the
> > line. No 3.9, 3.10 releases after that. If this is the decision, will
> your
> > concern about this infinite loop disappear?
>
> Well, the line was drawn in KIP-833. If we redraw it, what is to stop us
> from redrawing it again and again?
>
> >
> > Final note

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2413

[Apache Jenkins Server]

See 

RE: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0

[Anton Agestam]

Hi Ismael,

This looks like a healthy KIP for Kafka 👍

As the implementer of a Kafka Protocol library for Python, Aiven-Open/kio
[1], I'm curious how this change will affect the library.

We generate entities for the full protocol by introspecting the JSON schema
definitions under _. How will the KIP change those definitions? Will the
dropped versions be reflected as bumps of the lower limit
in "validVersions"?

Thanks and BR,
Anton

[1]: https://github.com/Aiven-Open/kio

On 2023/01/03 16:17:24 Ismael Juma wrote:
> Hi all,
>
> I would like to start a discussion regarding the removal of very old
client
> protocol API versions in Apache Kafka 4.0 to improve maintainability &
> supportability of Kafka. Please take a look at the proposal:
>
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
>
> Ismael
>

[PR] MINOR: Update docs for 3.6.1 [kafka-site]

[via GitHub]

mimaison opened a new pull request, #568:
URL: https://github.com/apache/kafka-site/pull/568

   I generated the javadoc with `OpenJDK Runtime Environment Temurin-17.0.2+8 
(build 17.0.2+8)`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[VOTE] 3.6.1 RC0

[Mickael Maison]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.6.1.

This is a bugfix release with several fixes, including dependency
version bumps for CVEs.

Release notes for the 3.6.1 release:
https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Friday, December 1

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~mimaison/kafka-3.6.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
https://github.com/apache/kafka/releases/tag/3.6.1-rc0

PR for updating docs:
https://github.com/apache/kafka-site/pull/568

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
Unit/integration tests: We still have a lot of flaky tests in the 3.6
branch. Looking at the last few 3.6 builds in
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ it seems all
tests passed at least once apart from
ClusterConnectionStatesTest.testSingleIP(). There's
https://issues.apache.org/jira/browse/KAFKA-15762 to fix that test.
System tests: Still running I'll post an update once they complete.

Thanks,
Mickael

Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0

[Ismael Juma]

Hi Anton,

The json schema definition files haven't been deemed public api yet
although they perhaps should (KIPs welcome).

Your suggestion to update `validVersions` sounds good to me. Longer-term,
it would also make sense to consider a `deprecatedVersions` field (this is
especially useful if we deprecate at the start of a new major versions
cycle, but this requires a new KIP).

Ismael

On Fri, Nov 24, 2023 at 5:31 AM Anton Agestam
 wrote:

> Hi Ismael,
>
> This looks like a healthy KIP for Kafka 👍
>
> As the implementer of a Kafka Protocol library for Python, Aiven-Open/kio
> [1], I'm curious how this change will affect the library.
>
> We generate entities for the full protocol by introspecting the JSON schema
> definitions under _. How will the KIP change those definitions? Will the
> dropped versions be reflected as bumps of the lower limit
> in "validVersions"?
>
> Thanks and BR,
> Anton
>
> [1]: https://github.com/Aiven-Open/kio
>
> On 2023/01/03 16:17:24 Ismael Juma wrote:
> > Hi all,
> >
> > I would like to start a discussion regarding the removal of very old
> client
> > protocol API versions in Apache Kafka 4.0 to improve maintainability &
> > supportability of Kafka. Please take a look at the proposal:
> >
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> >
> > Ismael
> >
>

Re: Apache Kafka 3.7.0 Release

[Stanislav Kozlovski]

Hey all,

The KIP Freeze has passed. I count 31 KIPs that will be going into the 3.7
Release. Thank you all for your hard work!

They are the following (some of these were accepted in previous releases
and have minor parts going out, some targeting a Preview release and the
rest being fully released as regular.):
 - KIP-1000: List Client Metrics Configuration Resources
 - KIP-1001: Add CurrentControllerId Metric
 - KIP-405: Kafka Tiered Storage
 - KIP-580: Exponential Backoff for Kafka Clients
 - KIP-714: Client metrics and observability
 - KIP-770: Replace "buffered.records.per.partition" &
"cache.max.bytes.buffering" with
"{statestore.cache}/{input.buffer}.max.bytes"
 - KIP-848: The Next Generation of the Consumer Rebalance Protocol
 - KIP-858: Handle JBOD broker disk failure in KRaft
 - KIP-890: Transactions Server-Side Defense
 - KIP-892: Transactional StateStores
 - KIP-896: Remove old client protocol API versions in Kafka 4.0 -
metrics/request log changes to identify deprecated apis
 - KIP-925: Rack aware task assignment in Kafka Streams
 - KIP-938: Add more metrics for measuring KRaft performance
 - KIP-951 - Leader discovery optimizations for the client
 - KIP-954: expand default DSL store configuration to custom types
 - KIP-959: Add BooleanConverter to Kafka Connect
 - KIP-960: Single-key single-timestamp IQv2 for state stores
 - KIP-963: Additional metrics in Tiered Storage
 - KIP-968: Support single-key_multi-timestamp Interactive Queries (IQv2)
for Versioned State Stores
 - KIP-970: Deprecate and remove Connect's redundant task configurations
endpoint
 - KIP-975: Docker Image for Apache Kafka
 - KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect
 - KIP-978: Allow dynamic reloading of certificates with different DN / SANs
 - KIP-979: Allow independently stop KRaft processes
 - KIP-980: Allow creating connectors in a stopped state
 - KIP-985: Add reverseRange and reverseAll query over kv-store in IQv2
 - KIP-988: Streams Standby Update Listener
 - KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and
TimestampedRangeQuery
 - KIP-998: Give ProducerConfig(props, doLog) constructor protected access

Notable KIPs that didn't make the Freeze were KIP-977 - it only got 2/3
votes.

For the full list and latest source of truth, refer to the Release Plan
3.7.0 Document
.

Thanks for your contributions once again!
Best,
Stan


On Thu, Nov 23, 2023 at 2:27 PM Nick Telford  wrote:

> Hi Stan,
>
> I'd like to propose including KIP-892 in the 3.7 release. The KIP has been
> accepted and I'm just working on rebasing the implementation against trunk
> before I open a PR.
>
> Regards,
> Nick
>
> On Tue, 21 Nov 2023 at 11:27, Mayank Shekhar Narula <
> mayanks.nar...@gmail.com> wrote:
>
> > Hi Stan
> >
> > Can you include KIP-951 to the 3.7 release plan? All PRs are merged in
> the
> > trunk.
> >
> > On Wed, Nov 15, 2023 at 4:05 PM Stanislav Kozlovski
> >  wrote:
> >
> > > Friendly reminder to everybody that the KIP Freeze is *exactly 7 days
> > away*
> > > - November 22.
> > >
> > > A KIP must be accepted by this date in order to be considered for this
> > > release. Note, any KIP that may not be implemented in time, or
> otherwise
> > > risks heavily destabilizing the release, should be deferred.
> > >
> > > Best,
> > > Stan
> > >
> > > On Fri, Nov 3, 2023 at 6:03 AM Sophie Blee-Goldman <
> > sop...@responsive.dev>
> > > wrote:
> > >
> > > > Looks great, thank you! +1
> > > >
> > > > On Thu, Nov 2, 2023 at 10:21 AM David Jacot
> >  > > >
> > > > wrote:
> > > >
> > > > > +1 from me as well. Thanks, Stan!
> > > > >
> > > > > David
> > > > >
> > > > > On Thu, Nov 2, 2023 at 6:04 PM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Thanks Stanislav, +1
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski
> > > > > >  wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Given the discussion here and the lack of any pushback, I have
> > > > changed
> > > > > > the
> > > > > > > dates of the release:
> > > > > > > - KIP Freeze - *November 22 *(moved 4 days later)
> > > > > > > - Feature Freeze - *December 6 *(moved 2 days earlier)
> > > > > > > - Code Freeze - *December 20*
> > > > > > >
> > > > > > > If anyone has any thoughts against this proposal - please let
> me
> > > > know!
> > > > > It
> > > > > > > would be good to settle on this early. These will be the dates
> > > we're
> > > > > > going
> > > > > > > with
> > > > > > >
> > > > > > > Best,
> > > > > > > Stanislav
> > > > > > >
> > > > > > > On Thu, Oct 26, 2023 at 12:15 AM Sophie Blee-Goldman <
> > > > > > > sop...@responsive.dev>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the response and explanations -- I think the main
> > > > question
> > > > > > for
> > > > > > > > me
> > > > > > > > was whether we intended to permanently increase the KF -- FF
> > gap
> > > > from
> > > >

[jira] [Created] (KAFKA-15891) Flaky test: testResetSinkConnectorOffsetsOverriddenConsumerGroupId – org.apache.kafka.connect.integration.OffsetsApiIntegrationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15891:
-

 Summary: Flaky test: 
testResetSinkConnectorOffsetsOverriddenConsumerGroupId – 
org.apache.kafka.connect.integration.OffsetsApiIntegrationTest
 Key: KAFKA-15891
 URL: https://issues.apache.org/jira/browse/KAFKA-15891
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


h4. Error
org.opentest4j.AssertionFailedError: Condition not met within timeout 3. 
Sink connector consumer group offsets should catch up to the topic end offsets 
==> expected:  but was: 
h4. Stacktrace
org.opentest4j.AssertionFailedError: Condition not met within timeout 3. 
Sink connector consumer group offsets should catch up to the topic end offsets 
==> expected:  but was: 
 at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
 at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
 at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
 at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
 at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210)
 at 
app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
 at 
app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328)
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312)
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302)
 at 
app//org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.verifyExpectedSinkConnectorOffsets(OffsetsApiIntegrationTest.java:917)
 at 
app//org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.resetAndVerifySinkConnectorOffsets(OffsetsApiIntegrationTest.java:725)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15892) Flaky test: testAlterSinkConnectorOffsets – org.apache.kafka.connect.integration.OffsetsApiIntegrationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15892:
-

 Summary: Flaky test: testAlterSinkConnectorOffsets – 
org.apache.kafka.connect.integration.OffsetsApiIntegrationTest
 Key: KAFKA-15892
 URL: https://issues.apache.org/jira/browse/KAFKA-15892
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


h4. Error
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Could not 
alter connector offsets. Error response: \{"error_code":500,"message":"Failed 
to alter consumer group offsets for connector test-connector either because its 
tasks haven't stopped completely yet or the connector was resumed before the 
request to alter its offsets could be successfully completed. If the connector 
is in a stopped state, this operation can be safely retried. If it doesn't 
eventually succeed, the Connect cluster may need to be restarted to get rid of 
the zombie sink tasks."}
h4. Stacktrace
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Could not 
alter connector offsets. Error response: \{"error_code":500,"message":"Failed 
to alter consumer group offsets for connector test-connector either because its 
tasks haven't stopped completely yet or the connector was resumed before the 
request to alter its offsets could be successfully completed. If the connector 
is in a stopped state, this operation can be safely retried. If it doesn't 
eventually succeed, the Connect cluster may need to be restarted to get rid of 
the zombie sink tasks."}
 at 
app//org.apache.kafka.connect.util.clusters.EmbeddedConnect.alterConnectorOffsets(EmbeddedConnect.java:614)
 at 
app//org.apache.kafka.connect.util.clusters.EmbeddedConnectCluster.alterConnectorOffsets(EmbeddedConnectCluster.java:48)
 at 
app//org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.alterAndVerifySinkConnectorOffsets(OffsetsApiIntegrationTest.java:363)
 at 
app//org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testAlterSinkConnectorOffsets(OffsetsApiIntegrationTest.java:287)
 at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method)
 at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
 at 
app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
 at 
app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
 at 
app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
 at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
 at 
app//org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
 at app//org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
 at 
app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
 at 
app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
 at app//org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
 at app//org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
 at app//org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
 at app//org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
 at app//org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
 at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
 at app//org.junit.runners.ParentRunner.run(ParentRunner.java:413)
 at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15464) Allow dynamic reloading of certificates with different DN / SANs

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15464.

Fix Version/s: 3.7.0
   Resolution: Fixed

> Allow dynamic reloading of certificates with different DN / SANs
> 
>
> Key: KAFKA-15464
> URL: https://issues.apache.org/jira/browse/KAFKA-15464
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jakub Scholz
>Assignee: Jakub Scholz
>Priority: Major
> Fix For: 3.7.0
>
>
> Kafka currently doesn't allow dynamic reloading of keystores when the new key 
> has a different DN or removes some of the SANs. While it might help to 
> prevent users from breaking their cluster, in some cases it would be great to 
> be able to bypass this validation when desired.
> More details are in the [KIP-978: Allow dynamic reloading of certificates 
> with different DN / 
> SANs|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429128]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

[Ismael Juma]

Hi all,

I also vote +1.

The vote passes with 4 binding +1s:

1. Colin McCabe
2. Jun Rao
3. Jose Sancio
4. Ismael Juma

Thanks,
Ismael

On Tue, Nov 21, 2023 at 12:06 PM Ismael Juma  wrote:

> Hi all,
>
> I would like to start a vote on KIP-896. Please take a look and let us
> know what you think.
>
> Even though most of the changes in this KIP will be done for Apache Kafka
> 4.0, I would like to introduce a new metric and new request log attribute
> in Apache 3.7 to help users identify usage of deprecated protocol api
> versions.
>
> Link:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
>
> Thanks,
> Ismael
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2414

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 429084 lines...]
Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldCancelBeforeGetIfMayCancelWhileRunning STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldCancelBeforeGetIfMayCancelWhileRunning PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldRecordOnlyFirstErrorBeforeGetOnFailedCompletion STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldRecordOnlyFirstErrorBeforeGetOnFailedCompletion PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldBlockUntilFailedCompletion STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldBlockUntilFailedCompletion PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateConnectorLoggingContext STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateConnectorLoggingContext PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateTaskLoggingContext STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateTaskLoggingContext PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldNotAllowNullConnectorNameForConnectorContext STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldNotAllowNullConnectorNameForConnectorContext PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldAllowNestedLoggingContexts STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldAllowNestedLoggingContexts PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldNotAllowNullTaskIdForTaskContext STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldNotAllowNullTaskIdForTaskContext PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateAndCloseLoggingContextEvenWithNullContextMap STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateAndCloseLoggingContextEvenWithNullContextMap PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldNotAllowNullTaskIdForOffsetContext STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldNotAllowNullTaskIdForOffsetContext PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateOffsetsLoggingContext STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.LoggingContextTest > 
shouldCreateOffsetsLoggingContext PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.SharedTopicAdminTest > 
shouldCloseWithDurationAfterTopicAdminUsed STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.SharedTopicAdminTest > 
shouldCloseWithDurationAfterTopicAdminUsed PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.SharedTopicAdminTest > 
shouldCloseWithoutBeingUsed STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.SharedTopicAdminTest > 
shouldCloseWithoutBeingUsed PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.SharedTopicAdminTest > 
shouldCloseAfterTopicAdminUsed STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.apache.kafka.connect.util.SharedTopicAdminTest > 
shouldCloseAfterTopicAdminUsed PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 45 > 
org.

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2415

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

[Chris Egerton]

Hi Yash,

Thanks for taking a look! Yeah, it looks like we'll be forced to hold onto
the property until 5.0 if we can't introduce it at least one minor release
before 4.0. I don't think this is the worst thing. Although it'd be nice to
have the property completely removed when designing features like KIP-987,
if necessary, we can also declare any new features incompatible with
connectors that have opted out of enforcement of the tasks.max property
(and likely enforce this restraint programmatically via preflight
validation, failing connectors/tasks, log messages, etc.).

Cheers,

Chris

On Wed, Nov 22, 2023 at 10:04 PM Yash Mayya  wrote:

> Hi Chris,
>
> Thanks for the well written and comprehensive KIP! Given that we're already
> past the KIP freeze deadline for 3.7.0 (
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) and
> there may not be a 3.8.0 release before the 4.0.0 release, would we then be
> forced to punt the removal of "tasks.max.enforce" to a future 5.0.0
> release? I don't have any other comments, and the proposed changes make
> sense to me.
>
> Thanks,
> Yash
>
> On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton 
> wrote:
>
> > Hi Hector,
> >
> > Thanks for taking a look! I think the key difference between the proposed
> > behavior and the rejected alternative is that the set of tasks that will
> be
> > running with the former is still a complete set of tasks, whereas the set
> > of tasks in the latter is a subset of tasks. Also noteworthy but slightly
> > less important: the problem will be more visible to users with the former
> > (the connector will still be marked FAILED) than with the latter.
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > hgerald...@bloomberg.net> wrote:
> >
> > > Thanks for the KIP Chris, adding this check makes total sense.
> > >
> > > I do have one question. The second paragraph in the Public Interfaces
> > > section states:
> > >
> > > "If the connector generated excessive tasks after being reconfigured,
> > then
> > > any existing tasks for the connector will be allowed to continue
> running,
> > > unless that existing set of tasks also exceeds the tasks.max property."
> > >
> > > Would not failing the connector land us in the second scenario of
> > > 'Rejected Alternatives'?
> > >
> > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To:
> > > dev@kafka.apache.org
> > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka
> Connect
> > >
> > > Hi all,
> > >
> > > I'd like to open up KIP-1004 for discussion:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+
> > > property+in+Kafka+Connect
> > >
> > > As a brief summary: this KIP proposes that the Kafka Connect runtime
> > start
> > > failing connectors that generate a greater number of tasks than the
> > > tasks.max property, with an optional emergency override that can be
> used
> > to
> > > continue running these (probably-buggy) connectors if absolutely
> > necessary.
> > >
> > > I'll be taking time off most of the next three weeks, so response
> latency
> > > may be a bit higher than usual, but I wanted to kick off the discussion
> > in
> > > case we can land this in time for the upcoming 3.7.0 release.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Should we continue to merge without a green build? No!

[Chris Egerton]

Hi all,

There's a lot to catch up on here but I wanted to clarify something.
Regarding this comment from Sophie:


> Yet multiple people in this thread so
far have voiced support for "gating merges on the successful completion of
all parts of the build except tests". Just to be totally clear, I really
don't think that was ever in question -- though it certainly doesn't hurt
to remind everyone.

> So, this thread is not about whether or not to merge with failing
*builds, *but it's
whether it should be acceptable to merge with failing *tests.*


I think there's a misunderstanding here. I was suggesting
programmatic gating, not manual. If we could disable these types of changes
from being merged, instead of relying on committers to check and interpret
Jenkins results, that'd be a quick win IMO. And, because of the
already-discussed issues with flaky tests, it seemed difficult to disable
PRs from being merged with failing tests--just for other parts of the build.

However, I think the retry logic brought up by David could be sufficient to
skip that kind of intermediate step and allow us to just start
programmatically disabling PR merges if the build (including) tests fails.
But if anyone's interested, we can still prevent failing tests from failing
the build with the ignoreFailures property [1].

[1] -
https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures

Cheers,

Chris

On Wed, Nov 22, 2023 at 3:00 AM Ismael Juma  wrote:

> I think it breaks the Jenkins output otherwise. Feel free to test it via a
> PR.
>
> Ismael
>
> On Wed, Nov 22, 2023, 12:42 AM David Jacot 
> wrote:
>
> > Hi Ismael,
> >
> > No, I was not aware of KAFKA-12216. My understanding is that we could
> still
> > do it without the JUnitFlakyTestDataPublisher plugin and we could use
> > gradle enterprise for this. Or do you think that reporting the flaky
> tests
> > in the build results is required?
> >
> > David
> >
> > On Wed, Nov 22, 2023 at 9:35 AM Ismael Juma  wrote:
> >
> > > Hi David,
> > >
> > > Did you take a look at
> https://issues.apache.org/jira/browse/KAFKA-12216
> > ?
> > > I
> > > looked into this option already (yes, there isn't much that we haven't
> > > considered in this space).
> > >
> > > Ismael
> > >
> > > On Wed, Nov 22, 2023 at 12:24 AM David Jacot
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks for the good discussion and all the comments. Overall, it
> seems
> > > that
> > > > we all agree on the bad state of our CI. That's a good first step!
> > > >
> > > > I have talked to a few folks this week about it and it seems that
> many
> > > > folks (including me) are not comfortable with merging PRs at the
> moment
> > > > because the results of our builds are so bad. I had 40+ failed tests
> in
> > > one
> > > > of my PRs, all unrelated to my changes. It is really hard to be
> > > productive
> > > > with this.
> > > >
> > > > Personally, I really want to move towards requiring a green build to
> > > merge
> > > > to trunk because this is a clear and binary signal. I agree that we
> > need
> > > to
> > > > stabilize the builds before we could even require this so here is my
> > > > proposal.
> > > >
> > > > 1) We could leverage the `reports.junitXml.mergeReruns` option in
> > gradle.
> > > > From the doc [1]:
> > > >
> > > > > When mergeReruns is enabled, if a test fails but is then retried
> and
> > > > succeeds, its failures will be recorded as  instead of
> > > > , within one . This is effectively the reporting
> > > > produced by the surefire plugin of Apache Maven™ when enabling
> reruns.
> > If
> > > > your CI server understands this format, it will indicate that the
> test
> > > was
> > > > flaky. If it > does not, it will indicate that the test succeeded as
> it
> > > > will ignore the  information. If the test does not
> > succeed
> > > > (i.e. it fails for every retry), it will be indicated as having
> failed
> > > > whether your tool understands this format or not.
> > > > > When mergeReruns is disabled (the default), each execution of a
> test
> > > will
> > > > be listed as a separate test case.
> > > >
> > > > It would not resolve all the faky tests for sure but it would at
> least
> > > > reduce the noise. I see this as a means to get to green builds
> faster.
> > I
> > > > played a bit with this setting and I discovered [2]. I was hoping
> that
> > > [3]
> > > > could help to resolve it but I need to confirm.
> > > >
> > > > 2) I suppose that we would still have flaky tests preventing us from
> > > > getting a green build even with the setting in place. For those, I
> > think
> > > > that we need to review them one by one and decide whether we want to
> > fix
> > > or
> > > > disable them. This is a short term effort to help us get to green
> > builds.
> > > >
> > > > 3) When we get to a point where we can get green builds consistently,
> > we
> > > > could enforce it.
> > > >
> > > > 4) Flaky tests won't disappear with thi

[jira] [Created] (KAFKA-15893) Bump MetadataVersion for directory assignments

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15893:
---

 Summary: Bump MetadataVersion for directory assignments
 Key: KAFKA-15893
 URL: https://issues.apache.org/jira/browse/KAFKA-15893
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez






--
This message was sent by Atlassian Jira
(v8.20.10#820010)