RE: Kafka consumer group crashing and not able to consume once service is up

Hi Philip, Please find logs of Kafka, Zookeeper, and Service logs from the below link. https://github.com/santhoshadityam/Kafka_consumer_issues Thanks, Santhosh Aditya From: Marigowda, Santhosh Aditya Sent: Friday, February 9, 2024 11:55 AM To: Philip Nee ; Jain, Ankit Cc: dev@kafka.apache.org;

[jira] [Created] (KAFKA-16238) ConnectRestApiTest broken after KIP-1004

Mickael Maison created KAFKA-16238: -- Summary: ConnectRestApiTest broken after KIP-1004 Key: KAFKA-16238 URL: https://issues.apache.org/jira/browse/KAFKA-16238 Project: Kafka Issue Type: Impr

[jira] [Created] (KAFKA-16239) Clean up references to non-existent IntegrationTestHelper

Divij Vaidya created KAFKA-16239: Summary: Clean up references to non-existent IntegrationTestHelper Key: KAFKA-16239 URL: https://issues.apache.org/jira/browse/KAFKA-16239 Project: Kafka Iss

[jira] [Resolved] (KAFKA-14616) Topic recreation with offline broker causes permanent URPs

[ https://issues.apache.org/jira/browse/KAFKA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim resolved KAFKA-14616. --- Fix Version/s: 3.7.0 Assignee: Colin McCabe Resolution: Fixed > Topic recrea

[jira] [Resolved] (KAFKA-16162) New created topics are unavailable after upgrading to 3.7

[ https://issues.apache.org/jira/browse/KAFKA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim resolved KAFKA-16162. --- Fix Version/s: 3.7.0 Resolution: Fixed > New created topics are unavailable after upg

[jira] [Resolved] (KAFKA-16238) ConnectRestApiTest broken after KIP-1004

[ https://issues.apache.org/jira/browse/KAFKA-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-16238. Fix Version/s: 3.8.0 Resolution: Fixed > ConnectRestApiTest broken after KIP-1004 >

[jira] [Created] (KAFKA-16240) Flaky test PlaintextAdminIntegrationTest.testReplicaCanFetchFromLogStartOffsetAfterDeleteRecords(String).quorum=kraft

Gantigmaa Selenge created KAFKA-16240: - Summary: Flaky test PlaintextAdminIntegrationTest.testReplicaCanFetchFromLogStartOffsetAfterDeleteRecords(String).quorum=kraft Key: KAFKA-16240 URL: https://issues.apach

Re: [DISCUSS] KIP-981: Manage Connect topics with custom implementation of Admin

Hi everyone, I'm still keen on implementing this. I'll start a vote if I don't hear back in the next few days Thanks Omnia On Mon, Jan 15, 2024 at 8:12 PM Omnia Ibrahim wrote: > Hi It has been a while! but can I have a feedback on this. It would be > nice to unify this between MM2 and Connect a

Github build queue

Hey folks, I recently learned about Github's Merge Queue feature, and I think it could help us out. Essentially, when you hit the Merge button on a PR, it will add the PR to a queue and let you run a CI job before merging. Just something simple like compile + static analysis would probably save u

Re: Github build queue

+1, would love this. On Fri, Feb 9, 2024, 10:04 David Arthur wrote: > Hey folks, > > I recently learned about Github's Merge Queue feature, and I think it could > help us out. > > Essentially, when you hit the Merge button on a PR, it will add the PR to a > queue and let you run a CI job before

Re: Github build queue

Hi David, I think this is a helpful thing (and something I hoped we would use when I learned about it), but it does require the validation checks to be reliable (or else the PR won't be merged). Sounds like you are suggesting to skip the tests for the merge queue validation. Could we perhaps inclu

[VOTE] 3.7.0 RC4

Hello Kafka users, developers and client-developers, This is the second candidate we are considering for release of Apache Kafka 3.7.0. Major changes include: - Early Access to KIP-848 - the next generation of the consumer rebalance protocol - Early Access to KIP-858: Adding JBOD support to KRaft

Re: Github build queue

Hi David, I like the idea, it will solve the problem we've seen a couple of times in the last 2 weeks where compilation for some Scala version failed, it was probably overlooked during the PR build because of the flakiness of tests and the compilation failure was buried among the amount of failed t

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2632

See

Re: Github build queue

I do think we can add a PR to the merge queue while bypassing branch potections (like we do for the Merge button today), but I'm not 100% sure. I like the idea of running unit tests, though I don't think we have data on how long just the unit tests run on Jenkins (since we run the "test" target whi

Re: Github build queue

Hi David, +1 on that strategy. I see several flaky tests that aren't marked with @Tag("integration") or @IntegrationTest, and I think those would make using the unitTest target ineffective here. We could also start a new tag @Tag("flaky") and exclude that. Thanks, Greg On Fri, Feb 9, 2024 at 8:

Re: Kafka-Streams-Scala for Scala 3

Hi Matthias, Hi Josep, I'm afraid I can't do the KIP thing as the signup process for Apache Confluence requires sending me a password reset link via E-Mail and said E-Mail doesn't seem to reach me for some reason. I've contacted the Apache infrastructure team but haven't yet heard back from them.

Re: Github build queue

Why can't we add @Tag("integration") for all of those tests? Seems like that would not be too hard. Ismael On Fri, Feb 9, 2024 at 9:03 AM Greg Harris wrote: > Hi David, > > +1 on that strategy. > > I see several flaky tests that aren't marked with @Tag("integration") > or @IntegrationTest, and

Re: Github build queue

Regarding "Split our CI "test" job into unit and integration so we can start collecting data on those suites", can we run these 2 tasks in the same machine? So they won't need to compile classes twice for the same exact code? On Fri, Feb 9, 2024 at 6:05 PM Ismael Juma wrote: > Why can't we add @

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #93

See

Re: Github build queue

> Regarding "Split our CI "test" job into unit and integration I believe all of the "steps" inside the "stage" directive are run on the same node sequentially. I think we could do something like steps { doValidation() doUnitTest() doIntegrationTest() tryStreamsArchetype() } and it should

Re: Github build queue

Hey Ismael, I took David's statement here: > Split our CI "test" job into unit and integration so we can start collecting > data on those suites to include moving all the flaky tests to the integrationTest target by adding the annotation. We can do that while the merge queue is just running the

Re: Github build queue

Please check https://github.com/apache/kafka/pull/14186 before making the `unitTest` and `integrationTest` split. Ismael On Fri, Feb 9, 2024 at 9:16 AM Josep Prat wrote: > Regarding "Split our CI "test" job into unit and integration so we can > start collecting data on those suites", can we run

Re: Kafka-Streams-Scala for Scala 3

Hi Matthias, It's not adding a new functionality but it's changing the way to generate artifacts. In the end we are talking about generating a new binary. I could live with not having a KIP, but a DISCUSS thread I think it's necessary. This signals the community members and maintainers that their

[jira] [Created] (KAFKA-16241) Kafka Streams hits IllegalStateException trying to recycle a task

Matthias J. Sax created KAFKA-16241: --- Summary: Kafka Streams hits IllegalStateException trying to recycle a task Key: KAFKA-16241 URL: https://issues.apache.org/jira/browse/KAFKA-16241 Project: Kafk

Re: Kafka-Streams-Scala for Scala 3

Hey Josep, I'm glad you agree that a KIP is not needed here, and I agree with you that how to publish these artifacts should be discussed with the Kafka team. In fact, this is what I created this thread for 😁 This is my first time contributing to Kafka, so I'm going to have to ask what a DISCUSS t

[jira] [Created] (KAFKA-16242) Mark flaky tests as integration tests

Greg Harris created KAFKA-16242: --- Summary: Mark flaky tests as integration tests Key: KAFKA-16242 URL: https://issues.apache.org/jira/browse/KAFKA-16242 Project: Kafka Issue Type: Test

Re: Kafka-Streams-Scala for Scala 3

Hi Matthias, Yes, it's just a matter of adding the [DISCUSS] prefix in the subject. By the way, I didn't say this won't need a KIP, just that I won't be pushing for it, but other maintainers might think it's needed. For the discuss thread, you should write down what changes in the build and what

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2633

See

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi, Andrew, Thanks for the KIP. A few comments below. 10. ShareFetchResponse: To consume transactional data, currently FetchResponse includes the AbortedTransactions fields for the client to properly skip aborted records. ShareFetchResponse doesn't include that. How do we prevent the consumer fro

Re: Github build queue

Hey all, I implemented a fairly aggressive PR [1] to demote flaky tests to integration tests, and the end result is a much faster (10m locally, 1h on Jenkins) build which is also very reliable. I believe this would make unitTest suitable for use in the merge queue, with the caveat that it doesn't

Re: Github build queue

Nice! Ismael On Fri, Feb 9, 2024 at 3:43 PM Greg Harris wrote: > Hey all, > > I implemented a fairly aggressive PR [1] to demote flaky tests to > integration tests, and the end result is a much faster (10m locally, > 1h on Jenkins) build which is also very reliable. > > I believe this would mak

Re: Github build queue

Also, on the mockito stubbings point, we did upgrade to Mockito 5.8 for the Java 11 and newer builds: https://github.com/apache/kafka/blob/trunk/gradle/dependencies.gradle#L64 So, we should be good when it comes to that too. Ismael On Fri, Feb 9, 2024 at 4:15 PM Ismael Juma wrote: > Nice! > >

[DISCUSS] Kafka-Streams-Scala for Scala 3

Hey there, I'd like to discuss a Scala 3 release of the Kafka-Streams-Scala library. As you might have seen already, I have recently created a ticket https://issues.apache.org/jira/browse/KAFKA-16237 and a PR https://github.com/apache/kafka/pull/15338 to move this forward. The changes required to

Re: Github build queue

I tried to enable the merge queue on my public fork, but the option is not available. I did a little searching and it looks like ASF does not allow this feature to be used. I've filed an INFRA ticket to ask again https://issues.apache.org/jira/browse/INFRA-25485 -David On Fri, Feb 9, 2024 at 7:18

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2634

See