[jira] [Resolved] (KAFKA-14421) OffsetFetchRequest throws NPE Exception

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14421.
---
Resolution: Resolved

Resolved in the later release.

> OffsetFetchRequest throws NPE Exception
> ---
>
> Key: KAFKA-14421
> URL: https://issues.apache.org/jira/browse/KAFKA-14421
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.0, 0.10.2.1, 0.10.2.2
>Reporter: yws
>Assignee: yws
>Priority: Major
> Attachments: image-2022-11-27-22-28-52-165.png, 
> image-2022-11-27-22-41-45-358.png
>
>
> when I use 0.10.2 client  send Metadata request to  0.10.0 server,  NPE 
> exception happens,
>  !image-2022-11-27-22-28-52-165.png! 
> the NPE exception quite confused me,  because if  just send Metadata request 
> doest not cause the NPE exception occurs, after troubleshooting the problem, 
> It is the NetworkClient#poll call  ConsumerNetworkClient#trySend  and further 
> call NetworkClient#doSendwhen trying to build OffsetFetchRequest, because 
> the 0.10.0 server doest not support  fetch all TopicPartitions, it throw 
> UnsupportedVersionException, 
> {code:java}
> private void doSend(ClientRequest clientRequest, boolean isInternalRequest, 
> long now) {
> String nodeId = clientRequest.destination();
> ..
> AbstractRequest request = null;
> AbstractRequest.Builder builder = clientRequest.requestBuilder();
> try {
> NodeApiVersions versionInfo = nodeApiVersions.get(nodeId);
> // Note: if versionInfo is null, we have no server version 
> information. This would be
> // the case when sending the initial ApiVersionRequest which 
> fetches the version
> // information itself.  It is also the case when 
> discoverBrokerVersions is set to false.
> if (versionInfo == null) {
> if (discoverBrokerVersions && log.isTraceEnabled())
> log.trace("No version information found when sending 
> message of type {} to node {}. " +
> "Assuming version {}.", clientRequest.apiKey(), 
> nodeId, builder.version());
> } else {
> short version = 
> versionInfo.usableVersion(clientRequest.apiKey());
> builder.setVersion(version);
> }
> // The call to build may also throw UnsupportedVersionException, 
> if there are essential
> // fields that cannot be represented in the chosen version.
> request = builder.build();
> } catch (UnsupportedVersionException e) {
> // If the version is not supported, skip sending the request over 
> the wire.
> // Instead, simply add it to the local queue of aborted requests.
> log.debug("Version mismatch when attempting to send {} to {}",
> clientRequest.toString(), clientRequest.destination(), e);
> ClientResponse clientResponse = new 
> ClientResponse(clientRequest.makeHeader(),
> clientRequest.callback(), clientRequest.destination(), 
> now, now,
> false, e, null);
> abortedSends.add(clientResponse);
> return;
> }
> {code}
>  !image-2022-11-27-22-41-45-358.png! 
> until now, all are expected, but unfortunately, in catch 
> UnsupportedVersionException code block, clientRequest.toString need to call 
> requestBuilder#toString, that is OffsetFetchRequest's Builder#toString, when 
> partition is ALL_TOPIC_PARTITIONS, it is null, therefore it cause the 
> unexpected NPE, and make the normal MetadataRequest failed..  
> {code:java}
> catch (UnsupportedVersionException e) {
>  
> log.debug("Version mismatch when attempting to send {} to {}",
> clientRequest.toString(), clientRequest.destination(), e);
> ClientResponse clientResponse = new 
> ClientResponse(clientRequest.makeHeader(),
> clientRequest.callback(), clientRequest.destination(), 
> now, now,
> false, e, null);
> abortedSends.add(clientResponse);
> return;
> }
> ClientRequest#toString()
>public String toString() {
> return "ClientRequest(expectResponse=" + expectResponse +
> ", callback=" + callback +
> ", destination=" + destination +
> ", correlationId=" + correlationId +
> ", clientId=" + clientId +
> ", createdTimeMs=" + createdTimeMs +
> ", requestBuilder=" + requestBuilder +
> ")";
> }
>   OffsetFetchRequest's Builder#toString
> public String toString() {
> StringBuilder bld = new StringBuilder();
> bld.append("(type=OffsetFetchRe

[jira] [Created] (KAFKA-15328) Inter broker partition reassignment triggers offline log dirs randomly

[dgiri (Jira)]

dgiri created KAFKA-15328:
-

 Summary: Inter broker partition reassignment triggers offline log 
dirs randomly
 Key: KAFKA-15328
 URL: https://issues.apache.org/jira/browse/KAFKA-15328
 Project: Kafka
  Issue Type: Bug
  Components: core, tools
Affects Versions: 3.3.2
 Environment: RHEL 8.8 (Ootpa)
Reporter: dgiri
 Attachments: kafka_partition_reassignment_offline_lod_dir_errors.txt

When running the intra-broker partition reassignments from one disk to another, 
kafka triggers offline log dir. This happens randomly and can be reproduced 
only after running the intra-broker reassignments multiple times.

 

Observations made:
 # After running the partition reassignment from (/data1 to /data2), kafka 
creates a *_topic-partition.uniqueId-future_* location on /data2 and starts 
copying the data.
 # Partition reassignment completes as expected by renaming the 
_*topic-partition.uniqueId-future*_ location __ to _*topic-partition*_ on 
/data2.
 # Within few milliseconds kafka still looks for 
_*topic-partition.uniqueId-future*_ location{_},{_} since it won't find that 
location anymore it marks the /data2 log dir as offline

 

Broker logs filtered with particular partition and error has been attached for 
more details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2084

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15329) Make default `remote.log.metadata.manager.class.name` as topic based RLMM

[Luke Chen (Jira)]

Luke Chen created KAFKA-15329:
-

 Summary: Make default `remote.log.metadata.manager.class.name` as 
topic based RLMM
 Key: KAFKA-15329
 URL: https://issues.apache.org/jira/browse/KAFKA-15329
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
 Fix For: 3.6.0


As described in 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Configs.1]
 , we should set default "remote.log.metadata.manager.class.name" as topic 
based RLMM.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15330) Migration from ZK to KRaft works with 3.4 but fails from 3.5 upwards

[Roland Sommer (Jira)]

Roland Sommer created KAFKA-15330:
-

 Summary: Migration from ZK to KRaft works with 3.4 but fails from 
3.5 upwards
 Key: KAFKA-15330
 URL: https://issues.apache.org/jira/browse/KAFKA-15330
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.1, 3.5.0
 Environment: Debian Bookworm/12.1
kafka 3.4 and 3.5 / scala 2.13
OpenJDK Runtime Environment (build 17.0.8+7-Debian-1deb12u1)
Reporter: Roland Sommer


We recently did some migration testing from our old ZK-based kafka clusters to 
KRaft while still being on kafka 3.4. The migration tests succeeded at first 
try. In the meantime we updated to kafka 3.5/3.5.1 and now we wanted to 
continue our migration work, which ran into unexpected problems.

On the controller we get messages like:
{code:java}
Aug 10 06:49:33 kafkactl01 kafka-server-start.sh[48572]: [2023-08-10 
06:49:33,072] INFO [KRaftMigrationDriver id=495] Still waiting for all 
controller nodes ready to begin the migration. due to: Missing apiVersion from 
nodes: [514, 760] 
(org.apache.kafka.metadata.migration.KRaftMigrationDriver){code}
On the broker side, we see:
{code:java}
06:52:56,109] INFO [BrokerLifecycleManager id=6 isZkBroker=true] Unable to 
register the broker because the RPC got timed out before it could be sent. 
(kafka.server.BrokerLifecycleManager){code}
If we reinstall the same development cluster with kafka 3.4, using the exact 
same steps provided by your migration documentation (only difference is using 
{{inter.broker.protocol.version=3.4}} instead of 
{{{}inter.broker.protocol.version=3.5{}}}), everything works as expected. 
Updating to kafka 3.5/3.5.1 yields the same problems.

Testing is done on a three-node kafka cluster with a three-node zookeeper 
ensemble and a three-node controller setup.

Besides our default configuration containing the active zookeeper hosts etc., 
this is what was added on the brokers:
{code:java}
# Migration
advertised.listeners=PLAINTEXT://kafka03:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
zookeeper.metadata.migration.enable=true
controller.quorum.voters=495@kafkactl01:9093,760@kafkactl02:9093,514@kafkactl03:9093
controller.listener.names=CONTROLLER
{code}
The main controller config looks like this:
{code:java}
process.roles=controller
node.id=495
controller.quorum.voters=495@kafkactl01:9093,760@kafkactl02:9093,514@kafkactl03:9093
listeners=CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
zookeeper.metadata.migration.enable=true
{code}
Both configs contain the identical {{zookeeper.connect}} settings, everything 
is setup automatically so it should be identical on every run and we can 
reliably reproduce migration success on kafka 3.4 and migration failure using 
the same setup with kafka 3.5.

There are other issues mentioning problems with ApiVersions like KAFKA-15230 - 
not quite sure if this is a duplicate of the underlying problem there.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

[Florin Akermann]

Thank you for the feedback.

> Not sure if this is the right phrasing?

You are right. I adjusted the phrasing accordingly.
Given your description of the current behavior, do I understand correctly
that the current documentation for the left join KStream-GlobalKtable is
out of date?
https://github.com/apache/kafka/blob/9318b591d7a57b9db1e7519986d78f0402cd5b5e/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java#L2948C7-L2948C7

> I would remove this from the KIP

I agree,Removed.
Plus, relevant doc links added.

> I think the way it's phrased is good. [...] We can cover details on the
PR.

Great. Yes, in general, I hope everybody agrees that we shouldn't add more
details to this KIP


On Thu, 10 Aug 2023 at 00:16, Matthias J. Sax  wrote:

> Thanks for the KIP.
>
> > left join KStream-GlobalTable: no longer drop left records with null-key
> and call KeyValueMapper with 'null' for left  key. The case where
> KeyValueMapper returns null is already handled in the current
> implementation.
>
> Not sure if this is the right phrasing? In the end, even now, the stream
> input record key can be null (cf
> https://issues.apache.org/jira/browse/KAFKA-10277) -- a stream record is
> only dropped if the `KeyValueMapper` returns `null` (note that the
> key-extractor has no default implemenation but is a required argument)
> -- this KIP would relax this case for left-join.
>
>
> > In the pull request all relevant Javadocs will be updated with the
> information on how to keep the old behavior for a given operator / method.
>
> I would remove this from the KIP -- I am also not sure if we should put
> it into the JavaDoc? -- I agree that it should go into the upgrade docs
> as well as "join section" in the docs:
>
> https://kafka.apache.org/35/documentation/streams/developer-guide/dsl-api.html#joining
>
> We also have
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics
> that we should update.
>
>
> > I added a remark about the repartition of null-key records.
>
> I think the way it's phrase is good. In the end, it's an optimization to
> drop records upstream (instead of piping them through the topic and drop
> the downstream), and thus we don't have to cover it in the KIP in
> detail. In general, for aggregations we can still apply the
> optimization, however, we need to be careful as we could also have two
> downstream operators with a shared repartition topic: for this case, we
> can only drop upstream if all downstream operator would drop null-key
> records anyway. We can cover details on the PR.
>
>
>
> -Matthias
>
>
>
> On 8/9/23 5:39 AM, Florin Akermann wrote:
> > Hi All,
> >
> > I added a remark about the repartition of null-key records.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Repartitionofnull-keyrecords
> >
> > Can we relax this constraint for any kind of repartitioning or should it
> > only be relaxed in the context of left stream-table and left/outer
> > stream-stream joins?
> >
> > Florin
> >
> > On Mon, 7 Aug 2023 at 13:23, Florin Akermann 
> > wrote:
> >
> >> Hi Lucas,
> >>
> >> Thanks. I added the point about the upgrade guide as well.
> >>
> >> Florin
> >>
> >> On Mon, 7 Aug 2023 at 11:06, Lucas Brutschy  .invalid>
> >> wrote:
> >>
> >>> Hi Florin,
> >>>
> >>> thanks for the KIP! This looks good to me. I agree that the precise
> >>> Java doc wording doesn't have to be discussed as part of the KIP.
> >>>
> >>> I would also suggest to include an update to
> >>> https://kafka.apache.org/documentation/streams/upgrade-guide
> >>>
> >>> Cheers,
> >>> Lucas
> >>>
> >>> On Mon, Aug 7, 2023 at 10:51 AM Florin Akermann
> >>>  wrote:
> 
>  Hi Both,
> 
>  Thanks.
>  I added remarks to account for this.
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks
> 
>  In short, let's add a note in the Java docs? The exact wording of the
> >>> note
>  can be scrutinized in the pull request?
> 
>  What do you think?
> 
> 
>  On Sun, 6 Aug 2023 at 19:41, Guozhang Wang <
> guozhang.wang...@gmail.com>
>  wrote:
> 
> > I'm just thinking we can try to encourage users to migrate from XX to
> > XXWithKey in the docs, giving this as one good example that the
> latter
> > can help you distinguish different scenarios whereas the former
> > cannot.
> >
> > On Fri, Aug 4, 2023 at 6:32 PM Matthias J. Sax 
> >>> wrote:
> >>
> >> Guozhang,
> >>
> >> thanks for pointing out ValueJoinerWithKey. In the end, it's just a
> >> documentation change, ie, point out that the passed in key could be
> >> `null` and similar?
> >>
> >> -Matthias
> >>
> >>
> >> On 8/2/23 3:20 PM, Guozhang Wang wrote:
> >>> Thanks Florin for the writeup,
> 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2085

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 400994 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(boolean, TestInfo) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(boolean, TestInfo) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(boolean, TestInfo) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(boolean, TestInfo) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(boolean, TestInfo) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(boolean, TestInfo) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(boolean, TestInfo) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(boolean, TestInfo) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldRe

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Mickael,

Thanks for voting, and for pointing out the mistake.
I've corrected it in the KIP now.
The proposed name is "QueuedReplicaToDirAssignments".

Best,

--
Igor

[jira] [Created] (KAFKA-15331) Handle remote log enabled topic deletion when leader is not available

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-15331:


 Summary: Handle remote log enabled topic deletion when leader is 
not available
 Key: KAFKA-15331
 URL: https://issues.apache.org/jira/browse/KAFKA-15331
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash


When a topic gets deleted, then there can be a case where all the replicas can 
be out of ISR. This case is not handled, See: 
[https://github.com/apache/kafka/pull/13947#discussion_r1289331347] for more 
details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Thanks Mickael!

The KIP has passed with 3 binding votes (Chris Egerton, Greg Harris, Mickael 
Maison) and 3 non-binding votes (Andrew Schofield, Yash Mayya, Kamal 
Chandraprakash).

I'll update the KIP status, meanwhile the PR is still pending: 
https://github.com/apache/kafka/pull/14093

From: dev@kafka.apache.org At: 08/08/23 08:33:21 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

Hi,

+1 (binding)

Thanks for the KIP!

On Mon, Aug 7, 2023 at 3:15 PM Hector Geraldino (BLOOMBERG/ 919 3RD A)
 wrote:
>
> Hello,
>
> I still need help from a committer to review/approve this (small) KIP, which 
adds a new BooleanConverter to the list of converters in Kafka Connect.
>
> The KIP has a companion PR implementing the feature as well.
>
> Thanks again!
> Sent from Bloomberg Professional for iPhone
>
> - Original Message -
> From: Hector Geraldino 
> To: dev@kafka.apache.org
> At: 08/01/23 11:48:23 UTC-04:00
>
>
> Hi,
>
> Still missing one binding vote for this (very small) KIP to pass :)
>
> From: dev@kafka.apache.org At: 07/28/23 09:37:45 UTC-4:00To:  
dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect
>
> Hi everyone,
>
> Thanks everyone who has reviewed and voted for this KIP.
>
> So far it has received 3 non-binding votes (Andrew Schofield, Yash Mayya, 
Kamal
> Chandraprakash) and 2 binding votes (Chris Egerton, Greg Harris)- still shy of
> one binding vote to pass.
>
> Can we get help from a committer to push it through?
>
> Thank you!
> Hector
>
> Sent from Bloomberg Professional for iPhone
>
> - Original Message -
> From: Greg Harris 
> To: dev@kafka.apache.org
> At: 07/26/23 12:23:20 UTC-04:00
>
>
> Hey Hector,
>
> Thanks for the straightforward and clear KIP!
> +1 (binding)
>
> Thanks,
> Greg
>
> On Wed, Jul 26, 2023 at 5:16 AM Chris Egerton  wrote:
> >
> > +1 (binding)
> >
> > Thanks Hector!
> >
> > On Wed, Jul 26, 2023 at 3:18 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > +1 (non-binding). Thanks for the KIP!
> > >
> > > On Tue, Jul 25, 2023 at 11:12 PM Yash Mayya  wrote:
> > >
> > > > Hi Hector,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Yash
> > > >
> > > > On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield <
> > > > andrew_schofield_j...@outlook.com> wrote:
> > > >
> > > > > Thanks for the KIP. As you say, not that controversial.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > > > > hgerald...@bloomberg.net> wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka
> > > Connect)
> > > > > have a limited scope and shouldn't be controversial. I'm opening a
> > > voting
> > > > > thread with the hope that it can be included in the next upcoming 3.6
> > > > > release.
> > > > > >
> > > > > > Here are some links:
> > > > > >
> > > > > > KIP:
> > > > >
> > > >
> > >
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte
> r+to+Kafka+Connect
> > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-15248
> > > > > > Discussion thread:
> > > > > https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt
> > > > > > Pull Request: https://github.com/apache/kafka/pull/14093
> > > > > >
> > > > > > Thanks!
> > > > >
> > > > >
> > > > >
> > > >
> > >

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

[Matthias J. Sax]

Good catch about the JavaDocs.

Seems we missed to update them when we did K10277. Would you like to do 
a PR to fix them right away for upcoming 3.6 release?


If there is no more other comments, I think you can start a VOTE thread.


-Matthias

On 8/10/23 4:22 AM, Florin Akermann wrote:

Thank you for the feedback.


Not sure if this is the right phrasing?


You are right. I adjusted the phrasing accordingly.
Given your description of the current behavior, do I understand correctly
that the current documentation for the left join KStream-GlobalKtable is
out of date?
https://github.com/apache/kafka/blob/9318b591d7a57b9db1e7519986d78f0402cd5b5e/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java#L2948C7-L2948C7


I would remove this from the KIP


I agree,Removed.
Plus, relevant doc links added.


I think the way it's phrased is good. [...] We can cover details on the

PR.

Great. Yes, in general, I hope everybody agrees that we shouldn't add more
details to this KIP


On Thu, 10 Aug 2023 at 00:16, Matthias J. Sax  wrote:


Thanks for the KIP.


left join KStream-GlobalTable: no longer drop left records with null-key

and call KeyValueMapper with 'null' for left  key. The case where
KeyValueMapper returns null is already handled in the current
implementation.

Not sure if this is the right phrasing? In the end, even now, the stream
input record key can be null (cf
https://issues.apache.org/jira/browse/KAFKA-10277) -- a stream record is
only dropped if the `KeyValueMapper` returns `null` (note that the
key-extractor has no default implemenation but is a required argument)
-- this KIP would relax this case for left-join.



In the pull request all relevant Javadocs will be updated with the

information on how to keep the old behavior for a given operator / method.

I would remove this from the KIP -- I am also not sure if we should put
it into the JavaDoc? -- I agree that it should go into the upgrade docs
as well as "join section" in the docs:

https://kafka.apache.org/35/documentation/streams/developer-guide/dsl-api.html#joining

We also have

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics
that we should update.



I added a remark about the repartition of null-key records.


I think the way it's phrase is good. In the end, it's an optimization to
drop records upstream (instead of piping them through the topic and drop
the downstream), and thus we don't have to cover it in the KIP in
detail. In general, for aggregations we can still apply the
optimization, however, we need to be careful as we could also have two
downstream operators with a shared repartition topic: for this case, we
can only drop upstream if all downstream operator would drop null-key
records anyway. We can cover details on the PR.



-Matthias



On 8/9/23 5:39 AM, Florin Akermann wrote:

Hi All,

I added a remark about the repartition of null-key records.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Repartitionofnull-keyrecords


Can we relax this constraint for any kind of repartitioning or should it
only be relaxed in the context of left stream-table and left/outer
stream-stream joins?

Florin

On Mon, 7 Aug 2023 at 13:23, Florin Akermann 
wrote:


Hi Lucas,

Thanks. I added the point about the upgrade guide as well.

Florin

On Mon, 7 Aug 2023 at 11:06, Lucas Brutschy 
.invalid>

wrote:


Hi Florin,

thanks for the KIP! This looks good to me. I agree that the precise
Java doc wording doesn't have to be discussed as part of the KIP.

I would also suggest to include an update to
https://kafka.apache.org/documentation/streams/upgrade-guide

Cheers,
Lucas

On Mon, Aug 7, 2023 at 10:51 AM Florin Akermann
 wrote:


Hi Both,

Thanks.
I added remarks to account for this.




https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks


In short, let's add a note in the Java docs? The exact wording of the

note

can be scrutinized in the pull request?

What do you think?


On Sun, 6 Aug 2023 at 19:41, Guozhang Wang <

guozhang.wang...@gmail.com>

wrote:


I'm just thinking we can try to encourage users to migrate from XX to
XXWithKey in the docs, giving this as one good example that the

latter

can help you distinguish different scenarios whereas the former
cannot.

On Fri, Aug 4, 2023 at 6:32 PM Matthias J. Sax 

wrote:


Guozhang,

thanks for pointing out ValueJoinerWithKey. In the end, it's just a
documentation change, ie, point out that the passed in key could be
`null` and similar?

-Matthias


On 8/2/23 3:20 PM, Guozhang Wang wrote:

Thanks Florin for the writeup,

One quick thing I'd like to bring up is that in KIP-149
(





https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner

)

we introdu

[jira] [Resolved] (KAFKA-15291) Implement Versioned interfaces in common Connect plugins

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-15291.
-
Resolution: Fixed

> Implement Versioned interfaces in common Connect plugins
> 
>
> Key: KAFKA-15291
> URL: https://issues.apache.org/jira/browse/KAFKA-15291
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Greg Harris
>Assignee: Aindriú Lavelle
>Priority: Major
> Fix For: 3.6.0
>
>
> In KAFKA-14863, we changed the plugin scanning logic to allow plugins to 
> opt-in to the Versioned interface individually, when previously it was 
> limited to Connector plugins.
> To take advantage of this change, we should have all of the plugins built via 
> the Kafka repository opt-in, and provide the environment's Kafka version from 
> the AppInfoParser.getVersion().
> See the FileStreamSinkConnector as an example of the the version() method 
> implementation.
> All subclasses of Converter, HeaderConverter, Transformation, Predicate, and 
> ConnectorClientConfigOverridePolicy should implement Versioned. The 
> interfaces themselves will _not_ extend Versioned, as that would be a 
> backwards-incompatible change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15332) Eligible Leader Replicas

[Calvin Liu (Jira)]

Calvin Liu created KAFKA-15332:
--

 Summary: Eligible Leader Replicas
 Key: KAFKA-15332
 URL: https://issues.apache.org/jira/browse/KAFKA-15332
 Project: Kafka
  Issue Type: New Feature
Reporter: Calvin Liu
Assignee: Calvin Liu


A root ticket for the KIP-966



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15333) Flaky build failure throwing Connect Exception: Could not connect to server....

[Philip Nee (Jira)]

Philip Nee created KAFKA-15333:
--

 Summary: Flaky build failure throwing Connect Exception: Could not 
connect to server
 Key: KAFKA-15333
 URL: https://issues.apache.org/jira/browse/KAFKA-15333
 Project: Kafka
  Issue Type: Test
  Components: connect, unit tests
Reporter: Philip Nee


We frequently observe flaky build failure with the following message.  The is 
from the most recent PR post 3.5.0:

 
{code:java}
> Task :generator:testClasses UP-TO-DATE
Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not read message 
from '/127.0.0.1:38354'.
at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:94)
at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at 
org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException
at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:72)
at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
... 6 more

> Task :streams:upgrade-system-tests-26:unitTest
org.gradle.internal.remote.internal.ConnectException: Could not connect to 
server [3156f144-9a89-4c47-91ad-88a8378ec726 port:37889, 
addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1].
at 
org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:67)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedClient.getConnection(MessageHubBackedClient.java:36)
at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:103)
at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at 
worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at 
worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:122)
at 
org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.tryConnect(TcpOutgoingConnector.java:81)
at 
org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:54)
... 5 more {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

[Igor Fomenko]

I don't mind you being a bit picky. I think it is a great discussion and it
helps me too. For example I clearly see now that the problem of aggregation
still needs to be solved for this use case.
Please see my answers below.

I used an example of OrderEvents to OrderItems relationship as 1:1 just to
demonstrate that even in tht simple case the existing table-table join on
FK will not work. However the use case I have in general may have 1:1, 1:0,
or 1:n relations. One complex business entity I had to deal with called
"transportation waybill" that has 25 child tables. Number of child records
in each child table could be 0:n for each record in the main waybill table.
When an event is generated for a certain waybill then a "complete" waybill
needs to be assembled from the subset of child tables. The subset of child
tables for waybill data assembly depends on the event type (any event type
has waybillId). There is also some additional filtering mapping and
enrichment that needs to be done in a real use case that is not relevant to
what we discuss. As you can see the use case is very complex and this is
why I wanted to distill it to very simple terms that are relevant to this
discussion.

Now I am switching back to the simple example of OrderEvent with OrderItems.
Please note that OrderEvent is a derived message. It is derived by joining
the actual event message that has orderId as its key with the Order message
that also has OrderId as its key. Because joining these two messages is
trivial I did not include this part and stated that we are sharing from
the  Order Event message right away.
So to summarize: We need to join each OrderEvent message (OrderId is key)
with 0 or 1 or many orderItems messages (OrderItem is the key and orderID
is one of the message fields).

Now, let's consider your solution:
1. We do not need to aggregate orderEvents around the key since we need to
provide an output for each orderEvent (each orderEvent needs to be joined
with an aggregate of OrderItems related to this orderEvent). So we can skip
this step.
2. Because OrderItems are multiple distinct records for each OrderId we can
not rekey them with OrderId PK to the table, uness we do some sort of
aggregation for them. So let's say we rekey orderItems with orderId and
aggregate each record field into an array. I think we also need to
co-partition with OrderEvents.
3. Now we can do stream-table join the orderEvents stream with the
OrderItemsAggregated table using the OrderId key that is common for both.

So the conclusion is that your solution will work with some tweaking
(basically aggregating on OrderItems instead of on events).
While this solution will work it has several issues as follows:

   - This solution was considered when in KIP-213 for the existing
   table-table FK join. There is a discussion on disadvantages of using this
   approach in the article related to KIP-213 and I think the same
   disadvantages will apply to this KIP. Please see here:
   
https://www.confluent.io/blog/data-enrichment-with-kafka-streams-foreign-key-joins/#workaround

   - I see FK join as the common operation in data manipulation so it would
   be nice to have a shortcut for it and not to try to design it from existing
   functionality all the time. Consider the real use case I discussed at the
   beginning when a business entity has 25 children
   - This solution similarly to mine is "mudding the water" by providing a
   hybrid outcome join + aggregate. At list with my proposal we could
   potentially control it with the flag, or maybe create some special
   aggregate that could be chained after (don't know how to do it yet :-))

Any thoughts?

Regards,

Igor

On Wed, Aug 9, 2023 at 7:19 PM Matthias J. Sax  wrote:

> Thanks for the details. And sorry for being a little bit picky. My goal
> is to really understand the use-case and the need for this KIP. It's a
> massive change and I just want to ensure we don't add (complex) things
> unnecessarily.
>
>
> So you have a streams of "orderEvents" with key=orderId. You cannot
> represent them as a KTable, because `orderId` is not a PK, but just an
> identify that a message belongs to a certain order. This part I understand.
>
> You also have a KTable "orderItems", with orderId as a value-field.
>
>
>
> >  Relationship between parent and child messages is 1:1
>
> If I understand correctly, you want to join on orderId. If the join is
> 1:1, it means that there is only a single table-record for each unique
> orderId. Thus, orderId could be the PK of the table. If that's correct,
> you could use orderId as the key of "orderItems" and do a regular
> stream-table join. -- Or do I miss something?
>
>
>
> > and to send it only once to the target system as one ‘complete order >
> message for each new ‘order event’ message.
>
> This sound like an aggregation to me, not a join? It seems that an order
> consists of multiple "orderEvent" messages, and you would want to
> aggregate them based on orderId (plus add 

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

[Florin Akermann]

Thanks.

Here is the PR for this catch: https://github.com/apache/kafka/pull/14187

On Thu, 10 Aug 2023 at 19:04, Matthias J. Sax  wrote:

> Good catch about the JavaDocs.
>
> Seems we missed to update them when we did K10277. Would you like to do
> a PR to fix them right away for upcoming 3.6 release?
>
> If there is no more other comments, I think you can start a VOTE thread.
>
>
> -Matthias
>
> On 8/10/23 4:22 AM, Florin Akermann wrote:
> > Thank you for the feedback.
> >
> >> Not sure if this is the right phrasing?
> >
> > You are right. I adjusted the phrasing accordingly.
> > Given your description of the current behavior, do I understand correctly
> > that the current documentation for the left join KStream-GlobalKtable is
> > out of date?
> >
> https://github.com/apache/kafka/blob/9318b591d7a57b9db1e7519986d78f0402cd5b5e/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java#L2948C7-L2948C7
> >
> >> I would remove this from the KIP
> >
> > I agree,Removed.
> > Plus, relevant doc links added.
> >
> >> I think the way it's phrased is good. [...] We can cover details on the
> > PR.
> >
> > Great. Yes, in general, I hope everybody agrees that we shouldn't add
> more
> > details to this KIP
> >
> >
> > On Thu, 10 Aug 2023 at 00:16, Matthias J. Sax  wrote:
> >
> >> Thanks for the KIP.
> >>
> >>> left join KStream-GlobalTable: no longer drop left records with
> null-key
> >> and call KeyValueMapper with 'null' for left  key. The case where
> >> KeyValueMapper returns null is already handled in the current
> >> implementation.
> >>
> >> Not sure if this is the right phrasing? In the end, even now, the stream
> >> input record key can be null (cf
> >> https://issues.apache.org/jira/browse/KAFKA-10277) -- a stream record
> is
> >> only dropped if the `KeyValueMapper` returns `null` (note that the
> >> key-extractor has no default implemenation but is a required argument)
> >> -- this KIP would relax this case for left-join.
> >>
> >>
> >>> In the pull request all relevant Javadocs will be updated with the
> >> information on how to keep the old behavior for a given operator /
> method.
> >>
> >> I would remove this from the KIP -- I am also not sure if we should put
> >> it into the JavaDoc? -- I agree that it should go into the upgrade docs
> >> as well as "join section" in the docs:
> >>
> >>
> https://kafka.apache.org/35/documentation/streams/developer-guide/dsl-api.html#joining
> >>
> >> We also have
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics
> >> that we should update.
> >>
> >>
> >>> I added a remark about the repartition of null-key records.
> >>
> >> I think the way it's phrase is good. In the end, it's an optimization to
> >> drop records upstream (instead of piping them through the topic and drop
> >> the downstream), and thus we don't have to cover it in the KIP in
> >> detail. In general, for aggregations we can still apply the
> >> optimization, however, we need to be careful as we could also have two
> >> downstream operators with a shared repartition topic: for this case, we
> >> can only drop upstream if all downstream operator would drop null-key
> >> records anyway. We can cover details on the PR.
> >>
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 8/9/23 5:39 AM, Florin Akermann wrote:
> >>> Hi All,
> >>>
> >>> I added a remark about the repartition of null-key records.
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Repartitionofnull-keyrecords
> >>>
> >>> Can we relax this constraint for any kind of repartitioning or should
> it
> >>> only be relaxed in the context of left stream-table and left/outer
> >>> stream-stream joins?
> >>>
> >>> Florin
> >>>
> >>> On Mon, 7 Aug 2023 at 13:23, Florin Akermann <
> florin.akerm...@gmail.com>
> >>> wrote:
> >>>
>  Hi Lucas,
> 
>  Thanks. I added the point about the upgrade guide as well.
> 
>  Florin
> 
>  On Mon, 7 Aug 2023 at 11:06, Lucas Brutschy  >> .invalid>
>  wrote:
> 
> > Hi Florin,
> >
> > thanks for the KIP! This looks good to me. I agree that the precise
> > Java doc wording doesn't have to be discussed as part of the KIP.
> >
> > I would also suggest to include an update to
> > https://kafka.apache.org/documentation/streams/upgrade-guide
> >
> > Cheers,
> > Lucas
> >
> > On Mon, Aug 7, 2023 at 10:51 AM Florin Akermann
> >  wrote:
> >>
> >> Hi Both,
> >>
> >> Thanks.
> >> I added remarks to account for this.
> >>
> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks
> >>
> >> In short, let's add a note in the Java docs? The exact wording of
> the
> > note
> >> can be scrutinized in the pull request?
> >>

[VOTE] KIP-962 Relax non-null key requirement in Kafka Streams

[Florin Akermann]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams

[jira] [Resolved] (KAFKA-13187) Replace EasyMock and PowerMock with Mockito for DistributedHerderTest

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13187.
---
Fix Version/s: 3.6.0
   Resolution: Done

> Replace EasyMock and PowerMock with Mockito for DistributedHerderTest
> -
>
> Key: KAFKA-13187
> URL: https://issues.apache.org/jira/browse/KAFKA-13187
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: YI-CHEN WANG
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2086

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 403180 lines...]
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':streams:upgrade-system-tests-0110:integrationTest'.
> Process 'Gradle Test Executor 63' finished with non-zero exit value 1
  This problem might be caused by incorrect test process configuration.
  For more on test execution, please refer to 
https://docs.gradle.org/8.2.1/userguide/java_testing.html#sec:test_execution in 
the Gradle documentation.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD FAILED in 2h 55m 50s
237 actionable tasks: 124 executed, 113 up-to-date

Publishing build scan...

Publishing failed.

The response from https://ge.apache.org/scans/publish/gradle/3.13.4/token was 
not from Gradle Enterprise.
The specified server address may be incorrect, or your network environment may 
be interfering.

Please report this problem to your Gradle Enterprise administrator via 
https://ge.apache.org/help and include the following via copy/paste:

--
Gradle version: 8.2.1
Plugin version: 3.13.4
Request URL: https://ge.apache.org/scans/publish/gradle/3.13.4/token
Request ID: 407c0dcd-3324-4619-b742-e4bae102d0ca
Response status code: 502
Response content type: text/html
--


See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2023-08-10-18-02-37.html
A fine-grained performance profile is available: use the --scan option.

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 17 and Scala 2.13

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgu

[jira] [Resolved] (KAFKA-15239) producerPerformance system test for old client failed after v3.5.0

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-15239.
-
Fix Version/s: 3.6.0
 Reviewer: Greg Harris
   Resolution: Fixed

> producerPerformance system test for old client failed after v3.5.0
> --
>
> Key: KAFKA-15239
> URL: https://issues.apache.org/jira/browse/KAFKA-15239
> Project: Kafka
>  Issue Type: Test
>  Components: system tests
>Affects Versions: 3.6.0
>Reporter: Luke Chen
>Assignee: Federico Valeri
>Priority: Major
> Fix For: 3.6.0
>
>
> While running producer performance tool in system test for old client (ex: 
> quota_test), we will try to run with the dev-branch's jar file, to make sure 
> it is backward compatible, as described 
> [here|https://github.com/apache/kafka/blob/trunk/tests/kafkatest/services/performance/producer_performance.py#L86-L88],.
> {code:java}
> # In order to ensure more consistent configuration between versions, always 
> use the ProducerPerformance tool from the development branch {code}
>  
> But in KAFKA-14525, we're moving tools from core module to a separate tool 
> module, we're actually breaking the backward compatibility. We should fix the 
> system test. Also maybe we should also mention anywhere about this backward 
> compatibility issue?
> Note:
> This is the command run in system test. Suppose it's testing old client 3.4.0 
> (file put under `~/Downloads/kafka_2.13-3.4.0` in my env), and running under 
> the latest trunk env.
> {code:java}
> > for file in ./tools/build/libs/kafka-tools*.jar; do 
> > CLASSPATH=$CLASSPATH:$file; done; for file in 
> > ./tools/build/dependant-libs*/*.jar; do CLASSPATH=$CLASSPATH:$file; done; 
> > export CLASSPATH;  export 
> > KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:config/tools-log4j.properties";
> >  KAFKA_OPTS= KAFKA_HEAP_OPTS="-XX:+HeapDumpOnOutOfMemoryError" 
> > ~/Downloads/kafka_2.13-3.4.0/bin/kafka-run-class.sh 
> > org.apache.kafka.tools.ProducerPerformance --topic test_topic --num-records 
> > 5 --record-size 3000 --throughput -1 --producer-props 
> > bootstrap.servers=localhost:9092 client.id=overridden_id 
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/kafka/common/utils/ThroughputThrottler
>     at 
> org.apache.kafka.tools.ProducerPerformance.start(ProducerPerformance.java:101)
>     at 
> org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:52)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.kafka.common.utils.ThroughputThrottler
>     at 
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
>     at 
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
>     at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
>     ... 2 more
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-966: Eligible Leader Replicas

[Calvin Liu]

Hi everyone,
I'd like to discuss a series of enhancement to the replication protocol.

A partition replica can experience local data loss in unclean shutdown
scenarios where unflushed data in the OS page cache is lost - such as an
availability zone power outage or a server error. The Kafka replication
protocol is designed to handle these situations by removing such replicas
from the ISR and only re-adding them once they have caught up and therefore
recovered any lost data. This prevents replicas that lost an arbitrary log
suffix, which included committed data, from being elected leader.
However, there is a "last replica standing" state which when combined with
a data loss unclean shutdown event can turn a local data loss scenario into
a global data loss scenario, i.e., committed data can be removed from all
replicas. When the last replica in the ISR experiences an unclean shutdown
and loses committed data, it will be reelected leader after starting up
again, causing rejoining followers to truncate their logs and thereby
removing the last copies of the committed records which the leader lost
initially.

The new KIP will maximize the protection and provides MinISR-1 tolerance to
data loss unclean shutdown events.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas

Re: [VOTE] KIP-962 Relax non-null key requirement in Kafka Streams

[Matthias J. Sax]

+1 (binding)

On 8/10/23 12:31 PM, Florin Akermann wrote:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams

[jira] [Reopened] (KAFKA-14598) Fix flaky ConnectRestApiTest

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris reopened KAFKA-14598:
-
  Assignee: (was: Ashwin Pankaj)

> Fix flaky ConnectRestApiTest
> 
>
> Key: KAFKA-14598
> URL: https://issues.apache.org/jira/browse/KAFKA-14598
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Ashwin Pankaj
>Priority: Minor
>  Labels: flaky-test
>
> ConnectRestApiTest sometimes fails with the message
> {{ConnectRestError(404, '\n\n content="text/html;charset=ISO-8859-1"/>\nError 404 Not 
> Found\n\nHTTP ERROR 404 Not 
> Found\n\nURI:/connector-plugins/\nSTATUS:404\nMESSAGE:Not
>  
> Found\nSERVLET:-\n\n\n\n\n',
>  'http://172.31.1.75:8083/connector-plugins/')}}
> This happens because ConnectDistributedService.start() by default waits till 
> the the line
> {{Joined group at generation ..}} is visible in the logs.
> In most cases this is sufficient. But in the cases where the test fails, we 
> see that this message appears even before Connect RestServer has finished 
> initialization.
>  {quote}   - [2022-12-15 15:40:29,064] INFO [Worker clientId=connect-1, 
> groupId=connect-cluster] Joined group at generation 2 with protocol version 1 
> and got assignment: Assignment{error=0, 
> leader='connect-1-07d9da63-9acb-4633-aee4-1ab79f4ab1ae', 
> leaderUrl='http://worker34:8083/', offset=-1, connectorIds=[], taskIds=[], 
> revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> - [2022-12-15 15:40:29,560] INFO 172.31.5.66 - - [15/Dec/2022:15:40:29 
> +] "GET /connector-plugins/ HTTP/1.1" 404 375 "-" 
> "python-requests/2.24.0" 71 (org.apache.kafka.connect.runtime.rest.RestServer)
> - [2022-12-15 15:40:29,579] INFO REST resources initialized; server is 
> started and ready to handle requests 
> (org.apache.kafka.connect.runtime.rest.RestServer)
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #57

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 281691 lines...]
> Task :core:integrationTest

Gradle Test Run :core:integrationTest > Gradle Test Executor 182 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 182 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor() PASSED

2025 tests completed, 1 failed, 4 skipped
There were failing tests. See the report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.5/core/build/reports/tests/integrationTest/index.html

> Task :streams:integrationTest

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Execut

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

[Matthias J. Sax]

Thanks. Seems we are on the same page now what the requirement are? 
That's good progress!




This solution was considered when in KIP-213 for the existing
table-table FK join. There is a discussion on disadvantages of using this
approach in the article related to KIP-213 and I think the same
disadvantages will apply to this KIP.


I am not sure. The difference (to me) is, that for KIP-213, if we 
aggregate the right input table, we would need to split the "fat result" 
record to flatten the individual result record we want to have. But for 
your case it seems, you want to have a single "fat" result record in the 
end, so the aggregation is not a workaround but a requirement anyway? If 
we go with KIP-955, your use case requires an aggregation (right?) 
because for each input stream record, you want one result record (not 
multiple?).




I see FK join as the common operation in data manipulation so it would
be nice to have a shortcut for it and not to try to design it from existing
functionality all the time.


Well, yes and no? In the end, a stream-table join is a _enrichment_ 
join, ie, for each left input stream event, we emit one (or none, if it 
does not match for inner join) result record. A stream-FK-table-join 
would imply that we emit multiple result records, what is (at least to 
me) a somewhat weird behavior, because it's kinda "flatMap" as join 
side-effect. (Or we add in an aggregation, and again, have a single 
composed operator with "weird" semantics.) It does not appeal 
semantically clean to me to do it this way.




Consider the real use case I discussed at the
beginning when a business entity has 25 children


Not sure if I fully understand? Are you saying a single stream record 
would join with 25 table rows? And that's why you think you cannot 
aggregate those 25 rows because such a "fat row" would be too large? If 
this is the case, (and I am correct about my understanding that your use 
case needs an aggregation step anyway), than this issue does not go way, 
because you build a single "fat" result record containing all these 25 
rows as final result anyway.




This solution similarly to mine is "mudding the water" by providing a
hybrid outcome join + aggregate. At list with my proposal we could
potentially control it with the flag, or maybe create some special
aggregate that could be chained after (don't know how to do it yet :-))


Why would it mud the waters if you combine multiple operators? If you 
apply an aggregation and a join operator, both operators provide 
well-know and clean semantics? To me, "muddying the waters" means to 
have a single operator that does "too much" at once (and adding a config 
makes it even worse IMHO, as it now actually merged even more things 
into a single operator).


From my POV, a good DSL is a tool set of operators each doing one (well 
defined) thing, and you can combine them to do complex stuff. Building 
"fat uber" operators is the opposite of it IMHO.


I am still on the fence if KIP-955 propose a well-defined operator or 
not, because it seems it's either a flatMap+join or join+aggregation -- 
for both cases, I am wondering why we would want to combine them into a 
single operator?


To me, there are two good argument for adding a new operator:

 (1) It's not possible to combine existing operators to semantically 
express the same at all.


 (2) Adding the operator provides significant performance improvements 
compared to combining a set of existing operators.


Do we think one of both cases apply?


Lets call a stream-fk-joins that emits multiple result records the 
"flatMapJoin" and the stream-fk-join that emit a single "fat" result 
record the "aggregationJoin".


If my understanding is correct, and you need an aggregation anyway, 
adding a flatMapJoin that need an additional aggregation downstream does 
not work anyway, because the aggregation cannot know when to start a new 
aggregation... Assume there is two input event both with orderId1; the 
first joins to two table rows, emitting two flatMapJoin result records, 
and the second joins to three table rows, emitting three flatMapJoin 
record. How would the downstream aggregation know, to put records 1+2 
and record 3+4+5 together to get back to the original two input records?


If flatMapJoin does not work, and we go with aggregationJoin, I would 
still argue that it's the same as doing an upstream table aggregation 
plus a regular stream-table join, and I don't see a big perf difference 
between both operations either. For both cases the table input is 
repartitioned. And we also build a fat-record for both cases. The 
difference is that we store the fat-record in the table for the explicit 
aggregation, but we avoid an expensive range scan... but the record size 
will be there in any case, so I am not sure what we gain by not storing 
the fat record in the table if we cannot get rid of the fat record in 
any case?)





-Matthias



On 8/10/23 12:09 PM, Igor Fomenko wrote:

I d

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2087

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 406314 lines...]
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 2h 39m 10s
237 actionable tasks: 124 executed, 113 up-to-date

Publishing build scan...
https://ge.apache.org/s/qjnm2bxqy3nkg


See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk@2/build/reports/profile/profile-2023-08-10-21-16-20.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 20
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExce

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #159

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 524853 lines...]

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testLogDirEventNotificationsDeletion() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetLogConfigs() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetLogConfigs() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testAclMethods() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testAclMethods() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testConditionalUpdatePath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testConditionalUpdatePath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testDeleteTopicZNode() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testDeleteTopicZNode() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testDeletePath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testDeletePath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetBrokerMethods() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetBrokerMethods() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testJuteMaxBufffer() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testJuteMaxBufffer() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testCreateTokenChangeNotification() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testCreateTokenChangeNotification() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetTopicsAndPartitions() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testGetTopicsAndPartitions() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testRegisterBrokerInfo() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testRegisterBrokerInfo() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testRetryRegisterBrokerInfo() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testRetryRegisterBrokerInfo() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testConsumerOffsetPath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testConsumerOffsetPath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 171 > 
KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck() STARTED

Gradle Test Run :core:integra

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #186

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 420679 lines...]
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_false] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_false] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_false] STARTED

> Task :core:integrationTest

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler 
in thread "delete-temp-file-shutdown-hook"

org.apache.kafka.streams.integration.StreamsUpgradeTestIntegrationTest > 
testVersionProbingUpgrade PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes STARTED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated STARTED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes STARTED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog STARTED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig STARTED

> Task :streams:integrationTest

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_false] PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated STARTED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated PASSED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault STARTED

org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault PASSED

org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor STARTED

org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_true] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_true] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_true] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_true] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_true] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_true] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_true] STARTED

> Task :core:integrationTest

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler 
in thread "delete-temp-file-shutdown-hook"

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_true] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_false] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_false] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_false] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_false] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_false] STARTED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_false] PASSED

org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_false] STARTED

org.apac

What are the biggest issues with Apache Kafka?

[Liam Hodges]

I'm working with a small team of engineers looking to contribute to the
open source tools for Apache Kafka. What is missing in the Kafka community
right now? Are there any problems an open source project could solve for
it's developers? Appreciate all feedback.

Re: What are the biggest issues with Apache Kafka?

[Philip Nee]

Hey Liam,

I think apache jira would be a great place to find lower hanging fruits.
We also have lots of flaky tests to resolve.  What is you familiarity with
Kafka? If you are new, i would suggest looking at the demos and examples in
the repo, and maybe try to improve them.  There is also a lot of work being
done in kraft and tiered storage, however im not familiar with them.

P

On Thu, Aug 10, 2023 at 5:17 PM Liam Hodges 
wrote:

> I'm working with a small team of engineers looking to contribute to the
> open source tools for Apache Kafka. What is missing in the Kafka community
> right now? Are there any problems an open source project could solve for
> it's developers? Appreciate all feedback.
>

[GitHub] [kafka-site] lbradstreet commented on pull request #537: MINOR: Add Lucas Bradstreet to committers

[via GitHub]

lbradstreet commented on PR #537:
URL: https://github.com/apache/kafka-site/pull/537#issuecomment-1674136550

   @jolshan if you or someone else is able to merge this it’d be helpful. This 
PR is what should get me the ability to merge :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-15287) Change NodeApiVersions.create() to contains both apis of zk and kraft broker

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15287.
-
Resolution: Fixed

> Change NodeApiVersions.create() to contains both apis of zk and kraft broker 
> -
>
> Key: KAFKA-15287
> URL: https://issues.apache.org/jira/browse/KAFKA-15287
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Deng Ziming
>Priority: Major
>  Labels: newbee
>
> We are using ApiKeys.zkBrokerApis() when calling NodeApiVersions.create(), 
> this means we only support zk broker apis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: What are the biggest issues with Apache Kafka?

[ziming deng]

Hi Liam,

The Apache Kafka project has several modules, I think you should firstly select 
a module you are interested in. 

For example, we are currently working on KIP-500 related features, which 
includes
1. KIP-856: KRaft Disk Failure Recovery,  
2. KIP-642: Dynamic quorum reassignment,
3. kafka-metadata-shell.sh, 
4. KIP-866: ZooKeeper to KRaft Migration, 
5. KIP-858: Handle JBOD broker disk failure in KRaft
6. Migrtion test cases to support Kraft mode
7. KRaft transactions

We even have the idea of implementing multi raft and using it to replace kakfa 
replica protocal. Apart from KRaft, you can also explore tired storage, kafka 
streams, kafka connect,  group coordinator, transaction coordinator, which are 
also In rapid iteration.

--,
Best,
Ziming


> On Aug 11, 2023, at 08:16, Liam Hodges  
> wrote:
> 
> I'm working with a small team of engineers looking to contribute to the
> open source tools for Apache Kafka. What is missing in the Kafka community
> right now? Are there any problems an open source project could solve for
> it's developers? Appreciate all feedback.

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2088

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2089

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-714: Client metrics and observability

[Andrew Schofield]

Hi Doguscan,
Thanks for your question.

If the target broker is unreachable, the client can send the metrics to another
broker. It can select any of the other brokers for this purpose. What I expect 
in
practice is that it loses connection to the broker it’s being using for metrics,
chooses or establishes a connection to another broker, and then selects that
broker for subsequent metrics pushes.

Thanks,
Andrew

> On 8 Aug 2023, at 08:34, Doğuşcan Namal  wrote:
>
> Thanks for your answers Andrew. I share your pain that it took a while for
> you to get this KIP approved and you want to reduce the scope of it, will
> be happy to help you with the implementation :)
>
> Could you help me walk through what happens if the target broker is
> unreachable? Is the client going to drop these metrics or is it going to
> send it to the other brokers it is connected to? This information is
> crucial to understand the client side impact on leadership failovers.
> Moreover, in case of partial outages, such as only the network between the
> client and the broker is partitioned whereas the network within the cluster
> is healthy, practically there is no other way than the client side metrics
> to identify this problem.
>
> Doguscan
>
> On Fri, 4 Aug 2023 at 15:33, Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
>> Hi Doguscan,
>> Thanks for your comments. I’m glad to hear you’re interested in this KIP.
>>
>> 1) It is preferred that a client sends its metrics to the same broker
>> connection
>> but actually it is able to send them to any broker. As a result, if a
>> broker becomes
>> unhealthy, the client can push its metrics to any other broker. It seems
>> to me that
>> pushing to KRaft controllers instead just has the effect of increasing the
>> load on
>> the controllers, while still having the characteristic that an unhealthy
>> controller
>> would present inconvenience for collecting metrics.
>>
>> 2) When the `PushTelemetryRequest.Terminating` flag is set, the standard
>> request
>> throttling is not disabled. The metrics rate-limiting based on the push
>> interval is
>> not applied in this case for a single request for the combination of
>> client instance ID
>> and subscription ID.
>>
>> (I have corrected the KIP text because it erroneously said “client ID and
>> subscription ID”.
>>
>> 3) While this is a theoretical problem, I’m not keen on adding yet more
>> configurations
>> to the broker or client. The `interval.ms` configuration on the
>> CLIENT_METRICS
>> resource could perhaps have a minimum and maximum value to prevent
>> accidental
>> misconfiguration.
>>
>> 4) One of the reasons that this KIP has taken so long to get to this stage
>> is that
>> it tried to do many things all at once. So, it’s greatly simplified
>> compared with
>> 6 months ago. I can see the value of collecting client configurations for
>> problem
>> determination, but I don’t want to make this KIP more complicated. I think
>> the
>> idea has merit as a separate follow-on KIP. I would be happy to collaborate
>> with you on this.
>>
>> 5) The default is set to 5 minutes to minimise the load on the broker for
>> situations
>> in which the administrator didn’t set an interval on a metrics
>> subscription. To
>> use an interval of 1 minute, it is only necessary to set `interval.ms` on
>> the metrics
>> subscription to 6ms.
>>
>> 6) Uncompressed data is always supported. The KIP says:
>> "The CompressionType of NONE will not be
>> "present in the response from the broker, though the broker does support
>> uncompressed
>> "client telemetry if none of the accepted compression codecs are supported
>> by the client.”
>> So in your example, the client need only use CompressionType=NONE.
>>
>> Thanks,
>> Andrew
>>
>>> On 4 Aug 2023, at 14:04, Doğuşcan Namal 
>> wrote:
>>>
>>> Hi Andrew, thanks a lot for this KIP. I was thinking of something similar
>>> so thanks for writing this down 😊
>>>
>>>
>>>
>>> Couple of questions related to the design:
>>>
>>>
>>>
>>> 1. Can we investigate the option for using the Kraft controllers instead
>> of
>>> the brokers for sending metrics? The disadvantage of sending these
>> metrics
>>> directly to the brokers tightly couples metric observability to data
>> plane
>>> availability. If the broker is unhealthy then the root cause of an
>> incident
>>> is clear however on partial failures it makes it hard to debug these
>>> incidents from the brokers perspective.
>>>
>>>
>>>
>>> 2. Ratelimiting will be disable if the `PushTelemetryRequest.Terminating`
>>> flag is set. However, this may cause unavailability on the broker if too
>>> many clients are terminated at once, especially network threads could
>>> become busy and introduce latency on the produce/consume on other
>>> non-terminating clients connections. I think there is a room for
>>> improvement here. If the client is gracefully shutting down, it could
>> wait
>>> for the request to be handled if it is being ratelimited, it doesn't need
>>> to

[jira] [Created] (KAFKA-15334) DescribeQuorum should not be seen as cluster action

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15334:
---

 Summary: DescribeQuorum should not be seen as cluster action 
 Key: KAFKA-15334
 URL: https://issues.apache.org/jira/browse/KAFKA-15334
 Project: Kafka
  Issue Type: Bug
Reporter: Deng Ziming






--
This message was sent by Atlassian Jira
(v8.20.10#820010)