Re: [VOTE] KIP-837 Allow MultiCasting a Result Record.

[Bruno Cadonna]

Hi Sagar,

I would not drop the support for dropping records. I would also not 
return null from partitions(). Maybe an Optional can help here. An empty 
Optional would mean to use the default partitioning behavior of the 
producer. So we would have:


- non-empty Optional, non-empty list of integers: partitions to send the 
record to

- non-empty Optional, empty list of integers: drop the record
- empty Optional: use default behavior

What do other think?

Best,
Bruno

On 02.09.22 13:53, Sagar wrote:

Hello Bruno/Chris,

Since these are the last set of changes(I am assuming haha), it would be
great if you could review the 2 options from above so that we can close the
voting. Of course I am happy to incorporate any other requisite changes.

Thanks!
Sagar.

On Wed, Aug 31, 2022 at 10:07 PM Sagar  wrote:


Thanks Bruno for the great points.

I see 2 options here =>

1) As Chris suggested, drop the support for dropping records in the
partitioner. That way, an empty list could signify the usage of a default
partitioner. Also, if the deprecated partition() method returns null
thereby signifying the default partitioner, the partitions() can return an
empty list i.e default partitioner.

2) OR we treat a null return type of partitions() method to signify the
usage of the default partitioner. In the default implementation of
partitions() method, if partition() returns null, then even partitions()
can return null(instead of an empty list). The RecordCollectorImpl code can
also be modified accordingly. @Chris, to your point, we can even drop the
support of dropping of records. It came up during KIP discussion, and I
thought it might be a useful feature. Let me know what you think.

3) Lastly about the partition number check. I wanted to avoid the throwing
of exception so I thought adding it might be a useful feature. But as you
pointed out, if it can break backwards compatibility, it's better to remove
it.

Thanks!
Sagar.


On Tue, Aug 30, 2022 at 6:32 PM Chris Egerton 
wrote:


+1 to Bruno's concerns about backward compatibility. Do we actually need
support for dropping records in the partitioner? It doesn't seem necessary
based on the motivation for the KIP. If we remove that feature, we could
handle null and/or empty lists by using the default partitioning,
equivalent to how we handle null return values from the existing partition
method today.

On Tue, Aug 30, 2022 at 8:55 AM Bruno Cadonna  wrote:


Hi Sagar,

Thank you for the updates!

I do not intend to prolong this vote thread more than needed, but I
still have some points.

The deprecated partition method can return null if the default
partitioning logic of the producer should be used.
With the new method partitions() it seems that it is not possible to use
the default partitioning logic, anymore.

Also, in the default implementation of method partitions(), a record
that would use the default partitioning logic in method partition()
would be dropped, which would break backward compatibility since Streams
would always call the new method partitions() even though the users
still implement the deprecated method partition().

I have a last point that we should probably discuss on the PR and not on
the KIP but since you added the code in the KIP I need to mention it. I
do not think you should check the validity of the partition number since
the ProducerRecord does the same check and throws an exception. If
Streams adds the same check but does not throw, the behavior is not
backward compatible.

Best,
Bruno


On 30.08.22 12:43, Sagar wrote:

Thanks Bruno/Chris,

Even I agree that might be better to keep it simple like the way Chris
suggested. I have updated the KIP accordingly. I made couple of minor
changes to the KIP:

1) One of them being the change of return type of partitions method

from

List to Set. This is to ensure that in case the implementation of
StreamPartitoner is buggy and ends up returning duplicate
partition numbers, we won't have duplicates thereby not trying to

send to

the same partition multiple times due to this.
2) I also added a check to send the record only to valid partition

numbers

and log and drop when the partition number is invalid. This is again

to

prevent errors for cases when the StreamPartitioner implementation has

some

bugs (since there are no validations as such).
3) I also updated the Test Plan section based on the suggestion from

Bruno.

4) I updated the default implementation of partitions method based on

the

great catch from Chris!

Let me know if it looks fine now.

Thanks!
Sagar.


On Tue, Aug 30, 2022 at 3:00 PM Bruno Cadonna 

wrote:



Hi,

I am favour of discarding the sugar for broadcasting and leave the
broadcasting to the implementation as Chris suggests. I think that is
the cleanest option.

Best,
Bruno

On 29.08.22 19:50, Chris Egerton wrote:

Hi all,

I think it'd be useful to be more explicit about broadcasting to all

topic

partitions rather than add implicit behavior for empty cases (empty
option

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1202

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 503674 lines...]
[2022-09-07T08:47:11.550Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2022-09-07T08:47:11.550Z] 
[2022-09-07T08:47:11.550Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2022-09-07T08:47:16.397Z] 
[2022-09-07T08:47:16.397Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2022-09-07T08:47:16.397Z] 
[2022-09-07T08:47:16.397Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2022-09-07T08:47:17.432Z] 
[2022-09-07T08:47:17.432Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2022-09-07T08:47:17.432Z] 
[2022-09-07T08:47:17.432Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2022-09-07T08:47:17.432Z] 
[2022-09-07T08:47:17.432Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2022-09-07T08:47:17.432Z] 
[2022-09-07T08:47:17.432Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2022-09-07T08:47:46.043Z] 
[2022-09-07T08:47:46.043Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-07T08:47:46.043Z] 
[2022-09-07T08:47:46.043Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-07T08:48:09.144Z] 
[2022-09-07T08:48:09.144Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-07T08:48:09.144Z] 
[2022-09-07T08:48:09.144Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-07T08:48:13.743Z] 
[2022-09-07T08:48:13.743Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-07T08:48:13.743Z] 
[2022-09-07T08:48:13.743Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-07T08:48:35.648Z] 
[2022-09-07T08:48:35.648Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-07T08:48:35.648Z] 
[2022-09-07T08:48:35.648Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-07T08:48:35.648Z] 
[2022-09-07T08:48:35.648Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-07T08:48:35.648Z] 
[2022-09-07T08:48:35.648Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-07T08:48:36.652Z] 
[2022-09-07T08:48:36.652Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-07T08:48:37.841Z] 
[2022-09-07T08:48:37.841Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-09-07T08:48:42.233Z] 
[2022-09-07T08:48:42.233Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-09-07T08:48:42.233Z] 
[2022-09-07T08:48:42.233Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() STARTED
[2022-09-07T08:48:49.367Z] 
[2022-09-07T08:48:49.367Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() PASSED
[2022-09-07T08:48:49.367Z] 
[2022-09-07T08:48:49.367Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() STARTED
[2022-09-07T08:48:59.637Z] 
[2022-09-07T08:48:59.637Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() PASSED
[2022-09-07T08:48:59.637Z] 
[2022-09-07T08:48:59.637Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-09-07T08:49:03.923Z] 
[2022-09-07T08:49:03.923Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-09-07T08:49:03.923Z] 
[2022-09-07T08:49:03.923Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-09-07T08:49:28.798Z] 
[2022-09-07T08:49:28.798Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-09-07T08:49:28.798Z] 
[2022-09-07T08:49:28.798Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() STARTED
[2022-09-07T08:49:35.255Z] 
[2022-09-07T08:49:35.255Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() PASSED
[2022-09-07T08:49:35.255Z] 
[2022-09-07T08:49:35.255Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-09-07T08:49:41.739Z] 
[2022-09-07T08:49:41.739Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-09-07T08:49:41.739Z] 
[2022-09-07T08:49:41.739Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-09-07T08:49:49.388Z] 
[2022-09-07T08:49:49.388Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-09-07T08:49:49.388Z] 
[2022-09-07T08:49:49.388Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut()

Re: [VOTE] KIP-844: Transactional State Stores

[Alexander Sorokoumov]

The vote passes with three binding +1s (Bruno, Guozhang, John), one
non-binding +1 (Colt), and no -1s.

Thanks everyone!

Best,
Alex

On Thu, Sep 1, 2022 at 11:12 PM John Roesler  wrote:

> Thanks for the KIP, Alex!
>
> +1 (binding) from me.
>
> -John
>
> On Thu, Sep 1, 2022, at 09:51, Guozhang Wang wrote:
> > +1, thanks Alex!
> >
> > On Thu, Sep 1, 2022 at 6:33 AM Bruno Cadonna  wrote:
> >
> >> Thanks for the KIP!
> >>
> >> +1 (binding)
> >>
> >> Best,
> >> Bruno
> >>
> >> On 01.09.22 15:26, Colt McNealy wrote:
> >> > +1
> >> >
> >> > Hi Alex,
> >> >
> >> > Thank you for your work on the KIP. I'm not a committer so my vote is
> >> > non-binding but I strongly support this improvement.
> >> >
> >> > Thank you,
> >> > Colt McNealy
> >> > *Founder, LittleHorse.io*
> >> >
> >> >
> >> > On Thu, Sep 1, 2022 at 8:20 AM Alexander Sorokoumov
> >> >  wrote:
> >> >
> >> >> Hi All,
> >> >>
> >> >> I would like to start a voting thread on KIP-844, which introduces
> >> >> transactional state stores to avoid wiping local state on crash
> failure
> >> >> under EOS.
> >> >>
> >> >> KIP:
> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-844%3A+Transactional+State+Stores
> >> >> Discussion thread:
> >> >> https://lists.apache.org/thread/4vc18t0o2wsk0n235dd4pd1hlr1p6gm2
> >> >> Jira: https://issues.apache.org/jira/browse/KAFKA-12549
> >> >>
> >> >> Best,
> >> >> Alex
> >> >>
> >> >
> >>
> >
> >
> > --
> > -- Guozhang
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #131

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 368958 lines...]
[2022-09-07T11:13:26.011Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-09-07T11:13:26.011Z] > Task :connect:api:javadocJar
[2022-09-07T11:13:26.011Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-09-07T11:13:26.011Z] > Task :connect:api:processTestResources NO-SOURCE
[2022-09-07T11:13:26.011Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-09-07T11:13:26.011Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-09-07T11:13:26.011Z] > Task :connect:json:publishToMavenLocal
[2022-09-07T11:13:26.955Z] > Task :connect:api:testJar
[2022-09-07T11:13:26.955Z] > Task :connect:api:testSrcJar
[2022-09-07T11:13:26.955Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-09-07T11:13:26.955Z] > Task :connect:api:publishToMavenLocal
[2022-09-07T11:13:27.370Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2022-09-07T11:13:27.371Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-09-07T11:13:30.206Z] > Task :streams:javadoc
[2022-09-07T11:13:30.206Z] > Task :streams:javadocJar
[2022-09-07T11:13:35.557Z] 
[2022-09-07T11:13:35.557Z] > Task :clients:javadoc
[2022-09-07T11:13:35.557Z] 
/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-09-07T11:13:36.592Z] 1 warning
[2022-09-07T11:13:37.803Z] 
[2022-09-07T11:13:37.803Z] > Task :clients:javadocJar
[2022-09-07T11:13:39.936Z] 
[2022-09-07T11:13:39.936Z] > Task :clients:srcJar
[2022-09-07T11:13:39.936Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-09-07T11:13:39.936Z]   - Gradle detected a problem with the following 
location: '/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/generated/java'. 
Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-09-07T11:13:50.002Z] 
[2022-09-07T11:13:50.002Z] > Task :core:compileScala
[2022-09-07T11:13:51.038Z] > Task :clients:testJar
[2022-09-07T11:13:56.741Z] > Task :clients:testSrcJar
[2022-09-07T11:13:56.741Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2022-09-07T11:13:56.741Z] > Task :clients:publishToMavenLocal
[2022-09-07T11:15:09.299Z] > Task :core:classes
[2022-09-07T11:15:09.299Z] > Task :core:compileTestJava NO-SOURCE
[2022-09-07T11:15:38.839Z] > Task :core:compileTestScala
[2022-09-07T11:16:32.538Z] > Task :core:testClasses
[2022-09-07T11:16:45.064Z] > Task :streams:compileTestJava
[2022-09-07T11:16:45.064Z] > Task :streams:testClasses
[2022-09-07T11:16:50.885Z] > Task :streams:testJar
[2022-09-07T11:16:59.867Z] > Task :streams:testSrcJar
[2022-09-07T11:16:59.867Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-09-07T11:16:59.867Z] > Task :streams:publishToMavenLocal
[2022-09-07T11:16:59.867Z] 
[2022-09-07T11:16:59.867Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-09-07T11:16:59.867Z] 
[2022-09-07T11:16:59.867Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-09-07T11:16:59.867Z] 
[2022-09-07T11:16:59.867Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2022-09-07T11:16:59.867Z] 
[2022-09-07T11:16:59.867Z] Execution optimizations have been disabled for 3 
invalid unit(s) of work during this build to ensure correctness.
[2022-09-07T11:16:59.867Z] Please consult deprecation warnings for more details.
[2022-09-07T11:16:59.867Z] 
[2022-09-07T11:16:59.867Z] BUILD SUCCESSFUL in 3m 57s
[2022-09-07T11:16:59.867Z] 77 actionable tasks: 38 executed, 39 up-to-date
[Pipeline] sh
[2022-09-07T11:17:03.536Z] + grep ^version= gradle.properties
[2022-09-07T11:17:03.536Z] + cut -d= -f 2
[Pipeline] dir
[2022-09-07T11:17:04.414Z] Running in 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-09-07T11:17:07.269Z] + mvn clean install -Dgpg.skip
[2022-09-07T11:17:08.439Z] [INFO] Scanning for projects...
[2022-09-07T11:17:08.439Z] [INFO] 

[2

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #77

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 458042 lines...]
[2022-09-07T11:42:35.200Z] Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-09-07T11:42:36.206Z] 
[2022-09-07T11:42:36.206Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-07T11:42:36.206Z] 
[2022-09-07T11:42:36.206Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-07T11:43:02.265Z] 
[2022-09-07T11:43:02.265Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-07T11:43:02.265Z] 
[2022-09-07T11:43:02.265Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-07T11:43:06.600Z] 
[2022-09-07T11:43:06.600Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-07T11:43:06.600Z] 
[2022-09-07T11:43:06.600Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-07T11:43:28.387Z] 
[2022-09-07T11:43:28.387Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-07T11:43:28.387Z] 
[2022-09-07T11:43:28.387Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-07T11:43:28.387Z] 
[2022-09-07T11:43:28.387Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-07T11:43:28.387Z] 
[2022-09-07T11:43:28.387Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-07T11:43:29.386Z] 
[2022-09-07T11:43:29.386Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-07T11:43:33.552Z] 
[2022-09-07T11:43:33.552Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-09-07T11:43:33.552Z] 
[2022-09-07T11:43:33.552Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-09-07T11:43:33.552Z] 
[2022-09-07T11:43:33.552Z] See 
https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings
[2022-09-07T11:43:33.552Z] 
[2022-09-07T11:43:33.552Z] BUILD SUCCESSFUL in 2h 35m 11s
[2022-09-07T11:43:33.552Z] 208 actionable tasks: 113 executed, 95 up-to-date
[2022-09-07T11:43:33.552Z] 
[2022-09-07T11:43:33.552Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.2/build/reports/profile/profile-2022-09-07-09-08-25.html
[2022-09-07T11:43:33.552Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2022-09-07T11:43:34.551Z] Recording test results
[2022-09-07T11:43:44.389Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2022-09-07T11:43:44.391Z] Verify that Kafka Streams archetype compiles
[Pipeline] sh
[2022-09-07T11:43:47.228Z] + ./gradlew streams:publishToMavenLocal 
clients:publishToMavenLocal connect:json:publishToMavenLocal 
connect:api:publishToMavenLocal
[2022-09-07T11:43:47.228Z] To honour the JVM settings for this build a 
single-use Daemon process will be forked. See 
https://docs.gradle.org/7.3.3/userguide/gradle_daemon.html#sec:disabling_the_daemon.
[2022-09-07T11:43:49.226Z] Daemon will be stopped at the end of the build 
[2022-09-07T11:43:55.969Z] 
[2022-09-07T11:43:55.969Z] > Configure project :
[2022-09-07T11:43:55.969Z] Starting build with version 3.2.2 (commit id 
d7398e61) using Gradle 7.3.3, Java 1.8 and Scala 2.13.8
[2022-09-07T11:43:55.969Z] Build properties: maxParallelForks=24, 
maxScalacThreads=8, maxTestRetries=0
[2022-09-07T11:44:02.047Z] 
[2022-09-07T11:44:02.047Z] > Task :connect:json:processResources NO-SOURCE
[2022-09-07T11:44:02.047Z] > Task :connect:api:processResources NO-SOURCE
[2022-09-07T11:44:02.047Z] > Task :streams:processResources UP-TO-DATE
[2022-09-07T11:44:02.047Z] > Task :core:processResources UP-TO-DATE
[2022-09-07T11:44:02.047Z] > Task :raft:processResources UP-TO-DATE
[2022-09-07T11:44:02.047Z] > Task :metadata:processResources UP-TO-DATE
[2022-09-07T11:44:02.047Z] > Task :raft:processTestResources UP-TO-DATE
[2022-09-07T11:44:02.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1203

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 502802 lines...]
[2022-09-07T11:36:50.828Z] KStreamAggregationIntegrationTest > 
shouldReduce(TestInfo) STARTED
[2022-09-07T11:36:57.626Z] 
[2022-09-07T11:36:57.626Z] KStreamAggregationIntegrationTest > 
shouldReduce(TestInfo) PASSED
[2022-09-07T11:36:57.626Z] 
[2022-09-07T11:36:57.626Z] KStreamAggregationIntegrationTest > 
shouldAggregate(TestInfo) STARTED
[2022-09-07T11:37:04.690Z] 
[2022-09-07T11:37:04.690Z] KStreamAggregationIntegrationTest > 
shouldAggregate(TestInfo) PASSED
[2022-09-07T11:37:04.690Z] 
[2022-09-07T11:37:04.690Z] KStreamAggregationIntegrationTest > 
shouldCount(TestInfo) STARTED
[2022-09-07T11:37:07.497Z] 
[2022-09-07T11:37:07.497Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithDefaultGlobalAutoOffsetResetEarliest()
 PASSED
[2022-09-07T11:37:07.497Z] 
[2022-09-07T11:37:07.497Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowStreamsExceptionNoResetSpecified() STARTED
[2022-09-07T11:37:07.497Z] 
[2022-09-07T11:37:07.497Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowStreamsExceptionNoResetSpecified() PASSED
[2022-09-07T11:37:10.130Z] 
[2022-09-07T11:37:10.130Z] GlobalKTableIntegrationTest > 
shouldGetToRunningWithOnlyGlobalTopology() STARTED
[2022-09-07T11:37:10.130Z] 
[2022-09-07T11:37:10.130Z] GlobalKTableIntegrationTest > 
shouldGetToRunningWithOnlyGlobalTopology() PASSED
[2022-09-07T11:37:10.130Z] 
[2022-09-07T11:37:10.130Z] GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin() STARTED
[2022-09-07T11:37:11.321Z] 
[2022-09-07T11:37:11.321Z] KStreamAggregationIntegrationTest > 
shouldCount(TestInfo) PASSED
[2022-09-07T11:37:11.321Z] 
[2022-09-07T11:37:11.321Z] KStreamAggregationIntegrationTest > 
shouldGroupByKey(TestInfo) STARTED
[2022-09-07T11:37:12.776Z] 
[2022-09-07T11:37:12.776Z] GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin() PASSED
[2022-09-07T11:37:12.776Z] 
[2022-09-07T11:37:12.776Z] GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart() STARTED
[2022-09-07T11:37:14.717Z] 
[2022-09-07T11:37:14.717Z] GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart() PASSED
[2022-09-07T11:37:14.717Z] 
[2022-09-07T11:37:14.717Z] GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin() STARTED
[2022-09-07T11:37:16.486Z] 
[2022-09-07T11:37:16.486Z] GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin() PASSED
[2022-09-07T11:37:17.432Z] 
[2022-09-07T11:37:17.432Z] GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() STARTED
[2022-09-07T11:37:17.953Z] 
[2022-09-07T11:37:17.954Z] KStreamAggregationIntegrationTest > 
shouldGroupByKey(TestInfo) PASSED
[2022-09-07T11:37:17.954Z] 
[2022-09-07T11:37:17.954Z] KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore(TestInfo) STARTED
[2022-09-07T11:37:23.384Z] 
[2022-09-07T11:37:23.384Z] KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore(TestInfo) PASSED
[2022-09-07T11:37:23.384Z] 
[2022-09-07T11:37:23.384Z] KStreamAggregationIntegrationTest > 
shouldCountUnlimitedWindows() STARTED
[2022-09-07T11:37:23.401Z] 
[2022-09-07T11:37:23.401Z] GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() PASSED
[2022-09-07T11:37:24.346Z] 
[2022-09-07T11:37:24.346Z] IQv2IntegrationTest > shouldFailStopped() STARTED
[2022-09-07T11:37:24.346Z] 
[2022-09-07T11:37:24.346Z] IQv2IntegrationTest > shouldFailStopped() PASSED
[2022-09-07T11:37:24.346Z] 
[2022-09-07T11:37:24.346Z] IQv2IntegrationTest > 
shouldNotRequireQueryHandler(TestInfo) STARTED
[2022-09-07T11:37:25.293Z] 
[2022-09-07T11:37:25.293Z] IQv2IntegrationTest > 
shouldNotRequireQueryHandler(TestInfo) PASSED
[2022-09-07T11:37:25.293Z] 
[2022-09-07T11:37:25.293Z] IQv2IntegrationTest > shouldFailNotStarted() STARTED
[2022-09-07T11:37:25.293Z] 
[2022-09-07T11:37:25.293Z] IQv2IntegrationTest > shouldFailNotStarted() PASSED
[2022-09-07T11:37:25.293Z] 
[2022-09-07T11:37:25.293Z] IQv2IntegrationTest > shouldFetchFromPartition() 
STARTED
[2022-09-07T11:37:27.063Z] 
[2022-09-07T11:37:27.063Z] IQv2IntegrationTest > shouldFetchFromPartition() 
PASSED
[2022-09-07T11:37:27.063Z] 
[2022-09-07T11:37:27.063Z] IQv2IntegrationTest > 
shouldFetchExplicitlyFromAllPartitions() STARTED
[2022-09-07T11:37:27.434Z] 
[2022-09-07T11:37:27.434Z] KStreamAggregationIntegrationTest > 
shouldCountUnlimitedWindows() PASSED
[2022-09-07T11:37:27.434Z] 
[2022-09-07T11:37:27.434Z] KStreamAggregationIntegrationTest > 
shouldReduceWindowed(TestInfo) STARTED
[2022-09-07T11:37:28.009Z] 
[2022-09-07T11:37:28.009Z] IQv2IntegrationTest > 
shouldFetchExplicitlyFromAllPartitions() PASSED
[2022-09-07T11:37:28.009Z] 
[2022-09-07T11:37:28.009Z] IQv2IntegrationTest > shouldFailUnknownStore() 
STARTED
[2022-09-07T11:37:28.954Z] 
[2022-09-07T11:37:28.954Z] IQv2IntegrationTest > shouldFai

[jira] [Created] (KAFKA-14206) Upgrade zookeeper to 3.7.1 to address security vulnerabilities

[Valeriy Kassenbayev (Jira)]

Valeriy Kassenbayev created KAFKA-14206:
---

 Summary: Upgrade zookeeper to 3.7.1 to address security 
vulnerabilities
 Key: KAFKA-14206
 URL: https://issues.apache.org/jira/browse/KAFKA-14206
 Project: Kafka
  Issue Type: Improvement
  Components: packaging
Affects Versions: 3.2.1
Reporter: Valeriy Kassenbayev


Kafka 3.2.1 is using ZooKeeper, which is affected by 
[CVE-2021-37136|https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584064] and 
[CVE-2021-37137:|https://www.cve.org/CVERecord?id=CVE-2021-37137]
{code:java}
  ✗ Denial of Service (DoS) [High 
Severity][https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584063] in 
io.netty:netty-codec@4.1.63.Final
    introduced by org.apache.kafka:kafka_2.13@3.2.1 > 
org.apache.zookeeper:zookeeper@3.6.3 > io.netty:netty-handler@4.1.63.Final > 
io.netty:netty-codec@4.1.63.Final
  This issue was fixed in versions: 4.1.68.Final
  ✗ Denial of Service (DoS) [High 
Severity][https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-1584064] in 
io.netty:netty-codec@4.1.63.Final
    introduced by org.apache.kafka:kafka_2.13@3.2.1 > 
org.apache.zookeeper:zookeeper@3.6.3 > io.netty:netty-handler@4.1.63.Final > 
io.netty:netty-codec@4.1.63.Final
  This issue was fixed in versions: 4.1.68.Final {code}
The issues were fixed in the next versions of ZooKeeper (starting from 3.6.4). 
ZooKeeper 3.7.1 is the next stable 
[release|https://zookeeper.apache.org/releases.html] at the moment.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

[David Jacot]

+1 from me. Thanks, Stan!

On Tue, Aug 23, 2022 at 12:10 PM Luke Chen  wrote:
>
> Hi Stanislav,
>
> Thanks for the KIP.
> The solution looks reasonable to me.
> +1 from me.
>
> Thank you.
> Luke
>
> On Tue, Aug 23, 2022 at 6:07 AM Stanislav Kozlovski
>  wrote:
>
> > Hello,
> >
> > I'd like to start a vote on KIP-860, which adds a client-provided option to
> > the AlterPartitionReassignmentsRequest that allows the user to guard
> > against an unintentional change in the replication factor during partition
> > reassignments.
> >
> > Discuss Thread:
> > https://lists.apache.org/thread/bhrqjd4vb05xtztkdo8py374m9dgq69r
> > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
> >
> >
> > --
> > Best,
> > Stanislav
> >

Re: Hosting Kafka Videos on ASF YouTube channel

[Joe Brockmeier]

Hi Bill, all,

Next steps would be to get the videos ready for posting + we can start
uploading them. I'd need them in a central location I can do that
from.

On Wed, Aug 31, 2022 at 2:45 PM Bill Bejeck  wrote:
>
> This thread has been open for 22 days, so I will close the vote now.
>
> The question of hosting the four Kafka Streams videos passes:
>
> +1 votes
> PMC Members:
> * Mickael Maison
> * John Roesler
> * Bill Bejeck
>
> Vote thread:
> https://www.mail-archive.com/dev@kafka.apache.org/msg126019.html
>
> Joe,
> Provided this vote is sufficient, what are the next steps?
>
> Thanks,
> Bill
>
> On Thu, Aug 25, 2022 at 12:48 PM John Roesler  wrote:
>>
>> Thanks all,
>>
>> I’m also +1 on the Kafka Streams videos.
>>
>> Thanks,
>> John
>>
>> On Tue, Aug 9, 2022, at 03:54, Mickael Maison wrote:
>> > Hi,
>> >
>> > I checked the four Streams videos
>> > (https://kafka.apache.org/32/documentation/streams/), they are good
>> > and don't mention any vendors.
>> > +1 (binding) for these four videos
>> >
>> > For the last video (https://kafka.apache.org/intro and
>> > https://kafka.apache.org/quickstart) we will have to wait till the
>> > intro is edited.
>> >
>> > Thanks,
>> > Mickael
>> >
>> >
>> > On Mon, Aug 8, 2022 at 11:12 PM Joe Brockmeier  wrote:
>> >>
>> >> Repurpose away. Thanks!
>> >>
>> >> On Mon, Aug 8, 2022 at 4:55 PM Bill Bejeck  wrote:
>> >> >
>> >> > Hi Joe,
>> >> >
>> >> > Thanks that works for me. As for you watching the videos, they are 
>> >> > about 10 minutes each, and you can watch them at 1.5 - 1.75 playback 
>> >> > speed.
>> >> >
>> >> > If it's ok with you, I'm going to repurpose this thread as a voting 
>> >> > thread for the videos.
>> >> >
>> >> > I watched the Kafka Streams videos on 
>> >> > https://kafka.apache.org/32/documentation/streams/, and I can confirm 
>> >> > they are vendor-neutral.
>> >> > The other videos and logo that show up at the end are coming from 
>> >> > YouTube, so once move the videos to the ASF channel, that should go 
>> >> > away.
>> >> >
>> >> > +1(binding).
>> >> >
>> >> > Thanks,
>> >> > Bill
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Aug 8, 2022 at 9:46 AM Joe Brockmeier  wrote:
>> >> >>
>> >> >> If we can get a +1 from the PMC on each video that they're happy that
>> >> >> the videos are vendor neutral I think we can do that. I'll also need
>> >> >> to view them as well. I hope they're not long videos. :-)
>> >> >>
>> >> >> On Tue, Aug 2, 2022 at 3:38 PM Bill Bejeck  wrote:
>> >> >> >
>> >> >> > Hi Joe,
>> >> >> >
>> >> >> > Yes, that is correct.  Sorry, I should have mentioned that in the 
>> >> >> > original email.  That is the only video where Tim says that.
>> >> >> > The Kafka Streams videos do not mention Confluent.
>> >> >> >
>> >> >> > We're currently pursuing editing the video to remove the "from 
>> >> >> > Confluent" part.
>> >> >> > Note that the site also uses the same video on the "quickstart" 
>> >> >> > page, so both places will be fixed when editing is completed.
>> >> >> >
>> >> >> > Can we pursue hosting the Kafka Streams videos for now, then revisit 
>> >> >> > the "What is Apache Kafka?" when the editing is done?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Bill
>> >> >> >
>> >> >> >
>> >> >> > On Tue, Aug 2, 2022 at 3:12 PM Joe Brockmeier  
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi Bill,
>> >> >> >>
>> >> >> >> I'm not sure changing hosting would quite solve the problem. The 
>> >> >> >> first
>> >> >> >> video I see on this page:
>> >> >> >>
>> >> >> >> https://kafka.apache.org/intro
>> >> >> >>
>> >> >> >> Starts with "Hi, I'm Bill Berglund from *Confluent*" rather than 
>> >> >> >> "Hi,
>> >> >> >> I'm Bill from Apache Kafka" -- so moving to the ASF Youtube channel
>> >> >> >> wouldn't completely solve the problem.
>> >> >> >>
>> >> >> >> On Tue, Aug 2, 2022 at 3:05 PM Bill Bejeck  
>> >> >> >> wrote:
>> >> >> >> >
>> >> >> >> > Hi,
>> >> >> >> >
>> >> >> >> > I am an Apache Kafka® committer and PMC member, and I'm working 
>> >> >> >> > on our site to address some issues around our embedded videos and 
>> >> >> >> > branding.
>> >> >> >> >
>> >> >> >> > The Kafka site has six embedded videos:  
>> >> >> >> > https://kafka.apache.org/intro, 
>> >> >> >> > https://kafka.apache.org/quickstart, and four videos on 
>> >> >> >> > https://kafka.apache.org/32/documentation/streams/.
>> >> >> >> >
>> >> >> >> > The videos are hosted on the Confluent YouTube channel, so the 
>> >> >> >> > branding on the video is from Confluent.  Since it's coming from 
>> >> >> >> > YouTube, there's no way to change it.
>> >> >> >> >
>> >> >> >> > Would it be possible to upload these videos to the Apache 
>> >> >> >> > Foundation YouTube channel 
>> >> >> >> > (https://www.youtube.com/c/TheApacheFoundation/featured)?  Doing 
>> >> >> >> > this would automatically change the branding to Apache.
>> >> >> >> >
>> >> >> >> > Thanks, and I look forward to working with you on this matter.
>> >> >> >> >
>> >> >> >> > Bill Bejeck
>> >> >> >>
>> >> >> >>
>> >> >

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

[Ismael Juma]

Thanks for the KIP. Can we explain a bit more why this is an important use
case to address? For example, do we have concrete examples of people
running into this? The way the KIP is written, it sounds like a potential
problem but no information is given on whether it's a real problem in
practice.

Ismael

On Thu, Jul 28, 2022 at 2:00 AM Stanislav Kozlovski
 wrote:

> Hey all,
>
> I'd like to start a discussion on a proposal to help API users from
> inadvertently increasing the replication factor of a topic through
> the alter partition reassignments API. The KIP describes two fairly
> easy-to-hit race conditions in which this can happen.
>
> The KIP itself is pretty simple, yet has a couple of alternatives that can
> help solve the same problem. I would appreciate thoughts from the community
> on how you think we should proceed, and whether the proposal makes sense in
> the first place.
>
> Thanks!
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
>
> --
> Best,
> Stanislav
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #132

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 381772 lines...]
[2022-09-07T15:07:25.626Z] PlaintextConsumerTest > testAsyncCommit() PASSED
[2022-09-07T15:07:25.626Z] 
[2022-09-07T15:07:25.626Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() STARTED
[2022-09-07T15:08:38.402Z] 
[2022-09-07T15:08:38.402Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() PASSED
[2022-09-07T15:08:38.402Z] 
[2022-09-07T15:08:38.402Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() STARTED
[2022-09-07T15:09:04.932Z] 
[2022-09-07T15:09:04.932Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() PASSED
[2022-09-07T15:09:04.932Z] 
[2022-09-07T15:09:04.932Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() STARTED
[2022-09-07T15:09:14.101Z] 
[2022-09-07T15:09:14.101Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() PASSED
[2022-09-07T15:09:14.101Z] 
[2022-09-07T15:09:14.101Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() STARTED
[2022-09-07T15:09:22.422Z] 
[2022-09-07T15:09:22.422Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() PASSED
[2022-09-07T15:09:22.422Z] 
[2022-09-07T15:09:22.422Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() STARTED
[2022-09-07T15:09:27.764Z] 
[2022-09-07T15:09:27.764Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() PASSED
[2022-09-07T15:09:27.764Z] 
[2022-09-07T15:09:27.764Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() STARTED
[2022-09-07T15:09:36.559Z] 
[2022-09-07T15:09:36.559Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() PASSED
[2022-09-07T15:09:36.559Z] 
[2022-09-07T15:09:36.559Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() STARTED
[2022-09-07T15:09:43.600Z] 
[2022-09-07T15:09:43.600Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() PASSED
[2022-09-07T15:09:43.600Z] 
[2022-09-07T15:09:43.600Z] PlaintextConsumerTest > testSeek() STARTED
[2022-09-07T15:09:53.598Z] 
[2022-09-07T15:09:53.598Z] PlaintextConsumerTest > testSeek() PASSED
[2022-09-07T15:09:53.598Z] 
[2022-09-07T15:09:53.598Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() STARTED
[2022-09-07T15:10:11.614Z] 
[2022-09-07T15:10:11.614Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() PASSED
[2022-09-07T15:10:11.614Z] 
[2022-09-07T15:10:11.614Z] PlaintextConsumerTest > testPositionAndCommit() 
STARTED
[2022-09-07T15:10:17.426Z] 
[2022-09-07T15:10:17.426Z] PlaintextConsumerTest > testPositionAndCommit() 
PASSED
[2022-09-07T15:10:17.426Z] 
[2022-09-07T15:10:17.426Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() STARTED
[2022-09-07T15:10:24.380Z] 
[2022-09-07T15:10:24.380Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() PASSED
[2022-09-07T15:10:24.380Z] 
[2022-09-07T15:10:24.380Z] PlaintextConsumerTest > testUnsubscribeTopic() 
STARTED
[2022-09-07T15:10:34.261Z] 
[2022-09-07T15:10:34.261Z] PlaintextConsumerTest > testUnsubscribeTopic() PASSED
[2022-09-07T15:10:34.261Z] 
[2022-09-07T15:10:34.261Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() STARTED
[2022-09-07T15:10:56.237Z] 
[2022-09-07T15:10:56.237Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() PASSED
[2022-09-07T15:10:56.237Z] 
[2022-09-07T15:10:56.237Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() STARTED
[2022-09-07T15:12:06.825Z] 
[2022-09-07T15:12:06.825Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() PASSED
[2022-09-07T15:12:06.825Z] 
[2022-09-07T15:12:06.825Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() STARTED
[2022-09-07T15:12:11.724Z] 
[2022-09-07T15:12:11.724Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() PASSED
[2022-09-07T15:12:11.724Z] 
[2022-09-07T15:12:11.724Z] PlaintextConsumerTest > testAutoCommitOnClose() 
STARTED
[2022-09-07T15:12:19.701Z] 
[2022-09-07T15:12:19.701Z] PlaintextConsumerTest > testAutoCommitOnClose() 
PASSED
[2022-09-07T15:12:19.701Z] 
[2022-09-07T15:12:19.701Z] PlaintextConsumerTest > testListTopics() STARTED
[2022-09-07T15:12:25.963Z] 
[2022-09-07T15:12:25.963Z] PlaintextConsumerTest > testListTopics() PASSED
[2022-09-07T15:12:25.963Z] 
[2022-09-07T15:12:25.963Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() STARTED
[2022-09-07T15:12:35.775Z] 
[2022-09-07T15:12:35.775Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() PASSED
[2022-09-07T15:12:35.775Z] 
[2022-09-07T15:12:35.775Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignor() STARTED
[2022-09-07T15:13:05.005Z] 
[2022-09-07T15:13:05.005Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignor() PASSED
[2022-09-07T15:13:05.005Z] 
[2022-09-07T15:13:05.005Z] PlaintextConsumerTest > testInterceptors() STARTE

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #78

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 458305 lines...]
[2022-09-07T15:16:32.161Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology STARTED
[2022-09-07T15:16:38.638Z] 
[2022-09-07T15:16:38.638Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology PASSED
[2022-09-07T15:16:38.638Z] 
[2022-09-07T15:16:38.638Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] STARTED
[2022-09-07T15:16:47.893Z] 
[2022-09-07T15:16:47.893Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-07T15:16:47.893Z] 
[2022-09-07T15:16:47.893Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-07T15:16:53.218Z] 
[2022-09-07T15:16:53.218Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-07T15:16:53.218Z] 
[2022-09-07T15:16:53.218Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-07T15:16:56.868Z] 
[2022-09-07T15:16:56.868Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] PASSED
[2022-09-07T15:16:56.868Z] 
[2022-09-07T15:16:56.868Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] STARTED
[2022-09-07T15:17:14.837Z] 
[2022-09-07T15:17:14.837Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-07T15:17:14.837Z] 
[2022-09-07T15:17:14.837Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-07T15:17:16.323Z] 
[2022-09-07T15:17:16.324Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2022-09-07T15:17:16.324Z] 
[2022-09-07T15:17:16.324Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2022-09-07T15:17:16.752Z] 
[2022-09-07T15:17:16.752Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-07T15:17:16.752Z] 
[2022-09-07T15:17:16.752Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-07T15:17:17.715Z] 
[2022-09-07T15:17:17.715Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-07T15:17:18.671Z] 
[2022-09-07T15:17:18.671Z] 492 tests completed, 1 failed
[2022-09-07T15:17:20.459Z] There were failing tests. See the report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/build/reports/tests/integrationTest/index.html
[2022-09-07T15:17:21.611Z] 
[2022-09-07T15:17:21.611Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-09-07T15:17:21.611Z] 
[2022-09-07T15:17:21.611Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-09-07T15:17:21.611Z] 
[2022-09-07T15:17:21.611Z] See 
https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings
[2022-09-07T15:17:21.611Z] 
[2022-09-07T15:17:21.611Z] BUILD SUCCESSFUL in 2h 54m 17s
[2022-09-07T15:17:21.611Z] 208 actionable tasks: 113 executed, 95 up-to-date
[2022-09-07T15:17:22.568Z] 
[2022-09-07T15:17:22.568Z] See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/build/reports/profile/profile-2022-09-07-12-23-09.html
[2022-09-07T15:17:22.568Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2022-09-07T15:17:23.450Z] Recording test results
[2022-09-07T15:17:25.298Z] 
[2022-09-07T15:17:25.298Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2022-09-07T15:17:25.298Z] 
[2022-09-07T15:17:25.298Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2022-09-07T15:17:36.160Z] 
[2022-09-07T15:17:36.160Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2022-09-07T15:17:36.160Z] 
[2022-09-07T15:17:36.160Z] 
org.apache.kafka.streams.integration.StreamStreamJoinI

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1204

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 375625 lines...]
[2022-09-07T15:45:10.824Z] > Task :connect:json:publishToMavenLocal
[2022-09-07T15:45:10.824Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-09-07T15:45:10.824Z] > Task :connect:api:publishToMavenLocal
[2022-09-07T15:45:10.824Z] 
[2022-09-07T15:45:10.824Z] > Task :streams:javadoc
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.824Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.825Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-07T15:45:10.825Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-09-07T15:45:10.825Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-09-07T15:45:10.825Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-09-07T15:45:11.819Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-09-07T15:45:11.819Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-09-07T15:45:11.819Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2022-09-07T15:45:11.819Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-09-07T15:45:11.819Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-09-07T15:45:11.819Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryRe

[jira] [Resolved] (KAFKA-14200) kafka-features.sh must exit with non-zero error code on error

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14200.
--
  Reviewer: David Arthur
Resolution: Fixed

> kafka-features.sh must exit with non-zero error code on error
> -
>
> Key: KAFKA-14200
> URL: https://issues.apache.org/jira/browse/KAFKA-14200
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.3.0, 3.3
>Reporter: Colin McCabe
>Assignee: Colin McCabe
>Priority: Blocker
> Fix For: 3.3.0
>
>
> kafka-features.sh must exit with a non-zero error code on error. We must do 
> this in order to catch regressions like KAFKA-13990.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14207) Add a 6.10 section for KRaft

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-14207:
--

 Summary: Add a 6.10 section for KRaft
 Key: KAFKA-14207
 URL: https://issues.apache.org/jira/browse/KAFKA-14207
 Project: Kafka
  Issue Type: Sub-task
  Components: documentation
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio
 Fix For: 3.3.0


The section should talk about:
 # Limitation
 # Recommended deployment: external controller
 # How to start a KRaft cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #60

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 572650 lines...]
[2022-09-07T19:31:57.685Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-07T19:31:58.613Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2022-09-07T19:31:58.613Z] streams-5: SMOKE-TEST-CLIENT-CLOSED
[2022-09-07T19:31:58.613Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2022-09-07T19:31:58.613Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2022-09-07T19:31:58.613Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2022-09-07T19:31:58.613Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2022-09-07T19:32:03.188Z] 
[2022-09-07T19:32:03.188Z] BUILD SUCCESSFUL in 3h 17m 58s
[2022-09-07T19:32:03.188Z] 212 actionable tasks: 115 executed, 97 up-to-date
[2022-09-07T19:32:04.115Z] 
[2022-09-07T19:32:04.115Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.3/build/reports/profile/profile-2022-09-07-16-14-08.html
[2022-09-07T19:32:04.115Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2022-09-07T19:32:04.961Z] Recording test results
[2022-09-07T19:32:11.689Z] 
[2022-09-07T19:32:11.689Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] PASSED
[2022-09-07T19:32:11.689Z] 
[2022-09-07T19:32:11.689Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once_v2] STARTED
[2022-09-07T19:32:21.883Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2022-09-07T19:32:21.885Z] Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-09-07T19:33:08.742Z] 
[2022-09-07T19:33:08.742Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once_v2] PASSED
[2022-09-07T19:33:08.742Z] 
[2022-09-07T19:33:08.742Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] STARTED
[2022-09-07T19:33:08.742Z] 
[2022-09-07T19:33:08.742Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] PASSED
[2022-09-07T19:33:08.742Z] 
[2022-09-07T19:33:08.742Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] STARTED
[2022-09-07T19:33:14.686Z] 
[2022-09-07T19:33:14.686Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] PASSED
[2022-09-07T19:33:14.686Z] 
[2022-09-07T19:33:14.686Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] STARTED
[2022-09-07T19:33:21.722Z] 
[2022-09-07T19:33:21.722Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] PASSED
[2022-09-07T19:33:21.722Z] 
[2022-09-07T19:33:21.722Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = true] STARTED
[2022-09-07T19:33:27.493Z] 
[2022-09-07T19:33:27.493Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = true] PASSED
[2022-09-07T19:33:27.493Z] 
[2022-09-07T19:33:27.493Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2022-09-07T19:33:34.529Z] 
[2022-09-07T19:33:34.529Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2022-09-07T19:33:34.529Z] 
[2022-09-07T19:33:34.529Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2022-09-07T19:33:40.301Z] 
[2022-09-07T19:33:40.301Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2022-09-07T19:33:40.301Z] 
[2022-09-07T19:33:40.301Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2022-09-07T19:33:46.075Z] 
[2022-09-07T19:33:46.075Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2022-09-07T19:33:46.075Z] 
[2022-09-07T19:33:46.075Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = true] STARTED
[2022-09-07T19:33:53.113Z] 
[2022-09-07T19:33:53.113Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = true] PASSED
[2022-09-07T19:33:53.113Z] 
[2022-09-07T19:33:53.113Z] 
org.apache.kafka.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1205

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 513828 lines...]
[2022-09-07T20:00:31.590Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2022-09-07T20:00:31.590Z] 
[2022-09-07T20:00:31.590Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2022-09-07T20:00:57.707Z] 
[2022-09-07T20:00:57.707Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-07T20:00:57.707Z] 
[2022-09-07T20:00:57.707Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-07T20:01:20.093Z] 
[2022-09-07T20:01:20.093Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-07T20:01:20.093Z] 
[2022-09-07T20:01:20.093Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-07T20:01:27.386Z] 
[2022-09-07T20:01:27.386Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-07T20:01:27.386Z] 
[2022-09-07T20:01:27.386Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-07T20:01:46.540Z] 
[2022-09-07T20:01:46.540Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-07T20:01:46.540Z] 
[2022-09-07T20:01:46.540Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-07T20:01:47.495Z] 
[2022-09-07T20:01:47.495Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-07T20:01:47.495Z] 
[2022-09-07T20:01:47.495Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-07T20:01:49.279Z] 
[2022-09-07T20:01:49.279Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-07T20:01:50.234Z] 
[2022-09-07T20:01:50.234Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-09-07T20:01:52.908Z] 
[2022-09-07T20:01:52.908Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-09-07T20:01:52.908Z] 
[2022-09-07T20:01:52.908Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() STARTED
[2022-09-07T20:02:01.444Z] 
[2022-09-07T20:02:01.444Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() PASSED
[2022-09-07T20:02:01.444Z] 
[2022-09-07T20:02:01.444Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() STARTED
[2022-09-07T20:02:05.100Z] 
[2022-09-07T20:02:05.100Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() PASSED
[2022-09-07T20:02:05.100Z] 
[2022-09-07T20:02:05.100Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-09-07T20:02:08.755Z] 
[2022-09-07T20:02:08.755Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-09-07T20:02:08.755Z] 
[2022-09-07T20:02:08.755Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-09-07T20:02:34.897Z] 
[2022-09-07T20:02:34.897Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-09-07T20:02:34.897Z] 
[2022-09-07T20:02:34.897Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() STARTED
[2022-09-07T20:02:42.040Z] 
[2022-09-07T20:02:42.040Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() PASSED
[2022-09-07T20:02:42.040Z] 
[2022-09-07T20:02:42.040Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-09-07T20:02:47.902Z] 
[2022-09-07T20:02:47.902Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-09-07T20:02:47.902Z] 
[2022-09-07T20:02:47.902Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-09-07T20:02:52.609Z] 
[2022-09-07T20:02:52.609Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-09-07T20:02:52.609Z] 
[2022-09-07T20:02:52.609Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() STARTED
[2022-09-07T20:02:56.262Z] 
[2022-09-07T20:02:56.262Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() PASSED
[2022-09-07T20:03:00.092Z] 
[2022-09-07T20:03:00.092Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 STARTED
[2022-09-07T20:03:01.052Z] 
[2022-09-07T20:03:01.052Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 PASSED
[2022-09-07T20:03:01.052Z] 
[2022-09-07T20:03:01.052Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() STARTED
[2022-09-07T20:03:01.052Z] 
[2022-09-07T20:03:01.052Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverla

Re: [VOTE] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

[Colin McCabe]

+1 (binding).

One nitpick: when the user sets AllowReplicationFactorChange = false, the 
exception the user gets back from AdminClient should mention that this was the 
problem.  If the exception just says "The broker does not support 
ALTER_PARTITION_REASSIGNMENTS with version in range [1, 1]. The supported range 
is [0, 0]." the user will be confused about what the problem is. Instead, the 
exception should mention that the broker does not support 
AllowReplicationFactorChange.

best,
Colin


On Wed, Sep 7, 2022, at 06:11, David Jacot wrote:
> +1 from me. Thanks, Stan!
>
> On Tue, Aug 23, 2022 at 12:10 PM Luke Chen  wrote:
>>
>> Hi Stanislav,
>>
>> Thanks for the KIP.
>> The solution looks reasonable to me.
>> +1 from me.
>>
>> Thank you.
>> Luke
>>
>> On Tue, Aug 23, 2022 at 6:07 AM Stanislav Kozlovski
>>  wrote:
>>
>> > Hello,
>> >
>> > I'd like to start a vote on KIP-860, which adds a client-provided option to
>> > the AlterPartitionReassignmentsRequest that allows the user to guard
>> > against an unintentional change in the replication factor during partition
>> > reassignments.
>> >
>> > Discuss Thread:
>> > https://lists.apache.org/thread/bhrqjd4vb05xtztkdo8py374m9dgq69r
>> > KIP:
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
>> > JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
>> >
>> >
>> > --
>> > Best,
>> > Stanislav
>> >

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

[Colin McCabe]

Hi Ismael,

I think this issue comes up when people write software that automatically 
creates partition reassignments to balance the cluster. Cruise Control is one 
example; Confluent also has some software that does this. If there is already a 
reassignment that is going on for some partition and the software tries to 
create a new reassignment for that partition, the software may inadvertently 
change the replication factor.

In general, I think some people find it surprising that reassignment can change 
the replication factor of a partition. When we outlined the reassignment API in 
KIP-455 we maintained the ability to do this, since the old ZK-based API had 
always been able to do it. But this was a bit controversial. Maybe it would 
have been more intuitive to preserve replication factor by default unless the 
user explicitly stated that they wanted to change it. So in a sense, you could 
view this as a fix for KIP-455 :) (in my opinion, at least)

best,
Colin


On Wed, Sep 7, 2022, at 07:07, Ismael Juma wrote:
> Thanks for the KIP. Can we explain a bit more why this is an important use
> case to address? For example, do we have concrete examples of people
> running into this? The way the KIP is written, it sounds like a potential
> problem but no information is given on whether it's a real problem in
> practice.
>
> Ismael
>
> On Thu, Jul 28, 2022 at 2:00 AM Stanislav Kozlovski
>  wrote:
>
>> Hey all,
>>
>> I'd like to start a discussion on a proposal to help API users from
>> inadvertently increasing the replication factor of a topic through
>> the alter partition reassignments API. The KIP describes two fairly
>> easy-to-hit race conditions in which this can happen.
>>
>> The KIP itself is pretty simple, yet has a couple of alternatives that can
>> help solve the same problem. I would appreciate thoughts from the community
>> on how you think we should proceed, and whether the proposal makes sense in
>> the first place.
>>
>> Thanks!
>>
>> KIP:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
>> JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
>>
>> --
>> Best,
>> Stanislav
>>

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

[Ismael Juma]

Thanks for the details, Colin. I understand how this can happen. But this
API has been out for a long time. Are we saying that we have seen Cruise
Control cause this kind of problem? If so, it would be good to mention it
in the KIP as evidence that the current approach is brittle.

Ismael

On Wed, Sep 7, 2022 at 2:15 PM Colin McCabe  wrote:

> Hi Ismael,
>
> I think this issue comes up when people write software that automatically
> creates partition reassignments to balance the cluster. Cruise Control is
> one example; Confluent also has some software that does this. If there is
> already a reassignment that is going on for some partition and the software
> tries to create a new reassignment for that partition, the software may
> inadvertently change the replication factor.
>
> In general, I think some people find it surprising that reassignment can
> change the replication factor of a partition. When we outlined the
> reassignment API in KIP-455 we maintained the ability to do this, since the
> old ZK-based API had always been able to do it. But this was a bit
> controversial. Maybe it would have been more intuitive to preserve
> replication factor by default unless the user explicitly stated that they
> wanted to change it. So in a sense, you could view this as a fix for
> KIP-455 :) (in my opinion, at least)
>
> best,
> Colin
>
>
> On Wed, Sep 7, 2022, at 07:07, Ismael Juma wrote:
> > Thanks for the KIP. Can we explain a bit more why this is an important
> use
> > case to address? For example, do we have concrete examples of people
> > running into this? The way the KIP is written, it sounds like a potential
> > problem but no information is given on whether it's a real problem in
> > practice.
> >
> > Ismael
> >
> > On Thu, Jul 28, 2022 at 2:00 AM Stanislav Kozlovski
> >  wrote:
> >
> >> Hey all,
> >>
> >> I'd like to start a discussion on a proposal to help API users from
> >> inadvertently increasing the replication factor of a topic through
> >> the alter partition reassignments API. The KIP describes two fairly
> >> easy-to-hit race conditions in which this can happen.
> >>
> >> The KIP itself is pretty simple, yet has a couple of alternatives that
> can
> >> help solve the same problem. I would appreciate thoughts from the
> community
> >> on how you think we should proceed, and whether the proposal makes
> sense in
> >> the first place.
> >>
> >> Thanks!
> >>
> >> KIP:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> >> JIRA: https://issues.apache.org/jira/browse/KAFKA-14121
> >>
> >> --
> >> Best,
> >> Stanislav
> >>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1206

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 501968 lines...]
[2022-09-08T01:47:23.569Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2022-09-08T01:47:24.506Z] 
[2022-09-08T01:47:24.506Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2022-09-08T01:47:24.506Z] 
[2022-09-08T01:47:24.506Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2022-09-08T01:47:52.006Z] 
[2022-09-08T01:47:52.007Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-08T01:47:52.007Z] 
[2022-09-08T01:47:52.007Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-08T01:48:12.156Z] 
[2022-09-08T01:48:12.156Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-08T01:48:12.156Z] 
[2022-09-08T01:48:12.156Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-08T01:48:18.004Z] 
[2022-09-08T01:48:18.004Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-08T01:48:18.004Z] 
[2022-09-08T01:48:18.004Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-08T01:48:38.712Z] 
[2022-09-08T01:48:38.712Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-08T01:48:38.712Z] 
[2022-09-08T01:48:38.712Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-08T01:48:39.743Z] 
[2022-09-08T01:48:39.743Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-08T01:48:39.743Z] 
[2022-09-08T01:48:39.743Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-08T01:48:41.879Z] 
[2022-09-08T01:48:41.879Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-08T01:48:41.879Z] 
[2022-09-08T01:48:41.879Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-09-08T01:48:44.684Z] 
[2022-09-08T01:48:44.684Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-09-08T01:48:44.684Z] 
[2022-09-08T01:48:44.684Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() STARTED
[2022-09-08T01:48:48.594Z] 
[2022-09-08T01:48:48.594Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() PASSED
[2022-09-08T01:48:48.594Z] 
[2022-09-08T01:48:48.594Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() STARTED
[2022-09-08T01:49:00.720Z] 
[2022-09-08T01:49:00.720Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() PASSED
[2022-09-08T01:49:00.720Z] 
[2022-09-08T01:49:00.720Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-09-08T01:49:05.679Z] 
[2022-09-08T01:49:05.679Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-09-08T01:49:05.679Z] 
[2022-09-08T01:49:05.679Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-09-08T01:49:26.636Z] 
[2022-09-08T01:49:26.636Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-09-08T01:49:26.636Z] 
[2022-09-08T01:49:26.636Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() STARTED
[2022-09-08T01:49:29.481Z] 
[2022-09-08T01:49:29.481Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() PASSED
[2022-09-08T01:49:29.481Z] 
[2022-09-08T01:49:29.481Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-09-08T01:49:34.692Z] 
[2022-09-08T01:49:34.692Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-09-08T01:49:34.692Z] 
[2022-09-08T01:49:34.692Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-09-08T01:49:38.446Z] 
[2022-09-08T01:49:38.446Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-09-08T01:49:38.446Z] 
[2022-09-08T01:49:38.446Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() STARTED
[2022-09-08T01:49:39.384Z] 
[2022-09-08T01:49:39.384Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() PASSED
[2022-09-08T01:49:43.140Z] 
[2022-09-08T01:49:43.140Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 STARTED
[2022-09-08T01:49:43.140Z] 
[2022-09-08T01:49:43.140Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 PASSED
[2022-09-08T01:49:43.140Z] 
[2022-09-08T01:49:43.140Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverla

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1207

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 504957 lines...]
[2022-09-08T04:22:05.955Z] 
[2022-09-08T04:22:05.955Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2022-09-08T04:22:16.521Z] 
[2022-09-08T04:22:16.521Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2022-09-08T04:22:16.521Z] 
[2022-09-08T04:22:16.521Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2022-09-08T04:22:17.732Z] 
[2022-09-08T04:22:17.732Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2022-09-08T04:22:17.732Z] 
[2022-09-08T04:22:17.732Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2022-09-08T04:22:18.677Z] 
[2022-09-08T04:22:18.677Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2022-09-08T04:22:18.677Z] 
[2022-09-08T04:22:18.677Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2022-09-08T04:23:03.742Z] 
[2022-09-08T04:23:03.742Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-08T04:23:03.742Z] 
[2022-09-08T04:23:03.742Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-08T04:23:44.302Z] 
[2022-09-08T04:23:44.302Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-08T04:23:44.302Z] 
[2022-09-08T04:23:44.302Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-08T04:23:56.733Z] 
[2022-09-08T04:23:56.733Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-08T04:23:56.733Z] 
[2022-09-08T04:23:56.733Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-08T04:24:49.340Z] 
[2022-09-08T04:24:49.340Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-08T04:24:49.340Z] 
[2022-09-08T04:24:49.340Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-08T04:24:49.340Z] 
[2022-09-08T04:24:49.340Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-08T04:24:49.340Z] 
[2022-09-08T04:24:49.340Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-08T04:24:50.390Z] 
[2022-09-08T04:24:50.390Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-08T04:24:51.344Z] 
[2022-09-08T04:24:51.344Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-09-08T04:24:56.690Z] 
[2022-09-08T04:24:56.690Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-09-08T04:24:56.690Z] 
[2022-09-08T04:24:56.690Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() STARTED
[2022-09-08T04:24:58.473Z] 
[2022-09-08T04:24:58.473Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() PASSED
[2022-09-08T04:24:58.473Z] 
[2022-09-08T04:24:58.473Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() STARTED
[2022-09-08T04:25:03.948Z] 
[2022-09-08T04:25:03.948Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() PASSED
[2022-09-08T04:25:03.948Z] 
[2022-09-08T04:25:03.948Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-09-08T04:25:05.076Z] 
[2022-09-08T04:25:05.076Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-09-08T04:25:05.076Z] 
[2022-09-08T04:25:05.076Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-09-08T04:25:25.237Z] 
[2022-09-08T04:25:25.237Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-09-08T04:25:25.237Z] 
[2022-09-08T04:25:25.237Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() STARTED
[2022-09-08T04:25:30.432Z] 
[2022-09-08T04:25:30.432Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() PASSED
[2022-09-08T04:25:30.432Z] 
[2022-09-08T04:25:30.432Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-09-08T04:25:34.180Z] 
[2022-09-08T04:25:34.180Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-09-08T04:25:34.180Z] 
[2022-09-08T04:25:34.180Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-09-08T04:25:38.384Z] 
[2022-09-08T04:25:38.384Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-09-08T04:25:38.384Z] 
[2022-09-08T04:25:38.384Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() STARTED
[2022-09-08T04:25:39.437Z] 
[2022-09-08T04:25:39.437Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfter

Re: [DISCUSS] KIP-864: Add End-To-End Latency Metrics to Connectors

[Yash Mayya]

Hi Jorge,

Thanks for the changes. With regard to having per batch vs per record
metrics, the additional overhead I was referring to wasn't about whether or
not we would need to iterate over all the records in a batch. I was
referring to the potential additional overhead caused by the higher volume
of calls to Sensor::record on the sensors for the new metrics (as compared
to the existing batch only metrics), especially for high throughput
connectors where batch sizes could be large. I guess we may want to do some
sort of performance testing and get concrete numbers to verify whether this
is a valid concern or not?

Thanks,
Yash

On Tue, Sep 6, 2022 at 4:42 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi Sagar and Yash,
>
> > the way it's defined in
> https://kafka.apache.org/documentation/#connect_monitoring for the metrics
>
> 4.1. Got it. Add it to the KIP.
>
> > The only thing I would argue is do we need sink-record-latency-min? Maybe
> we
> > could remove this min metric as well and make all of the 3 e2e metrics
> > consistent
>
> 4.2 I see. Will remove it from the KIP.
>
> > Probably users can track the metrics at their end to
> > figure that out. Do you think that makes sense?
>
> 4.3. Yes, agree. With these new metrics it should be easier for users to
> track this.
>
> > I think it makes sense to not have a min metric for either to remain
> > consistent with the existing put-batch and poll-batch metrics
>
> 5.1. Got it. Same as 4.2
>
> > Another naming related suggestion I had was with the
> > "convert-time" metrics - we should probably include transformations in
> the
> > name since SMTs could definitely be attributable to a sizable chunk of
> the
> > latency depending on the specific transformation chain.
>
> 5.2. Make sense. I'm proposing to add `sink-record-convert-transform...`
> and `source-record-transform-convert...` to represent correctly the order
> of operations.
>
> > it seems like both source and sink tasks only record metrics at a "batch"
> > level, not on an individual record level. I think it might be additional
> > overhead if we want to record these new metrics all at the record level?
>
> 5.3. I considered at the beginning to implement all metrics at the batch
> level, but given how the framework process records, I fallback to the
> proposed approach:
> - Sink Task:
>   - `WorkerSinkTask#convertMessages(msgs)` already iterates over records,
> so there is no additional overhead to capture record latency per record.
> -
>
> https://github.com/apache/kafka/blob/9841647c4fe422532f448423c92d26e4fdcb8932/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L490-L514
>   - `WorkerSinkTask#convertAndTransformRecord(record)` actually happens
> individually. Measuring this operation per batch would include processing
> that is not strictly part of "convert and transform"
> -
>
> https://github.com/apache/kafka/blob/9841647c4fe422532f448423c92d26e4fdcb8932/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L518
> - Source Task:
>   - `AbstractWorkerSourceTask#sendRecords` iterates over a batch and
> applies transforms and convert record individually as well:
> -
>
> https://github.com/apache/kafka/blob/9841647c4fe422532f448423c92d26e4fdcb8932/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L389-L390
>
> > This might require some additional changes -
> > for instance, with the "sink-record-latency" metric, we might only want
> to
> > have a "max" metric since "avg" would require recording a value on the
> > sensor for each record (whereas we can get a "max" by only recording a
> > metric value for the oldest record in each batch).
>
> 5.4. Recording record-latency per batch may not be as useful as there is no
> guarantee that the oldest record will be representative of the batch.
>
> On Sat, 3 Sept 2022 at 16:02, Yash Mayya  wrote:
>
> > Hi Jorge and Sagar,
> >
> > I think it makes sense to not have a min metric for either to remain
> > consistent with the existing put-batch and poll-batch metrics (it doesn't
> > seem particularly useful either anyway). Also, the new
> > "sink-record-latency" metric name looks fine to me, thanks for making the
> > changes! Another naming related suggestion I had was with the
> > "convert-time" metrics - we should probably include transformations in
> the
> > name since SMTs could definitely be attributable to a sizable chunk of
> the
> > latency depending on the specific transformation chain.
> >
> > I have one high level question with respect to implementation -
> currently,
> > it seems like both source and sink tasks only record metrics at a "batch"
> > level, not on an individual record level. I think it might be additional
> > overhead if we want to record these new metrics all at the record level?
> > Could we instead make all of these new metrics for batches of records
> > rather than individual records in orde