date:20210923

Building 3.0.0 and Task :connect:signMavenJavaPublication FAILED

2021-09-23 Thread Jacek Laskowski

Hi,

I've been trying to build the latest version of Kafka 3.0.0 and no joy so
far.

$ ./gradlew clean releaseTarGz install
...
> Configure project :
Starting build with version 3.0.0 using Gradle 7.1.1, Java 11 and Scala
2.13.6

> Task :connect:signMavenJavaPublication FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':connect:signMavenJavaPublication'.
> Cannot perform signing task ':connect:signMavenJavaPublication' because
it has no configured signatory

How to fix it? I can't seem to find anything in README.md.

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski

Re: Building 3.0.0 and Task :connect:signMavenJavaPublication FAILED

2021-09-23 Thread Jacek Laskowski

Hi,

I found a solution! It's -PskipSigning=true and it is in README.md actually
(but somehow it was not required in the earlier versions, e.g. 2.8.0):

> skipSigning: skips signing of artifacts.

The following works nicely!

./gradlew clean releaseTarGz install -PskipSigning=true

[1]
https://github.com/apache/kafka/blob/3.0.0/README.md#common-build-options

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Sep 23, 2021 at 10:05 AM Jacek Laskowski  wrote:

> Hi,
>
> I've been trying to build the latest version of Kafka 3.0.0 and no joy so
> far.
>
> $ ./gradlew clean releaseTarGz install
> ...
> > Configure project :
> Starting build with version 3.0.0 using Gradle 7.1.1, Java 11 and Scala
> 2.13.6
>
> > Task :connect:signMavenJavaPublication FAILED
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Execution failed for task ':connect:signMavenJavaPublication'.
> > Cannot perform signing task ':connect:signMavenJavaPublication' because
> it has no configured signatory
>
> How to fix it? I can't seem to find anything in README.md.
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>

[jira] [Resolved] (KAFKA-10544) Convert KTable aggregations to new PAPI

2021-09-23 Thread Jorge Esteban Quilcate Otoya (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Esteban Quilcate Otoya resolved KAFKA-10544.
--
Resolution: Fixed

https://github.com/apache/kafka/pull/11316

> Convert KTable aggregations to new PAPI
> ---
>
> Key: KAFKA-10544
> URL: https://issues.apache.org/jira/browse/KAFKA-10544
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: John Roesler
>Assignee: Jorge Esteban Quilcate Otoya
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

GC pressure of the java client

2021-09-23 Thread Michail Prusakov

Hi All,

I've been recently profiling the java client and noticed that it produces
considerable pressure on GC.

My main concern is the consumer as it creates multiple collections during a
single poll command: in my test system i have a consumer that polls a topic
with 100 partitions and even though no messages are coming through, the
code allocates around 100M per 5 minutes.

I've investigated the allocations and the biggest ones can be easily
avoided by moving them to the instance level, something that can be done as
KafkaConsumer is not thread safe. I've created a PR, available here
https://github.com/apache/kafka/pull/11353, that fixes the biggest
allocation.

Do you agree with the approach and if so could possibly look at my PR? Many
thanks in advance.

Regards,
Michail

[jira] [Created] (KAFKA-13320) Suggestion: SMT support for null key/value should be documented

2021-09-23 Thread Ben Ellis (Jira)

Ben Ellis created KAFKA-13320:
-

 Summary: Suggestion: SMT support for null key/value should be 
documented
 Key: KAFKA-13320
 URL: https://issues.apache.org/jira/browse/KAFKA-13320
 Project: Kafka
  Issue Type: Wish
  Components: KafkaConnect
Reporter: Ben Ellis


While working with a JDBC Sink Connector, I noticed that some SMT choke on a 
tombstone (null value) while others handle tombstones fine.

For example:

```
"transforms": "flattenKey,valueToJSON,wrapValue,addTimestamp", 
"transforms.flattenKey.type": 
"org.apache.kafka.connect.transforms.Flatten$Key", 
"transforms.flattenKey.delimiter": "_", "transforms.valueToJSON.type": 
"com.github.jcustenborder.kafka.connect.transform.common.ToJSON$Value", 
"transforms.valueToJSON.schemas.enable": "false", 
"transforms.valueToJSON.predicate": "tombstone", 
"transforms.valueToJSON.negate": true, 
"transforms.wrapValue.type":"org.apache.kafka.connect.transforms.HoistField$Value",
 "transforms.wrapValue.field":"matrix", "transforms.wrapValue.predicate": 
"tombstone", "transforms.wrapValue.negate": true, 
"transforms.addTimestamp.type": 
"org.apache.kafka.connect.transforms.InsertField$Value", 
"transforms.addTimestamp.timestamp.field": "message_timestamp", "predicates": 
"tombstone", "predicates.tombstone.type": 
"org.apache.kafka.connect.transforms.predicates.RecordIsTombstone"

```

To avoid the cryptic error “java.lang.ClassCastException: class 
java.util.HashMap cannot be cast to class org.apache.kafka.connect.data.Struct” 
when processing a tombstone record, I had to add a negated predicate of 
RecordIsTombstone for ToJSON (community SMT) and HoistField, but did not need 
to add that to InsertField.

Digging in the source, I find that InsertField handles the case where key or 
value is null:
https://github.com/a0x8o/kafka/blob/f8237749f6ad34c09154f807e53273be64e1261e/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/InsertField.java#L130

^ Thanks to this, there's no need to add a predicate to skip InsertField$Value 
when value is null.

It would help if the docs listed how the individual SMTs behave when dealing 
with a null key/value.

Of course we can always find this out by trial and error or by studying the 
source code.
But if we were to make a best practice of describing how an SMT handles null 
key/value, that would have two benefits:
1) Save developers time when working with the official (shipped with Kafka) SMT
2) Inspire developers who write their own SMT to likewise document how they 
handle null key/value

Perhaps a standard way of dealing with nulls ("no-op if key/value is null") 
could be promoted, and SMT authors would only need to document their behavior 
when it differs.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #491

2021-09-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 494148 lines...]
[2021-09-23T15:33:50.012Z] 
[2021-09-23T15:33:50.012Z] FetchRequestBetweenDifferentIbpTest > 
testControllerOldToNewIBP() PASSED
[2021-09-23T15:33:50.012Z] 
[2021-09-23T15:33:50.012Z] FetchRequestBetweenDifferentIbpTest > 
testControllerNewIBP() STARTED
[2021-09-23T15:33:59.005Z] 
[2021-09-23T15:33:59.005Z] AclCommandTest > testAclCliWithAuthorizer() PASSED
[2021-09-23T15:33:59.005Z] 
[2021-09-23T15:33:59.005Z] AclCommandTest > testAclCliWithClientId() STARTED
[2021-09-23T15:33:59.078Z] 
[2021-09-23T15:33:59.078Z] FetchRequestBetweenDifferentIbpTest > 
testControllerNewIBP() PASSED
[2021-09-23T15:33:59.078Z] 
[2021-09-23T15:33:59.078Z] FetchRequestBetweenDifferentIbpTest > 
testControllerOldIBP() STARTED
[2021-09-23T15:34:06.713Z] 
[2021-09-23T15:34:06.713Z] AclCommandTest > testAclCliWithClientId() PASSED
[2021-09-23T15:34:06.713Z] 
[2021-09-23T15:34:06.713Z] AclCommandTest > testInvalidAuthorizerProperty() 
STARTED
[2021-09-23T15:34:06.713Z] 
[2021-09-23T15:34:06.713Z] AclCommandTest > testInvalidAuthorizerProperty() 
PASSED
[2021-09-23T15:34:06.713Z] 
[2021-09-23T15:34:06.713Z] AclCommandTest > 
testAclsOnPrefixedResourcesWithAdminAPI() STARTED
[2021-09-23T15:34:07.058Z] 
[2021-09-23T15:34:07.058Z] FetchRequestBetweenDifferentIbpTest > 
testControllerOldIBP() PASSED
[2021-09-23T15:34:07.058Z] 
[2021-09-23T15:34:07.058Z] FetchRequestBetweenDifferentIbpTest > 
testControllerNewToOldIBP() STARTED
[2021-09-23T15:34:08.282Z] 
[2021-09-23T15:34:08.282Z] AclCommandTest > 
testAclsOnPrefixedResourcesWithAdminAPI() PASSED
[2021-09-23T15:34:08.282Z] 
[2021-09-23T15:34:08.282Z] AclCommandTest > testPatternTypes() STARTED
[2021-09-23T15:34:11.635Z] 
[2021-09-23T15:34:11.635Z] AclCommandTest > testPatternTypes() PASSED
[2021-09-23T15:34:11.635Z] 
[2021-09-23T15:34:11.636Z] AclCommandTest > 
testProducerConsumerCliWithAdminAPI() STARTED
[2021-09-23T15:34:19.913Z] 
[2021-09-23T15:34:19.913Z] AclCommandTest > 
testProducerConsumerCliWithAdminAPI() PASSED
[2021-09-23T15:34:19.913Z] 
[2021-09-23T15:34:19.913Z] AclCommandTest > 
testAclsOnPrefixedResourcesWithAuthorizer() STARTED
[2021-09-23T15:34:19.913Z] 
[2021-09-23T15:34:19.913Z] AclCommandTest > 
testAclsOnPrefixedResourcesWithAuthorizer() PASSED
[2021-09-23T15:34:19.913Z] 
[2021-09-23T15:34:19.913Z] AclCommandTest > 
testProducerConsumerCliWithAuthorizer() STARTED
[2021-09-23T15:34:25.893Z] 
[2021-09-23T15:34:25.893Z] FetchRequestBetweenDifferentIbpTest > 
testControllerNewToOldIBP() PASSED
[2021-09-23T15:34:27.076Z] 
[2021-09-23T15:34:27.076Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-09-23T15:34:27.076Z] 
[2021-09-23T15:34:27.076Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2021-09-23T15:34:27.076Z] 
[2021-09-23T15:34:27.076Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-09-23T15:34:27.076Z] 
[2021-09-23T15:34:27.076Z] BUILD SUCCESSFUL in 1h 54m 8s
[2021-09-23T15:34:27.076Z] 202 actionable tasks: 109 executed, 93 up-to-date
[2021-09-23T15:34:27.076Z] 
[2021-09-23T15:34:27.076Z] See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2021-09-23-13-40-30.html
[2021-09-23T15:34:27.076Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2021-09-23T15:34:28.230Z] Recording test results
[2021-09-23T15:34:29.941Z] 
[2021-09-23T15:34:29.941Z] AclCommandTest > 
testProducerConsumerCliWithAuthorizer() PASSED
[2021-09-23T15:34:29.941Z] 
[2021-09-23T15:34:29.941Z] AclCommandTest > testAclCliWithAdminAPI() STARTED
[2021-09-23T15:34:36.218Z] 
[2021-09-23T15:34:36.218Z] AclCommandTest > testAclCliWithAdminAPI() PASSED
[2021-09-23T15:34:36.218Z] 
[2021-09-23T15:34:36.218Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsExportImportPlanSingleGroupArg() STARTED
[2021-09-23T15:34:43.280Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2021-09-23T15:34:43.284Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2021-09-23T15:34:45.641Z] 
[2021-09-23T15:34:45.641Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsExportImportPlanSingleGroupArg() PASSED
[2021-09-23T15:34:45.641Z] 
[2021-09-23T15:34:45.641Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByHigherThanLatest() STARTED
[2021-09-23T15:34:54.043Z] 
[2021-09-23T15:34:54.043Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByHigherThanLatest() PASSED
[2021-09-2

[jira] [Created] (KAFKA-13321) Notify listener of leader change on registration

2021-09-23 Thread Jose Armando Garcia Sancio (Jira)

Jose Armando Garcia Sancio created KAFKA-13321:
--

 Summary: Notify listener of leader change on registration
 Key: KAFKA-13321
 URL: https://issues.apache.org/jira/browse/KAFKA-13321
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio


When a Listener is registered with the RaftClient, the RaftClient doesn't 
notify the listener of the current leader when it is an follower. The current 
implementation of RaftClient notifies this listener of the leader change if it 
is the current leader and it has caught up to the leader epoch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #492

2021-09-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 494953 lines...]
[2021-09-23T19:47:06.264Z] > Task :raft:compileTestJava UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :raft:testClasses UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :connect:json:testJar
[2021-09-23T19:47:06.264Z] > Task :connect:json:testSrcJar
[2021-09-23T19:47:06.264Z] > Task :core:compileScala UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :core:classes UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :metadata:testClasses UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :core:compileTestJava NO-SOURCE
[2021-09-23T19:47:06.264Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-09-23T19:47:06.264Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-09-23T19:47:06.264Z] > Task :core:compileTestScala UP-TO-DATE
[2021-09-23T19:47:06.264Z] > Task :core:testClasses UP-TO-DATE
[2021-09-23T19:47:07.467Z] 
[2021-09-23T19:47:07.467Z] > Task :streams:processMessages
[2021-09-23T19:47:07.467Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-09-23T19:47:07.467Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-09-23T19:47:07.467Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-09-23T19:47:07.467Z] 
[2021-09-23T19:47:07.467Z] > Task :streams:compileJava UP-TO-DATE
[2021-09-23T19:47:07.467Z] > Task :streams:classes UP-TO-DATE
[2021-09-23T19:47:07.467Z] > Task :streams:jar UP-TO-DATE
[2021-09-23T19:47:07.467Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-09-23T19:47:07.467Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-09-23T19:47:09.677Z] > Task :connect:api:javadoc
[2021-09-23T19:47:09.677Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-09-23T19:47:09.677Z] > Task :connect:api:jar UP-TO-DATE
[2021-09-23T19:47:09.677Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-09-23T19:47:09.677Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-09-23T19:47:09.677Z] > Task :connect:json:jar UP-TO-DATE
[2021-09-23T19:47:09.677Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-09-23T19:47:09.677Z] > Task :connect:api:javadocJar
[2021-09-23T19:47:09.677Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-09-23T19:47:09.677Z] > Task :connect:json:publishToMavenLocal
[2021-09-23T19:47:09.677Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-09-23T19:47:09.677Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-09-23T19:47:09.677Z] > Task :connect:api:testJar
[2021-09-23T19:47:09.677Z] > Task :connect:api:testSrcJar
[2021-09-23T19:47:09.677Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-09-23T19:47:09.677Z] > Task :connect:api:publishToMavenLocal
[2021-09-23T19:47:13.051Z] > Task :streams:javadoc
[2021-09-23T19:47:13.051Z] > Task :streams:javadocJar
[2021-09-23T19:47:14.679Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-09-23T19:47:14.679Z] > Task :streams:testClasses UP-TO-DATE
[2021-09-23T19:47:14.679Z] > Task :streams:testJar
[2021-09-23T19:47:14.679Z] > Task :streams:testSrcJar
[2021-09-23T19:47:14.679Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-09-23T19:47:14.679Z] > Task :streams:publishToMavenLocal
[2021-09-23T19:47:14.679Z] > Task :clients:javadoc
[2021-09-23T19:47:14.679Z] > Task :clients:javadocJar
[2021-09-23T19:47:15.858Z] 
[2021-09-23T19:47:15.858Z] > Task :clients:srcJar
[2021-09-23T19:47:15.858Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-09-23T19:47:15.858Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-09-23T19:47:17.064Z] 
[2021-09-23T19:47:17.064Z] > Task :clients:testJar
[2021-09-23T19:47:17.064Z] > Task :client

Re: [DISCUSS] KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

2021-09-23 Thread Kirk True

Hi Jun,

On Tue, Sep 21, 2021, at 10:51 AM, Jun Rao wrote:
> Hi, Kirk,
> 
> Thanks for the KIP. Does the proposal support reauthentication outlined
> in KIP-368?

Yes, the existing mechanism for re-authentication is implemented at a higher 
layer of the code, so we just get it for "free."

I've added a couple of sentences to that effect to the KIP.

Thanks,
Kirk

> 
> Jun
> 
> On Wed, Aug 25, 2021 at 8:54 PM Manikumar  wrote:
> 
> > Thanks for the reply,
> >
> > Can we also update the KIP about the testing approach?
> >
> > Thanks,
> >
> > On Wed, Aug 25, 2021 at 12:01 AM Kirk True  wrote:
> >
> > > Hi Manikumar!
> > >
> > > On Mon, Aug 23, 2021, at 12:59 PM, Manikumar wrote:
> > >
> > > Hi Kirk,
> > >
> > > Thanks for the KIP!
> > >
> > > 1. Do we want to support validating issuers using the issuer uri?
> > >
> > >
> > > Are you referring to validating the JWT was issued by a known, configured
> > > issuer, or something more different/more dynamic?
> > >
> > > The current design allows for optionally requiring that the iss claim
> > > from the JWT match a statically configured issuer from the configuration.
> > > See expectedIssuer under the broker configuration.
> > >
> > > 2. Can the access token be reused for multiple connections from the same
> > > client?
> > >
> > >
> > > That's an excellent question - I will double-check that it is reused.
> > > Hopefully the existing client authentication mechanism supports that
> > level
> > > of reuse.
> > >
> > > 3. Do we support configuring separate ssl configs for connecting
> > > authorization server/jwks endpoint?
> > >
> > >
> > > Yes, that documentation is forthcoming soon.
> > >
> > > It will be a set of configuration options similar to the existing SSL
> > > socket configuration, but specific to the HTTPS endpoint for the
> > OAuth/OIDC
> > > provider connections.
> > >
> > > 4. Do we want support any readable username as principal if it is present
> > > in the token claims
> > >
> > >
> > > Yes. See the subClaimName and principalClaimName configuration options.
> > > Those will allow specifying a claim name other than sub for the
> > principal.
> > >
> > > Now that I'm writing this out I realize that the configuration names are
> > > different on the client and broker for some reason 🤔  I'll change that.
> > >
> > > 5. I agree with Ron, We should move the public classes to
> > > "o.a.k.c.s.oauthbearer.secured" package.
> > >
> > >
> > > Sounds good. I made the change in the KIP.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > Thanks for your feedback!
> > >
> > >
> > > On Thu, Aug 19, 2021 at 11:46 PM Kirk True 
> > wrote:
> > >
> > > > Hi Ron,
> > > >
> > > > On Sat, Aug 14, 2021, at 11:27 AM, Ron Dagostino wrote:
> > > > > Hi Kirk -- thanks for the KIP!  Having concrete implementations
> > > > > out-of-the-box will be very helpful.
> > > > >
> > > > > > As seen in this diagram, the login callback is executed on the
> > client
> > > > and
> > > > > the validate callback is executed on the broker.
> > > > >
> > > > > There was no diagram when I looked.  Maybe there is a broken link or
> > > > > something?
> > > >
> > > > I double-checked and it's showing for me on the published version of
> > the
> > > > wiki, even after I've logged out.
> > > >
> > > > Would you mind checking again when you get the chance?
> > > >
> > > > > > The name of the implementation class will be
> > > > >
> > > >
> > >
> > org.apache.kafka.common.security.oauthbearer.internals.secured.OAuthBearerLoginCallbackHandler
> > > > >
> > > > > I think the internals package was meant for non-public stuff  Most of
> > > it
> > > > > seems that way, although the "unsecured" implementation is in there
> > --
> > > > but
> > > > > that's maybe a grey area since it isn't meant to be used in
> > production
> > > > > scenarios and is mostly leveraged in unit tests.  Perhaps move the
> > > > proposed
> > > > > class into a "o.a.k.c.s.oauthbearer.secured" package?  Then any
> > > > > implementation details beyond the public stuff can live under the
> > > > > "...internals.secured" package that you mentioned?  The same comment
> > > > > applies to the validator callback handler class.
> > > >
> > > > In a draft I had the secured package directly under the oauthbearer
> > > > package as you describe but I received some out-of-band feedback to aim
> > > for
> > > > parity with the unsecured package layout.
> > > >
> > > > I don't have a preference for either. I do agree that it seems weird
> > for
> > > a
> > > > package named internals to be used in configuration since its name
> > > implies
> > > > that things could change.
> > > >
> > > > > I'm confused by loginRetryMaxWaitMs and loginRetryWaitMs.  The former
> > > has
> > > > > "Max" in the name, but only the description of the latter mentions it
> > > > being
> > > > > a max amount of time?  Are the descriptions incorrect or perhaps
> > > > reversed?
> > > >
> > > > Yes. Thanks for catching that. I've added more description in a
> > sep

Re: [DISCUSS] KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

2021-09-23 Thread Kirk True

Hi Manikumar,

On Wed, Aug 25, 2021, at 8:54 PM, Manikumar wrote:
> Thanks for the reply,
> 
> Can we also update the KIP about the testing approach? 

Yes, I've added that as a dedicated section in the KIP:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575#KIP768:ExtendSASL/OAUTHBEARERwithSupportforOIDC-Testing

Thanks,
Kirk

> Thanks,
> 
> On Wed, Aug 25, 2021 at 12:01 AM Kirk True  wrote:
>> __
>> Hi Manikumar!
>> 
>> On Mon, Aug 23, 2021, at 12:59 PM, Manikumar wrote:
>>> Hi Kirk,
>>> 
>>> Thanks for the KIP!
>>> 
>>> 1. Do we want to support validating issuers using the issuer uri?
>> 
>> Are you referring to validating the JWT was issued by a known, configured 
>> issuer, or something more different/more dynamic?
>> 
>> The current design allows for optionally requiring that the iss claim from 
>> the JWT match a statically configured issuer from the configuration. See 
>> expectedIssuer under the broker configuration.
>> 
>>> 2. Can the access token be reused for multiple connections from the same
>>> client?
>> 
>> That's an excellent question - I will double-check that it is reused. 
>> Hopefully the existing client authentication mechanism supports that level 
>> of reuse.
>> 
>>> 3. Do we support configuring separate ssl configs for connecting
>>> authorization server/jwks endpoint?
>> 
>> Yes, that documentation is forthcoming soon.
>> 
>> It will be a set of configuration options similar to the existing SSL socket 
>> configuration, but specific to the HTTPS endpoint for the OAuth/OIDC 
>> provider connections.
>> 
>>> 4. Do we want support any readable username as principal if it is present
>>> in the token claims
>> 
>> Yes. See the subClaimName and principalClaimName configuration options. 
>> Those will allow specifying a claim name other than sub for the principal.
>> 
>> Now that I'm writing this out I realize that the configuration names are 
>> different on the client and broker for some reason 🤔  I'll change that.
>> 
>>> 5. I agree with Ron, We should move the public classes to
>>> "o.a.k.c.s.oauthbearer.secured" package.
>> 
>> Sounds good. I made the change in the KIP.
>> 
>>> Thanks,
>>> Manikumar
>> 
>> Thanks for your feedback!
>> 
>>> 
>>> On Thu, Aug 19, 2021 at 11:46 PM Kirk True  wrote:
>>> 
>>> > Hi Ron,
>>> >
>>> > On Sat, Aug 14, 2021, at 11:27 AM, Ron Dagostino wrote:
>>> > > Hi Kirk -- thanks for the KIP!  Having concrete implementations
>>> > > out-of-the-box will be very helpful.
>>> > >
>>> > > > As seen in this diagram, the login callback is executed on the client
>>> > and
>>> > > the validate callback is executed on the broker.
>>> > >
>>> > > There was no diagram when I looked.  Maybe there is a broken link or
>>> > > something?
>>> >
>>> > I double-checked and it's showing for me on the published version of the
>>> > wiki, even after I've logged out.
>>> >
>>> > Would you mind checking again when you get the chance?
>>> >
>>> > > > The name of the implementation class will be
>>> > >
>>> > org.apache.kafka.common.security.oauthbearer.internals.secured.OAuthBearerLoginCallbackHandler
>>> > >
>>> > > I think the internals package was meant for non-public stuff  Most of it
>>> > > seems that way, although the "unsecured" implementation is in there --
>>> > but
>>> > > that's maybe a grey area since it isn't meant to be used in production
>>> > > scenarios and is mostly leveraged in unit tests.  Perhaps move the
>>> > proposed
>>> > > class into a "o.a.k.c.s.oauthbearer.secured" package?  Then any
>>> > > implementation details beyond the public stuff can live under the
>>> > > "...internals.secured" package that you mentioned?  The same comment
>>> > > applies to the validator callback handler class.
>>> >
>>> > In a draft I had the secured package directly under the oauthbearer
>>> > package as you describe but I received some out-of-band feedback to aim 
>>> > for
>>> > parity with the unsecured package layout.
>>> >
>>> > I don't have a preference for either. I do agree that it seems weird for a
>>> > package named internals to be used in configuration since its name implies
>>> > that things could change.
>>> >
>>> > > I'm confused by loginRetryMaxWaitMs and loginRetryWaitMs.  The former 
>>> > > has
>>> > > "Max" in the name, but only the description of the latter mentions it
>>> > being
>>> > > a max amount of time?  Are the descriptions incorrect or perhaps
>>> > reversed?
>>> >
>>> > Yes. Thanks for catching that. I've added more description in a separate
>>> > paragraph above the enumerated configurations.
>>> >
>>> > > >  Ensure the encoding algorithm isn't none and matches what the 
>>> > > > expected
>>> > > algorithm expecting
>>> > >
>>> > > "expected algorithm expecting" some kind of grammar issue?
>>> >
>>> > Haha! Yes - thanks for catching that too!
>>> >
>>> > It now reads:
>>> >
>>> > > Ensure the encoding algorithm (`alg` from the header) isn't `none` and
>>> > matches the expected algorithm for the JWK ID
>>> >
>>> > >

[jira] [Resolved] (KAFKA-13315) log layer exception during shutdown that caused an unclean shutdown

2021-09-23 Thread Jun Rao (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-13315.
-
Fix Version/s: 3.0.1
   3.1.0
   Resolution: Fixed

merged the PR to trunk and 3.0.

> log layer exception during shutdown that caused an unclean shutdown
> ---
>
> Key: KAFKA-13315
> URL: https://issues.apache.org/jira/browse/KAFKA-13315
> Project: Kafka
>  Issue Type: Bug
>Reporter: Cong Ding
>Assignee: Cong Ding
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
>
> We have seen an exception caused by shutting down scheduler before shutting 
> down LogManager.
> When LogManager was closing partitons one by one, scheduler called to delete 
> old segments due to retention. However, the old segments could have been 
> closed by the LogManager, which subsequently marked logdir as offline and 
> didn't write the clean shutdown marker. Ultimately the broker would take 
> hours to restart.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-13070) LogManager shutdown races with periodic work scheduled by the instance

2021-09-23 Thread Jun Rao (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-13070.
-
Resolution: Duplicate

> LogManager shutdown races with periodic work scheduled by the instance
> --
>
> Key: KAFKA-13070
> URL: https://issues.apache.org/jira/browse/KAFKA-13070
> Project: Kafka
>  Issue Type: Bug
>Reporter: Kowshik Prakasam
>Assignee: Cong Ding
>Priority: Major
>
> In the LogManager shutdown sequence (in LogManager.shutdown()), we don't 
> cancel the periodic work scheduled by it prior to shutdown. As a result, the 
> periodic work could race with the shutdown sequence causing some unwanted 
> side effects. This is reproducible by a unit test in LogManagerTest.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #493

2021-09-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 424459 lines...]
[2021-09-24T01:29:17.585Z] [INFO] 
[2021-09-24T01:29:17.585Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart ---
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart ---
[2021-09-24T01:29:18.629Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.1.0-SNAPSHOT/streams-quickstart-3.1.0-SNAPSHOT.pom
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --< 
org.apache.kafka:streams-quickstart-java >--
[2021-09-24T01:29:18.629Z] [INFO] Building streams-quickstart-java 
3.1.0-SNAPSHOT[2/2]
[2021-09-24T01:29:18.629Z] [INFO] --[ maven-archetype 
]---
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart-java ---
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart-java ---
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ streams-quickstart-java ---
[2021-09-24T01:29:18.629Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-09-24T01:29:18.629Z] [INFO] Copying 6 resources
[2021-09-24T01:29:18.629Z] [INFO] Copying 3 resources
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ streams-quickstart-java ---
[2021-09-24T01:29:18.629Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-09-24T01:29:18.629Z] [INFO] Copying 2 resources
[2021-09-24T01:29:18.629Z] [INFO] Copying 3 resources
[2021-09-24T01:29:18.629Z] [INFO] 
[2021-09-24T01:29:18.629Z] [INFO] --- maven-archetype-plugin:2.2:jar 
(default-jar) @ streams-quickstart-java ---
[2021-09-24T01:29:19.675Z] [INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.1.0-SNAPSHOT
[2021-09-24T01:29:19.675Z] [INFO] 
[2021-09-24T01:29:19.675Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart-java ---
[2021-09-24T01:29:19.675Z] [INFO] 
[2021-09-24T01:29:19.675Z] [INFO] --- 
maven-archetype-plugin:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[2021-09-24T01:29:19.675Z] [INFO] 
[2021-09-24T01:29:19.675Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart-java ---
[2021-09-24T01:29:19.675Z] [INFO] 
[2021-09-24T01:29:19.675Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart-java ---
[2021-09-24T01:29:19.675Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.1.0-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.1.0-SNAPSHOT/streams-quickstart-java-3.1.0-SNAPSHOT.jar
[2021-09-24T01:29:19.675Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.1.0-SNAPSHOT/streams-quickstart-java-3.1.0-SNAPSHOT.pom
[2021-09-24T01:29:19.675Z] [INFO] 
[2021-09-24T01:29:19.675Z] [INFO] --- 
maven-archetype-plugin:2.2:update-local-catalog (default-update-local-catalog) 
@ streams-quickstart-java ---
[2021-09-24T01:29:19.675Z] [INFO] 

[2021-09-24T01:29:19.675Z] [INFO] Reactor Summary for Kafka Streams :: 
Quickstart 3.1.0-SNAPSHOT:
[2021-09-24T01:29:19.675Z] [INFO] 
[2021-09-24T01:29:19.675Z] [INFO] Kafka Streams :: Quickstart 
 SUCCESS [  2.008 s]
[2021-09-24T01:29:19.675Z] [INFO] streams-quickstart-java 
 SUCCESS [  0.988 s]
[2021-09-24T01:29:19.675Z] [INFO] 

[2021-09-24T01:29:19.675Z] [INFO] BUILD SUCCESS
[2021-09-24T01:29:19.675Z] [INFO] 

[2021-09-24T01:29:19.675Z] [INFO] Total time:  3.383 s
[2021-09-24T01:29:19.675Z] [INFO] Finished at: 2021-09-24T01:29:18Z
[2021-09-24T01:29:19.675Z] [INFO] 

[Pipeline] dir
[2021-09-24T01:29:20.371Z] Running in 
/home/jenkins/workspace/Kafka

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #139

2021-09-23 Thread Apache Jenkins Server

See

Building 3.0.0 and Task :connect:signMavenJavaPublication FAILED

Re: Building 3.0.0 and Task :connect:signMavenJavaPublication FAILED

[jira] [Resolved] (KAFKA-10544) Convert KTable aggregations to new PAPI

GC pressure of the java client

[jira] [Created] (KAFKA-13320) Suggestion: SMT support for null key/value should be documented

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #491

[jira] [Created] (KAFKA-13321) Notify listener of leader change on registration

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #492

Re: [DISCUSS] KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

Re: [DISCUSS] KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

[jira] [Resolved] (KAFKA-13315) log layer exception during shutdown that caused an unclean shutdown

[jira] [Resolved] (KAFKA-13070) LogManager shutdown races with periodic work scheduled by the instance

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #493

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #139

14 matches

Site Navigation

Mail list logo

Footer information