[GitHub] [kafka] cmccabe commented on pull request #11649: KAFKA-13646: Implement KIP-801: KRaft authorizer

[GitBox]

cmccabe commented on pull request #11649:
URL: https://github.com/apache/kafka/pull/11649#issuecomment-1033469240


   Test failures are not related


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (KAFKA-13422) Even if the correct username and password are configured, when ClientBroker or KafkaClient tries to establish a SASL connection to ServerBroker, an exception is thrown

[RivenSun (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489372#comment-17489372
 ] 

RivenSun commented on KAFKA-13422:
--

Hi [~rsivaram]  [~ijuma] , [~guozhang] 

can you give any suggestions?
Thanks.

> Even if the correct username and password are configured, when ClientBroker 
> or KafkaClient tries to establish a SASL connection to ServerBroker, an 
> exception is thrown: (Authentication failed: Invalid username or password)
> --
>
> Key: KAFKA-13422
> URL: https://issues.apache.org/jira/browse/KAFKA-13422
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 2.7.1, 3.0.0
>Reporter: RivenSun
>Priority: Major
> Attachments: CustomerAuthCallbackHandler.java, 
> LoginContext_login_debug.png, SaslClientCallbackHandler_handle_debug.png
>
>
>  
> h1. Foreword:
> When deploying a Kafka cluster with a higher version (2.7.1), I encountered 
> an exception of communication identity authentication failure between 
> brokers. In the current latest version 3.0.0, this problem can also be 
> reproduced.
> h1. Problem recurring:
> h2. 1）broker Version is 3.0.0
> h3. The content of kafka_server_jaas.conf of each broker is exactly the same, 
> the content is as follows：
>  
>  
> {code:java}
> KafkaServer {
>   org.apache.kafka.common.security.plain.PlainLoginModule required
>   username="admin"
>   password="kJTVDziatPgjXG82sFHc4O1EIuewmlvS"
>   user_admin="kJTVDziatPgjXG82sFHc4O1EIuewmlvS"
>   user_alice="alice";
>   org.apache.kafka.common.security.scram.ScramLoginModule required
>   username="admin_scram"
>   password="admin_scram_password";
>  
> };
> {code}
>  
>  
> h3. broker server.properties：
> One of the broker configuration files is provided, and the content of the 
> configuration files of other brokers is only different from the localPublicIp 
> of advertised.listeners.
>  
> {code:java}
> broker.id=1
> broker.rack=us-east-1a
> advertised.listeners=SASL_PLAINTEXT://localPublicIp:9779,SASL_SSL://localPublicIp:9889,INTERNAL_SSL://:9009,PLAIN_PLUGIN_SSL://localPublicIp:9669
> log.dirs=/asyncmq/kafka/data_1,/asyncmq/kafka/data_2
> zookeeper.connect=***
> listeners=SASL_PLAINTEXT://:9779,SASL_SSL://:9889,INTERNAL_SSL://:9009,PLAIN_PLUGIN_SSL://:9669
> listener.security.protocol.map=INTERNAL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAIN_PLUGIN_SSL:SASL_SSL
> listener.name.plain_plugin_ssl.plain.sasl.server.callback.handler.class=org.apache.kafka.common.security.plain.internals.PlainServerCallbackHandler
> #ssl config
> ssl.keystore.password=***
> ssl.key.password=***
> ssl.truststore.password=***
> ssl.keystore.location=***
> ssl.truststore.location=***
> ssl.client.auth=none
> ssl.endpoint.identification.algorithm=
> #broker communicate config
> #security.inter.broker.protocol=SASL_PLAINTEXT
> inter.broker.listener.name=INTERNAL_SSL
> sasl.mechanism.inter.broker.protocol=PLAIN
> #sasl authentication config
> sasl.kerberos.service.name=kafka
> sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,GSSAPI
> delegation.token.master.key=***
> delegation.token.expiry.time.ms=8640
> delegation.token.max.lifetime.ms=31536
> {code}
>  
>  
> Then start all brokers at the same time. Each broker has actually been 
> started successfully, but when establishing a connection between the 
> controller node and all brokers, the identity authentication has always 
> failed. The connection between brokers cannot be established normally, 
> causing the entire Kafka cluster to be unable to provide external services.
> h3. The server log keeps printing abnormally like crazy：
> The real ip sensitive information of the broker in the log, I use ** 
> instead of here
>  
> {code:java}
> [2021-10-29 14:16:19,831] INFO [SocketServer listenerType=ZK_BROKER, 
> nodeId=3] Started socket server acceptors and processors 
> (kafka.network.SocketServer)
> [2021-10-29 14:16:19,836] INFO Kafka version: 3.0.0 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,836] INFO Kafka commitId: 8cb0a5e9d3441962 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,836] INFO Kafka startTimeMs: 1635516979831 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,837] INFO [KafkaServer id=3] started 
> (kafka.server.KafkaServer)
> [2021-10-29 14:16:20,249] INFO [SocketServer listenerType=ZK_BROKER, 
> nodeId=3] Failed authentication with /** (Authentication failed: Invalid 
> username or password) (org.apache.kafka.common.network.Selector)
> [2021-10-29 14:16:20,680] IN

[GitHub] [kafka] dajac commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802417366



##
File path: 
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java
##
@@ -696,8 +698,12 @@ public void handle(JoinGroupResponse joinResponse, 
RequestFuture fut
 private RequestFuture onJoinLeader(JoinGroupResponse 
joinResponse) {
 try {
 // perform the leader synchronization and send back the assignment 
for the group
-Map groupAssignment = 
performAssignment(joinResponse.data().leader(), 
joinResponse.data().protocolName(),
-joinResponse.data().members());
+Map groupAssignment = performAssignment(
+joinResponse.data().leader(),
+joinResponse.data().protocolName(),
+joinResponse.data().members(),
+joinResponse.data().skipAssignment()

Review comment:
   That makes sense. I have renamed the method to avoid the confusion and 
updated its javadoc.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dajac commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802450152



##
File path: 
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinatorTest.java
##
@@ -386,15 +386,39 @@ public void 
testPerformAssignmentShouldValidateCooperativeAssignment() {
 if (protocol == COOPERATIVE) {
 // in cooperative protocol, we should throw exception when 
validating cooperative assignment
 Exception e = assertThrows(IllegalStateException.class,
-() -> coordinator.performAssignment("1", 
partitionAssignor.name(), metadata));
+() -> coordinator.performAssignment("1", 
partitionAssignor.name(), metadata, false));
 assertTrue(e.getMessage().contains("Assignor supporting the 
COOPERATIVE protocol violates its requirements"));
 } else {
 // in eager protocol, we should not validate assignment
-coordinator.performAssignment("1", partitionAssignor.name(), 
metadata);
+coordinator.performAssignment("1", partitionAssignor.name(), 
metadata, false);
 }
 }
 }
 
+@Test
+public void testPerformAssignmentShouldSkipAssignment() {
+SubscriptionState mockSubscriptionState = 
Mockito.mock(SubscriptionState.class);
+
+Map> memberSubscriptions = 
singletonMap(consumerId, singletonList(topic1));
+
+List metadata = new 
ArrayList<>();
+for (Map.Entry> subscriptionEntry : 
memberSubscriptions.entrySet()) {
+ConsumerPartitionAssignor.Subscription subscription = new 
ConsumerPartitionAssignor.Subscription(subscriptionEntry.getValue());
+ByteBuffer buf = 
ConsumerProtocol.serializeSubscription(subscription);
+metadata.add(new JoinGroupResponseData.JoinGroupResponseMember()
+.setMemberId(subscriptionEntry.getKey())
+.setMetadata(buf.array()));
+}
+
+// `partitionAssignor.prepare` is not called therefore calling 
`partitionAssignor.assign` will throw

Review comment:
   That method is part of `MockPartitionAssignor` which is extensively used 
in tests in this suite. I do agree with you than using a mock is more explicit 
here. Let me update that.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dajac commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802484108



##
File path: 
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinatorTest.java
##
@@ -1550,6 +1573,34 @@ public void testMetadataChangeTriggersRebalance() {
 assertTrue(coordinator.rejoinNeededOrPending());
 }
 
+@Test
+public void testStaticLeaderRejoinsGroupAndCanTriggersRebalance() {
+// ensure metadata is up-to-date for leader
+subscriptions.subscribe(singleton(topic1), rebalanceListener);
+client.updateMetadata(metadataResponse);
+
+client.prepareResponse(groupCoordinatorResponse(node, Errors.NONE));
+coordinator.ensureCoordinatorReady(time.timer(Long.MAX_VALUE));
+
+// the leader is responsible for picking up metadata changes and 
forcing a group rebalance.
+// note that `partitionAssignor.prepare` is not called therefore 
calling `partitionAssignor.assign`
+// will throw a IllegalStateException. this indirectly verifies that 
`assign` is correctly skipped.
+Map> memberSubscriptions = 
singletonMap(consumerId, singletonList(topic1));

Review comment:
   Sure. Let me add another test.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (KAFKA-13658) Upgrade vulnerable dependencies jan 2022

[Dominique Mongelli (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489429#comment-17489429
 ] 

Dominique Mongelli commented on KAFKA-13658:


Hi,

For information here is the github issue on jackson repo: 
[https://github.com/FasterXML/jackson-databind/issues/3328.]

Snyk rates the vulnerability as 5.9: 
[https://security.snyk.io/vuln/SNYK-JAVA-COMFASTERXMLJACKSONCORE-2326698.]

For info, the vulnerability is applicable only when using JDK 
serialization/deserialization of JsonNode.

as a first step, could you tell us if kafka is actually using this kind of 
serialization/deserialization ?

Thanks

> Upgrade vulnerable dependencies jan 2022
> 
>
> Key: KAFKA-13658
> URL: https://issues.apache.org/jira/browse/KAFKA-13658
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Shivakumar
>Assignee: Luke Chen
>Priority: Major
>  Labels: secutiry
>
> |Packages|Package Version|CVSS|Fix Status|
> |com.fasterxml.jackson.core_jackson-databind| 2.10.5.1| 7.5| fixed in 2.14, 
> 2.13.1, 2.12.6|
> | | | | |
> Our security scan detected the above vulnerabilities
> upgrade to correct versions for fixing vulnerabilities



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka] dajac commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802514285



##
File path: 
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
##
@@ -655,11 +667,8 @@ private void maybeUpdateGroupSubscription(String 
assignorName,
 
 maybeUpdateGroupSubscription(assignorName, assignments, 
allSubscribedTopics);
 
-assignmentSnapshot = metadataSnapshot;

Review comment:
   Yeah, it seems that you are right. I updated the PR as you suggested.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dajac commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802515290



##
File path: core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala
##
@@ -1791,4 +1793,33 @@ class PlaintextConsumerTest extends BaseConsumerTest {
 assertTrue(records2.count() == 1 && 
records2.records(tp).asScala.head.offset == 1,
   "Expected consumer2 to consume one message from offset 1, which is the 
committed offset of consumer1")
   }
+  @Test
+  def testStaticConsumerDetectsNewPartitionCreatedAfterRestart(): Unit = {

Review comment:
   Well.. We can't really restart a consumer, right? The only way is to 
recreate the consumer. What I meant here is that the static member is restarted.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dajac commented on pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#issuecomment-1033635997


   @hachikuji @showuon Thanks for reviewing. I have updated the PR based on 
your comments. I have also bumped to version of the JoinGroup API to v9. I 
thought that we could lean on v8 which is new in 3.2 but it seems safer to bump 
it again just in case someone would run with trunk.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] FireBurn commented on pull request #11685: Update dependencies.gradle

[GitBox]

FireBurn commented on pull request #11685:
URL: https://github.com/apache/kafka/pull/11685#issuecomment-1033636049


   Can we replace with reload4j 
   
   https://reload4j.qos.ch/ it fixes the bugs of log4j1, is written by the same 
developer and doesn't have api changes like log4j2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-13616) Log4j 1.X CVE-2022-23302/5/7 vulnerabilities

[Dongjin Lee (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjin Lee resolved KAFKA-13616.
-
Resolution: Duplicate

> Log4j 1.X CVE-2022-23302/5/7 vulnerabilities
> 
>
> Key: KAFKA-13616
> URL: https://issues.apache.org/jira/browse/KAFKA-13616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dominique Mongelli
>Priority: Major
>
> Some log4j 1.x vulnerabilities have been disclosed recently:   
>  * CVE-2022-23302: https://nvd.nist.gov/vuln/detail/CVE-2022-23302    
>  * CVE-2022-23305 : https://nvd.nist.gov/vuln/detail/CVE-2022-23305    
>  * CVE-2022-23307 : [https://nvd.nist.gov/vuln/detail/CVE-2022-23307]
> We would like to know if kafka is affected by these vulnerabilities ?
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka] tombentley commented on a change in pull request #11672: KAFKA-13577: Replace easymock with mockito in kafka:core - part 1

[GitBox]

tombentley commented on a change in pull request #11672:
URL: https://github.com/apache/kafka/pull/11672#discussion_r802574591



##
File path: core/src/test/scala/kafka/common/InterBrokerSendThreadTest.scala
##
@@ -134,36 +139,45 @@ class InterBrokerSendThreadTest {
 
 val clientRequest = new ClientRequest("dest", request, 0, "1", 0, true, 
requestTimeoutMs, handler.handler)
 
-EasyMock.expect(networkClient.newClientRequest(
-  EasyMock.eq("1"),
-  EasyMock.same(handler.request),
-  EasyMock.anyLong(),
-  EasyMock.eq(true),
-  EasyMock.eq(requestTimeoutMs),
-  EasyMock.same(handler.handler)))
-  .andReturn(clientRequest)
+when(networkClient.newClientRequest(
+  ArgumentMatchers.eq("1"),
+  same(handler.request),
+  anyLong(),
+  ArgumentMatchers.eq(true),
+  ArgumentMatchers.eq(requestTimeoutMs),
+  same(handler.handler)))
+  .thenReturn(clientRequest)
 
-EasyMock.expect(networkClient.ready(node, time.milliseconds()))
-  .andReturn(false)
+when(networkClient.ready(node, time.milliseconds()))
+  .thenReturn(false)
 
-EasyMock.expect(networkClient.connectionDelay(EasyMock.anyObject(), 
EasyMock.anyLong()))
-  .andReturn(0)
+when(networkClient.connectionDelay(any[Node], anyLong()))

Review comment:
   Ah, OK, no that makes sense. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-13659) MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset

[Kanalas Vidor (Jira)]

Kanalas Vidor created KAFKA-13659:
-

 Summary: MM2 should read all offset syncs at start up and should 
not set consumer offset higher than the end offset
 Key: KAFKA-13659
 URL: https://issues.apache.org/jira/browse/KAFKA-13659
 Project: Kafka
  Issue Type: Improvement
  Components: mirrormaker
Reporter: Kanalas Vidor


- MirrorCheckpointTask uses OffsetSyncStore, and does not check whether 
OffsetSyncStore managed to read to the "end" of the offset-syncs topic. 
OffsetSyncStore should fetch the endoffset of the topic at startup, and set a 
flag when it finally reaches the endoffset in consumption. 
MirrorCheckpointTask.poll should wait for this flag to be true before doing any 
in-memory updates and group offset management.
 - MirrorCheckpointTask can create checkpoints which point into the "future" - 
meaning it sometimes translates consumer offsets in a way that the target 
offset is greater than the endoffset of the replica topic partition. 
MirrorCheckpointTask should fetch the endoffsets of the affected topics, and 
make sure that it does not try to set the consumer offset to anything higher 
than the endoffset.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka] FireBurn opened a new pull request #11743: Switch log4j12 to reload4j

[GitBox]

FireBurn opened a new pull request #11743:
URL: https://github.com/apache/kafka/pull/11743


   This bumps the slf4j version to 1.7.36 and swaps out log4j 1.2.17 with
   reload4j 1.2.19
   
   Signed-off-by: Mike Lothian 
   
   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] FireBurn commented on pull request #11685: Update dependencies.gradle

[GitBox]

FireBurn commented on pull request #11685:
URL: https://github.com/apache/kafka/pull/11685#issuecomment-1033694596


   I've raised https://github.com/apache/kafka/pull/11743 to switch to reload4j


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] showuon commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

showuon commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802624600



##
File path: core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala
##
@@ -1791,4 +1793,33 @@ class PlaintextConsumerTest extends BaseConsumerTest {
 assertTrue(records2.count() == 1 && 
records2.records(tp).asScala.head.offset == 1,
   "Expected consumer2 to consume one message from offset 1, which is the 
committed offset of consumer1")
   }
+  @Test
+  def testStaticConsumerDetectsNewPartitionCreatedAfterRestart(): Unit = {

Review comment:
   Make sense. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] lmr3796 opened a new pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append

[GitBox]

lmr3796 opened a new pull request #11744:
URL: https://github.com/apache/kafka/pull/11744


   The Java doc for the thrown exception is added in (apache#4975)
   
https://github.com/apache/kafka/pull/4975/files#diff-f3714fa2bb7e07c857d2cafde9dcb5d310fafd2cceed9f4124cc6342671a2c89R137
   
   By the time it was already a typo.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] lmr3796 commented on pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append

[GitBox]

lmr3796 commented on pull request #11744:
URL: https://github.com/apache/kafka/pull/11744#issuecomment-1033729356


   Hi @chia7712 can I get a review from you for this minor patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] ijuma commented on pull request #9972: KAFKA-8779: Reintroduce flaky tests

[GitBox]

ijuma commented on pull request #9972:
URL: https://github.com/apache/kafka/pull/9972#issuecomment-1033748707


   @chia7712 Does this look good to you?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] showuon commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

showuon commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802633329



##
File path: 
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
##
@@ -648,10 +648,14 @@ private void maybeUpdateGroupSubscription(String 
assignorName,
 updateGroupSubscription(allSubscribedTopics);
 
 isLeader = true;
-assignmentSnapshot = metadataSnapshot;
 
-if (skipAssignment)
+

Review comment:
   nit: extra empty line here

##
File path: 
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinatorTest.java
##
@@ -1601,6 +1604,36 @@ public void 
testStaticLeaderRejoinsGroupAndCanTriggersRebalance() {
 assertTrue(coordinator.rejoinNeededOrPending());
 }
 
+@Test
+public void 
testStaticLeaderRejoinsGroupAndCanDetectMetadataChangesForOtherMembers() {

Review comment:
   nice new test added.

##
File path: 
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
##
@@ -648,10 +648,14 @@ private void maybeUpdateGroupSubscription(String 
assignorName,
 updateGroupSubscription(allSubscribedTopics);
 
 isLeader = true;
-assignmentSnapshot = metadataSnapshot;
 
-if (skipAssignment)
+
+if (skipAssignment) {
+log.info("Skipped assignment for returning static leader at 
generation {}. The static leader " +
+"will collect its existing assignment.", 
generation().generationId);

Review comment:
   I'm not sure if we need to put the 2nd sentence. Maybe the 1st one is 
enough? Or maybe change the 2nd one with:
   `The static leader will collect its existing assignment with empty 
assignment syncGroup request.` 
   
   WDYT?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison closed pull request #4934: KIP-81: KAFKA-4133: Bound memory usage of the Consumer

[GitBox]

mimaison closed pull request #4934:
URL: https://github.com/apache/kafka/pull/4934


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison closed pull request #6193: [WIP] KIP-81 Bound Fetch memory usage in the consumer

[GitBox]

mimaison closed pull request #6193:
URL: https://github.com/apache/kafka/pull/6193


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dajac commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)

[GitBox]

dajac commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r802733093



##
File path: 
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
##
@@ -648,10 +648,14 @@ private void maybeUpdateGroupSubscription(String 
assignorName,
 updateGroupSubscription(allSubscribedTopics);
 
 isLeader = true;
-assignmentSnapshot = metadataSnapshot;
 
-if (skipAssignment)
+
+if (skipAssignment) {
+log.info("Skipped assignment for returning static leader at 
generation {}. The static leader " +
+"will collect its existing assignment.", 
generation().generationId);

Review comment:
   I would not mention the sync group request. How about `The static leader 
will continue with its existing assignment`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] tombentley commented on a change in pull request #11642: MINOR: Improve Connect docs

[GitBox]

tombentley commented on a change in pull request #11642:
URL: https://github.com/apache/kafka/pull/11642#discussion_r802748846



##
File path: 
connect/runtime/src/main/java/org/apache/kafka/connect/tools/PredicateDoc.java
##
@@ -61,7 +61,7 @@ private static void printPredicateHtml(PrintStream out, 
DocInfo docInfo) {
 out.println("");
 
 out.print("");
-out.print(docInfo.predicateName);
+out.print("" + 
docInfo.predicateName + "");

Review comment:
   Are this really `href` not `name`? If so what are they linking to?

##
File path: docs/connect.html
##
@@ -414,53 +403,49 @@ Connector
 We'll cover the SourceConnector as a simple example. 
SinkConnector implementations are very similar. Start by creating 
the class that inherits from SourceConnector and add a couple of 
fields that will store parsed configuration information (the filename to read 
from and the topic to send data to):
 
 
-public class FileStreamSourceConnector extends SourceConnector {
-private String filename;
-private String topic;
-
+public class FileStreamSourceConnector extends SourceConnector {
+private String filename;
+private String topic;
 
 The easiest method to fill in is taskClass(), which 
defines the class that should be instantiated in worker processes to actually 
read the data:
 
 
-@Override
-public Class taskClass() {
-return FileStreamSourceTask.class;
-}
-
+@Override
+public Class taskClass() {
+return FileStreamSourceTask.class;
+}
 
 We will define the FileStreamSourceTask class below. Next, 
we add some standard lifecycle methods, start() and 
stop():
 
 
-@Override
-public void start(Map props) {
-// The complete version includes error handling as well.
-filename = props.get(FILE_CONFIG);
-topic = props.get(TOPIC_CONFIG);
-}
-
-@Override
-public void stop() {
-// Nothing to do since no background monitoring is required.
-}
-
+@Override
+public void start(Map props) {
+// The complete version includes error handling as well.
+filename = props.get(FILE_CONFIG);
+topic = props.get(TOPIC_CONFIG);
+}
+
+@Override
+public void stop() {
+// Nothing to do since no background monitoring is required.
+}
 
 Finally, the real core of the implementation is in 
taskConfigs(). In this case we are only
 handling a single file, so even though we may be permitted to generate 
more tasks as per the
 maxTasks argument, we return a list with only one entry:
 
 
-@Override
-public List> taskConfigs(int maxTasks) {
-ArrayList> configs = new 
ArrayList<>();
-// Only one input stream makes sense.
-Map config = new HashMap<>();
-if (filename != null)
-config.put(FILE_CONFIG, filename);
-config.put(TOPIC_CONFIG, topic);
-configs.add(config);
-return configs;
-}
-
+@Override

Review comment:
   Does putting this in a CDATA let us avoid the eye-bleeding HTML escaping?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dengziming commented on a change in pull request #11667: MINOR; Enable Kraft in ApiVersionTest

[GitBox]

dengziming commented on a change in pull request #11667:
URL: https://github.com/apache/kafka/pull/11667#discussion_r802758354



##
File path: 
core/src/test/scala/unit/kafka/server/AbstractApiVersionsRequestTest.scala
##
@@ -34,18 +36,25 @@ import scala.jdk.CollectionConverters._
 abstract class AbstractApiVersionsRequestTest(cluster: ClusterInstance) {
 
   def sendApiVersionsRequest(request: ApiVersionsRequest, listenerName: 
ListenerName): ApiVersionsResponse = {
-IntegrationTestUtils.connectAndReceive[ApiVersionsResponse](request, 
cluster.brokerSocketServers().asScala.head, listenerName)
+val socket = if (listenerName == controlPlaneListenerName) {
+  cluster.controllerSocketServers().asScala.head
+} else {
+  cluster.brokerSocketServers().asScala.head
+}
+IntegrationTestUtils.connectAndReceive[ApiVersionsResponse](request, 
socket, listenerName)
   }
 
   def controlPlaneListenerName = new ListenerName("CONTROLLER")

Review comment:
   This idea is better than having only one controllerListener,  I removed 
the `controllerListener` here and added the two methods in `ClusterInstance`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] dengziming commented on a change in pull request #11667: MINOR; Enable Kraft in ApiVersionTest

[GitBox]

dengziming commented on a change in pull request #11667:
URL: https://github.com/apache/kafka/pull/11667#discussion_r802760455



##
File path: 
core/src/test/scala/unit/kafka/server/AbstractApiVersionsRequestTest.scala
##
@@ -59,15 +68,40 @@ abstract class AbstractApiVersionsRequestTest(cluster: 
ClusterInstance) {
 } finally socket.close()
   }
 
-  def validateApiVersionsResponse(apiVersionsResponse: ApiVersionsResponse): 
Unit = {
-val expectedApis = ApiKeys.zkBrokerApis()
+  def validateApiVersionsResponse(apiVersionsResponse: ApiVersionsResponse, 
controllerApi: Boolean = false): Unit = {

Review comment:
   Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (KAFKA-13659) MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset

[Viktor Somogyi-Vass (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass reassigned KAFKA-13659:
---

Assignee: Kanalas Vidor

> MM2 should read all offset syncs at start up and should not set consumer 
> offset higher than the end offset
> --
>
> Key: KAFKA-13659
> URL: https://issues.apache.org/jira/browse/KAFKA-13659
> Project: Kafka
>  Issue Type: Improvement
>  Components: mirrormaker
>Reporter: Kanalas Vidor
>Assignee: Kanalas Vidor
>Priority: Major
>
> - MirrorCheckpointTask uses OffsetSyncStore, and does not check whether 
> OffsetSyncStore managed to read to the "end" of the offset-syncs topic. 
> OffsetSyncStore should fetch the endoffset of the topic at startup, and set a 
> flag when it finally reaches the endoffset in consumption. 
> MirrorCheckpointTask.poll should wait for this flag to be true before doing 
> any in-memory updates and group offset management.
>  - MirrorCheckpointTask can create checkpoints which point into the "future" 
> - meaning it sometimes translates consumer offsets in a way that the target 
> offset is greater than the endoffset of the replica topic partition. 
> MirrorCheckpointTask should fetch the endoffsets of the affected topics, and 
> make sure that it does not try to set the consumer offset to anything higher 
> than the endoffset.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-12635) Mirrormaker 2 offset sync is incorrect if the target partition is empty

[Mickael Maison (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489631#comment-17489631
 ] 

Mickael Maison commented on KAFKA-12635:


Looking at this again (sorry for the delay).

The offset on the target being negative should not have a functional impact on 
the consumer. The offset is "out of range" so the auto.offset.reset 
configuration will be used to find a new valid position. As there are no 
records in the target partition, whether the consumer resets to latest or 
earliest will have make no difference and it will set its position to 0.

But I understand it may be annoying in terms of metrics. I guess in theory it 
could also lead to records being skipped if suddenly records are produced to 
the source cluster and we start a consumer with auto.offset.reset to latest on 
the target cluster before MirrorMaker is able to emit a new checkpoint/commit 
offsets.

I think a better alternative than resetting the offset to 0 is to actually not 
commit any offsets in the target cluster until some records have been mirrored. 



> Mirrormaker 2 offset sync is incorrect if the target partition is empty
> ---
>
> Key: KAFKA-12635
> URL: https://issues.apache.org/jira/browse/KAFKA-12635
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0
>Reporter: Frank Yi
>Assignee: Ning Zhang
>Priority: Major
>
> This bug occurs when using Mirrormaker with "sync.group.offsets.enabled = 
> true".
> If a source partition is empty, but the source consumer group's offset for 
> that partition is non-zero, then Mirrormaker sets the target consumer group's 
> offset for that partition to the literal, not translated, offset of the 
> source consumer group. This state can be reached if the source consumer group 
> consumed some records that were now deleted (like by a retention policy), or 
> if Mirrormaker replication is set to start at "latest". This bug causes the 
> target consumer group's lag for that partition to be negative and breaks 
> offset sync for that partition until lag is positive.
> The correct behavior when the source partition is empty would be to set the 
> target offset to the translated offset, not literal offset, which in this 
> case would always be 0. 
> Original email thread on this issue: 
> https://lists.apache.org/thread.html/r7c54ee5f57227367b911d4abffa72781772d8dd3b72d75eb65ee19f7%40%3Cusers.kafka.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-13659) MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset

[Mickael Maison (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489633#comment-17489633
 ] 

Mickael Maison commented on KAFKA-13659:


Is this a DUP of https://issues.apache.org/jira/browse/KAFKA-12635 ?

Overall I agree with the proposed approach to not commit any offsets for a 
partition until some records have been mirrored.

> MM2 should read all offset syncs at start up and should not set consumer 
> offset higher than the end offset
> --
>
> Key: KAFKA-13659
> URL: https://issues.apache.org/jira/browse/KAFKA-13659
> Project: Kafka
>  Issue Type: Improvement
>  Components: mirrormaker
>Reporter: Kanalas Vidor
>Assignee: Kanalas Vidor
>Priority: Major
>
> - MirrorCheckpointTask uses OffsetSyncStore, and does not check whether 
> OffsetSyncStore managed to read to the "end" of the offset-syncs topic. 
> OffsetSyncStore should fetch the endoffset of the topic at startup, and set a 
> flag when it finally reaches the endoffset in consumption. 
> MirrorCheckpointTask.poll should wait for this flag to be true before doing 
> any in-memory updates and group offset management.
>  - MirrorCheckpointTask can create checkpoints which point into the "future" 
> - meaning it sometimes translates consumer offsets in a way that the target 
> offset is greater than the endoffset of the replica topic partition. 
> MirrorCheckpointTask should fetch the endoffsets of the affected topics, and 
> make sure that it does not try to set the consumer offset to anything higher 
> than the endoffset.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-13638) Slow KTable update when forwarding multiple values from transformer

[Ulrik (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489636#comment-17489636
 ] 

Ulrik commented on KAFKA-13638:
---

[~cadonna]  Very strange. I added the same code as you and got the same numbers 
as I said before.

Could probably be related to local setup, but not sure in what way

> Slow KTable update when forwarding multiple values from transformer
> ---
>
> Key: KAFKA-13638
> URL: https://issues.apache.org/jira/browse/KAFKA-13638
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Ulrik
>Priority: Major
> Attachments: KafkaTest.java
>
>
> I have a topology where I stream messages from an input topic, transform the 
> message to multiple messages (via context.forward), and then store those 
> messages in a KTable.
> Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my 
> tests take significantly longer time to run. 
>  
> I have attached a test class to demonstrate my scenario. When running this 
> test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following 
> numbers:
>  
> *Version 2.8.1*
>  * one input message and one output message: 541 ms
>  * 8 input message and 30 output message per input message (240 output 
> messages in total): 919 ms
>  
> *Version 3.1.0*
>  * one input message and one output message: 908 ms
>  * 8 input message and 30 output message per input message (240 output 
> messages in total): 6 sec 94 ms
>  
> Even when the transformer just transforms and forwards one input message to 
> one output message, the test takes approx. 400 ms longer to run.
> When transforming 8 input messages to 240 output messages it takes approx 5 
> seconds longer.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka] dajac commented on a change in pull request #11742: KAFKA-13636: Fix for the group coordinator issue where the offsets are deleted for unstable groups

[GitBox]

dajac commented on a change in pull request #11742:
URL: https://github.com/apache/kafka/pull/11742#discussion_r802805279



##
File path: 
core/src/test/scala/unit/kafka/coordinator/group/GroupMetadataTest.scala
##
@@ -259,6 +259,47 @@ class GroupMetadataTest {
 assertFalse(group.supportsProtocols(protocolType, Set("range", "foo")))
   }
 
+  @Test
+  def testOffsetRemovalDuringTransitionFromEmptyToNonEmpty(): Unit = {
+val topic = "foo"
+val partition = new TopicPartition(topic, 0)
+val time = new MockTime()

Review comment:
   `group` as defined in the `setUp` method uses `Time.SYSTEM`. It 
basically means that we are not using the same clock across the unit test. We 
should probably create a `GroupMetadata` in this test directly. I suppose that 
this causes the failing tests.

##
File path: core/src/main/scala/kafka/coordinator/group/GroupMetadata.scala
##
@@ -763,7 +763,7 @@ private[group] class GroupMetadata(val groupId: String, 
initialState: GroupState
 }
 
 val expiredOffsets: Map[TopicPartition, OffsetAndMetadata] = protocolType 
match {
-  case Some(_) if is(Empty) =>
+  case Some(_) if is(Empty) || !is(Stable)=>

Review comment:
   That does not seem right to me. You are basically saying that it is fine 
to consider offsets for expiration while the group is rebalancing (not stable). 
I say consider because in practice the offsets would be protected by 
`currentStateTimestamp` which is set when the group transition. Would it make 
sense to add `is(Stable)` to the second `case` at L777 or is there an issue 
with this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison commented on a change in pull request #11167: Kafka-13158 Replace EasyMock and PowerMock with Mockito for ConnectClusterStateImpl Test and ConnectorPluginsResourceTest

[GitBox]

mimaison commented on a change in pull request #11167:
URL: https://github.com/apache/kafka/pull/11167#discussion_r802807309



##
File path: build.gradle
##
@@ -2373,7 +2373,7 @@ project(':connect:runtime') {
 testImplementation libs.junitVintageEngine
 testImplementation libs.powermockJunit4
 testImplementation libs.powermockEasymock
-testImplementation libs.mockitoCore
+testImplementation libs.mockitoInline // supports mocking static methods, 
final classes, etc.

Review comment:
   Why are we switching to `mockitoInline`? We don't seem to use any of 
these feature in this PR and according to the Mockito project, inline may be 
removed in the future when all of its features are integrated in core.

##
File path: 
connect/runtime/src/test/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorPluginsResourceTest.java
##
@@ -182,29 +174,24 @@
 @Mock
 private Plugins plugins;
 private ConnectorPluginsResource connectorPluginsResource;
+@Captor

Review comment:
   We don't use these annotations in the other tests. Do you think we 
should not use them here to keep tests consistent? WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison merged pull request #11672: KAFKA-13577: Replace easymock with mockito in kafka:core - part 1

[GitBox]

mimaison merged pull request #11672:
URL: https://github.com/apache/kafka/pull/11672


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-13660) Replace log4j with reload4j

[Mike Lothian (Jira)]

Mike Lothian created KAFKA-13660:


 Summary: Replace log4j with reload4j
 Key: KAFKA-13660
 URL: https://issues.apache.org/jira/browse/KAFKA-13660
 Project: Kafka
  Issue Type: Bug
  Components: logging
Affects Versions: 3.0.0, 2.4.0
Reporter: Mike Lothian


Kafka is using a known vulnerable version of log4j, the reload4j project was 
created by the code's original authors to address those issues. It is designed 
as a drop in replacement without any api changes

 

I've raised a merge request, replacing log4j with reload4j, slf4j-log4j12 with 
slf4j-reload4j and bumping the slf4j version

 

this is my first time contributing to the Kafka project and I'm not too 
familiar with the process, I'll go back and amend my PR with this issue number



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13660) Replace log4j with reload4j

[Mike Lothian (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Lothian updated KAFKA-13660:
-
Description: 
Kafka is using a known vulnerable version of log4j, the reload4j project was 
created by the code's original authors to address those issues. It is designed 
as a drop in replacement without any api changes

 

https://reload4j.qos.ch/

 

I've raised a merge request, replacing log4j with reload4j, slf4j-log4j12 with 
slf4j-reload4j and bumping the slf4j version

 

This is my first time contributing to the Kafka project and I'm not too 
familiar with the process, I'll go back and amend my PR with this issue number

  was:
Kafka is using a known vulnerable version of log4j, the reload4j project was 
created by the code's original authors to address those issues. It is designed 
as a drop in replacement without any api changes

 

I've raised a merge request, replacing log4j with reload4j, slf4j-log4j12 with 
slf4j-reload4j and bumping the slf4j version

 

this is my first time contributing to the Kafka project and I'm not too 
familiar with the process, I'll go back and amend my PR with this issue number


> Replace log4j with reload4j
> ---
>
> Key: KAFKA-13660
> URL: https://issues.apache.org/jira/browse/KAFKA-13660
> Project: Kafka
>  Issue Type: Bug
>  Components: logging
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Mike Lothian
>Priority: Major
>
> Kafka is using a known vulnerable version of log4j, the reload4j project was 
> created by the code's original authors to address those issues. It is 
> designed as a drop in replacement without any api changes
>  
> https://reload4j.qos.ch/
>  
> I've raised a merge request, replacing log4j with reload4j, slf4j-log4j12 
> with slf4j-reload4j and bumping the slf4j version
>  
> This is my first time contributing to the Kafka project and I'm not too 
> familiar with the process, I'll go back and amend my PR with this issue number



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka] mimaison commented on a change in pull request #11642: MINOR: Improve Connect docs

[GitBox]

mimaison commented on a change in pull request #11642:
URL: https://github.com/apache/kafka/pull/11642#discussion_r802920885



##
File path: docs/connect.html
##
@@ -414,53 +403,49 @@ Connector
 We'll cover the SourceConnector as a simple example. 
SinkConnector implementations are very similar. Start by creating 
the class that inherits from SourceConnector and add a couple of 
fields that will store parsed configuration information (the filename to read 
from and the topic to send data to):
 
 
-public class FileStreamSourceConnector extends SourceConnector {
-private String filename;
-private String topic;
-
+public class FileStreamSourceConnector extends SourceConnector {
+private String filename;
+private String topic;
 
 The easiest method to fill in is taskClass(), which 
defines the class that should be instantiated in worker processes to actually 
read the data:
 
 
-@Override
-public Class taskClass() {
-return FileStreamSourceTask.class;
-}
-
+@Override
+public Class taskClass() {
+return FileStreamSourceTask.class;
+}
 
 We will define the FileStreamSourceTask class below. Next, 
we add some standard lifecycle methods, start() and 
stop():
 
 
-@Override
-public void start(Map props) {
-// The complete version includes error handling as well.
-filename = props.get(FILE_CONFIG);
-topic = props.get(TOPIC_CONFIG);
-}
-
-@Override
-public void stop() {
-// Nothing to do since no background monitoring is required.
-}
-
+@Override
+public void start(Map props) {
+// The complete version includes error handling as well.
+filename = props.get(FILE_CONFIG);
+topic = props.get(TOPIC_CONFIG);
+}
+
+@Override
+public void stop() {
+// Nothing to do since no background monitoring is required.
+}
 
 Finally, the real core of the implementation is in 
taskConfigs(). In this case we are only
 handling a single file, so even though we may be permitted to generate 
more tasks as per the
 maxTasks argument, we return a list with only one entry:
 
 
-@Override
-public List> taskConfigs(int maxTasks) {
-ArrayList> configs = new 
ArrayList<>();
-// Only one input stream makes sense.
-Map config = new HashMap<>();
-if (filename != null)
-config.put(FILE_CONFIG, filename);
-config.put(TOPIC_CONFIG, topic);
-configs.add(config);
-return configs;
-}
-
+@Override

Review comment:
   I'm not familiar with CDATA but from a quick search it seems it 
shouldn't be used in HTML documents.
   https://developer.mozilla.org/en-US/docs/Web/API/CDATASection states:
   ```
   Note: CDATA sections should not be used within HTML they are considered as 
comments and not displayed.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison commented on a change in pull request #11642: MINOR: Improve Connect docs

[GitBox]

mimaison commented on a change in pull request #11642:
URL: https://github.com/apache/kafka/pull/11642#discussion_r802932268



##
File path: 
connect/runtime/src/main/java/org/apache/kafka/connect/tools/PredicateDoc.java
##
@@ -61,7 +61,7 @@ private static void printPredicateHtml(PrintStream out, 
DocInfo docInfo) {
 out.println("");
 
 out.print("");
-out.print(docInfo.predicateName);
+out.print("" + 
docInfo.predicateName + "");

Review comment:
   It's linking to the existing `div id=` just above online 61




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison commented on a change in pull request #11642: MINOR: Improve Connect docs

[GitBox]

mimaison commented on a change in pull request #11642:
URL: https://github.com/apache/kafka/pull/11642#discussion_r802932268



##
File path: 
connect/runtime/src/main/java/org/apache/kafka/connect/tools/PredicateDoc.java
##
@@ -61,7 +61,7 @@ private static void printPredicateHtml(PrintStream out, 
DocInfo docInfo) {
 out.println("");
 
 out.print("");
-out.print(docInfo.predicateName);
+out.print("" + 
docInfo.predicateName + "");

Review comment:
   It's linking to the existing `` just above on line 61




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] hachikuji commented on a change in pull request #11655: KAFKA-13316; Enable KRaft mode in CreateTopics tests

[GitBox]

hachikuji commented on a change in pull request #11655:
URL: https://github.com/apache/kafka/pull/11655#discussion_r802933225



##
File path: 
core/src/test/scala/unit/kafka/server/CreateTopicsRequestWithPolicyTest.scala
##
@@ -55,11 +63,9 @@ class CreateTopicsRequestWithPolicyTest extends 
AbstractCreateTopicsRequestTest
   assignment = Map(0 -> List(1, 0), 1 -> List(0, 1))
   }
 
-  @Test
-  def testErrorCreateTopicsRequests(): Unit = {
-val existingTopic = "existing-topic"
-createTopic(existingTopic, 1, 1)

Review comment:
   Do we need to? Seems like the main thing we're testing here is the case 
when the topic already exists. Seems like it would be just as good to create 
with the admin client and then verify the `TOPIC_ALREADY_EXISTS` error code.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (KAFKA-13159) Enable system tests for transactions in KRaft mode

[Mickael Maison (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489704#comment-17489704
 ] 

Mickael Maison commented on KAFKA-13159:


[~mumrah] I see https://github.com/apache/kafka/pull/11166 has been merged 
already, can we close this ticket?

> Enable system tests for transactions in KRaft mode
> --
>
> Key: KAFKA-13159
> URL: https://issues.apache.org/jira/browse/KAFKA-13159
> Project: Kafka
>  Issue Type: Test
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Critical
> Fix For: 3.0.1, 3.2.0
>
>
> Previously, we disabled several system tests involving system tests in KRaft 
> mode. Now that KIP-730 is complete and transactions work in KRaft, we need to 
> re-enable these tests.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-12622) Automate LICENSE file validation

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-12622:
---
Fix Version/s: (was: 3.0.1)

> Automate LICENSE file validation
> 
>
> Key: KAFKA-12622
> URL: https://issues.apache.org/jira/browse/KAFKA-12622
> Project: Kafka
>  Issue Type: Task
>Reporter: John Roesler
>Priority: Major
> Fix For: 3.2.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-12602, we manually constructed 
> a correct license file for 2.8.0. This file will certainly become wrong again 
> in later releases, so we need to write some kind of script to automate a 
> check.
> It crossed my mind to automate the generation of the file, but it seems to be 
> an intractable problem, considering that each dependency may change licenses, 
> may package license files, link to them from their poms, link to them from 
> their repos, etc. I've also found multiple URLs listed with various 
> delimiters, broken links that I have to chase down, etc.
> Therefore, it seems like the solution to aim for is simply: list all the jars 
> that we package, and print out a report of each jar that's extra or missing 
> vs. the ones in our `LICENSE-binary` file.
> The check should be part of the release script at least, if not part of the 
> regular build (so we keep it up to date as dependencies change).
>  
> Here's how I do this manually right now:
> {code:java}
> // build the binary artifacts
> $ ./gradlewAll releaseTarGz
> // unpack the binary artifact 
> $ tar xf core/build/distributions/kafka_2.13-X.Y.Z.tgz
> $ cd xf kafka_2.13-X.Y.Z
> // list the packaged jars 
> // (you can ignore the jars for our own modules, like kafka, kafka-clients, 
> etc.)
> $ ls libs/
> // cross check the jars with the packaged LICENSE
> // make sure all dependencies are listed with the right versions
> $ cat LICENSE
> // also double check all the mentioned license files are present
> $ ls licenses {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-13142) KRaft brokers do not validate dynamic configs before forwarding them to controller

[Mickael Maison (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489710#comment-17489710
 ] 

Mickael Maison commented on KAFKA-13142:


[~rdielhenn] Is this really a blocker? Kafka 3.1 has been released without this 
fix so it does not look like this should be blocking 3.0.1.

> KRaft brokers do not validate dynamic configs before forwarding them to 
> controller
> --
>
> Key: KAFKA-13142
> URL: https://issues.apache.org/jira/browse/KAFKA-13142
> Project: Kafka
>  Issue Type: Task
>  Components: kraft
>Affects Versions: 3.0.0
>Reporter: Ryan Dielhenn
>Assignee: Ryan Dielhenn
>Priority: Blocker
> Fix For: 3.0.1
>
>
> The KRaft brokers are not currently validating dynamic configs before 
> forwarding them to the controller. To ensure that KRaft clusters are easily 
> upgradable it would be a good idea to validate dynamic configs in the first 
> release of KRaft so that invalid dynamic configs are never stored.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-12644) Add Missing Class-Level Javadoc to Descendants of org.apache.kafka.common.errors.ApiException

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-12644:
---
Fix Version/s: (was: 3.0.1)

> Add Missing Class-Level Javadoc to Descendants of 
> org.apache.kafka.common.errors.ApiException
> -
>
> Key: KAFKA-12644
> URL: https://issues.apache.org/jira/browse/KAFKA-12644
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, documentation
>Affects Versions: 2.8.1, 3.0.0
>Reporter: Israel Ekpo
>Assignee: Israel Ekpo
>Priority: Major
>  Labels: documentation
> Fix For: 3.2.0
>
>
> I noticed that class-level Javadocs are missing from some classes in the 
> org.apache.kafka.common.errors package. This issue is for tracking the work 
> of adding the missing class-level javadocs for those Exception classes.
> https://kafka.apache.org/27/javadoc/org/apache/kafka/common/errors/package-summary.html
> https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/common/errors
> Basic class-level documentation could be derived by mapping the error 
> conditions documented in the protocol
> https://kafka.apache.org/protocol#protocol_constants



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-12774) kafka-streams 2.8: logging in uncaught-exceptionhandler doesn't go through log4j

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-12774:
---
Fix Version/s: (was: 3.0.1)

> kafka-streams 2.8: logging in uncaught-exceptionhandler doesn't go through 
> log4j
> 
>
> Key: KAFKA-12774
> URL: https://issues.apache.org/jira/browse/KAFKA-12774
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.8.0
>Reporter: Jørgen
>Priority: Minor
> Fix For: 3.2.0
>
>
> When exceptions is handled in the uncaught-exception handler introduced in 
> KS2.8, the logging of the stacktrace doesn't seem to go through the logging 
> framework configured by the application (log4j2 in our case), but gets 
> printed to console "line-by-line".
> All other exceptions logged by kafka-streams go through log4j2 and gets 
> formatted properly according to the log4j2 appender (json in our case). 
> Haven't tested this on other frameworks like logback.
> Application setup:
>  * Spring-boot 2.4.5
>  * Log4j 2.13.3
>  * Slf4j 1.7.30
> Log4j2 appender config:
> {code:java}
> 
> 
>  stacktraceAsString="true" properties="true">
>  value="$${date:-MM-dd'T'HH:mm:ss.SSSZ}"/>
> 
> 
>  {code}
> Uncaught exception handler config:
> {code:java}
> kafkaStreams.setUncaughtExceptionHandler { exception ->
> logger.warn("Uncaught exception handled - replacing thread", exception) 
> // logged properly
> 
> StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.REPLACE_THREAD
> } {code}
> Stacktrace that gets printed line-by-line:
> {code:java}
> Exception in thread "xxx-f5860dff-9a41-490e-8ab0-540b1a7f9ce4-StreamThread-2" 
> org.apache.kafka.streams.errors.StreamsException: Error encountered sending 
> record to topic xxx-repartition for task 3_2 due 
> to:org.apache.kafka.common.errors.InvalidPidMappingException: The producer 
> attempted to use a producer id which is not currently assigned to its 
> transactional id.Exception handler choose to FAIL the processing, no more 
> records would be sent.  at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:226)
>at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
>  at 
> org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1365)
> at 
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:231)
>  at 
> org.apache.kafka.clients.producer.internals.ProducerBatch.abort(ProducerBatch.java:159)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortUndrainedBatches(RecordAccumulator.java:783)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:430)
>  at 
> org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:315)  
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242)
>   at java.base/java.lang.Thread.run(Unknown Source)Caused by: 
> org.apache.kafka.common.errors.InvalidPidMappingException: The producer 
> attempted to use a producer id which is not currently assigned to its 
> transactional id. {code}
>  
> It's a little bit hard to reproduce as I haven't found any way to trigger 
> uncaught-exception-handler through junit-tests.
> Link to discussion on slack: 
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1620389197436700



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13411) Exception while connecting from kafka client consumer producers deployed in a wildfly context to a kafka broker implementing OAUTHBEARER sasl mechanism

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-13411:
---
Fix Version/s: (was: 3.0.1)

> Exception while connecting from kafka client consumer producers deployed in a 
> wildfly context to a kafka broker implementing OAUTHBEARER sasl mechanism
> ---
>
> Key: KAFKA-13411
> URL: https://issues.apache.org/jira/browse/KAFKA-13411
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 3.0.0
> Environment: Windows, Linux , Wildfly Application server
>Reporter: Shankar Bhaskaran
>Priority: Major
> Fix For: 3.2.0
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> I have set up a Kafka cluster on my linux machine secured using keycloak
> (OAUTHBEARER) Mechanism. I can use the Kafka Console Consumers and
> Producers to send and receive messages.
>  
> I have tried to connect to Kafka from my consumers and producers deployed
> as module on the wildfly App serve (version 19, java 11) . I have set up
> all the required configuration (Config Section at the bottom) .
> The SASL_JAAS_CONFIG provided as consumerconfig option has the details
> like (apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
> required LoginStringClaim_sub='kafka-client');
>  
> I am able to get authenticated with the broker , but in the client callback
> I am getting an Unsupported Callback error . I have 3 modules in wildfly
> 1) kafka producer consumer code dependent on the 2) oauth jar (for
> logincallbackhandler and login module) dependent on the 3) kafka-client
> jar (2.8.0)]
>  
> I can see that the CLIENT CALL BACK IS CLIENTCREDENTIAL INSTEAD OF
> OAuthBearerTokenCallback. The saslclient is getting set as
> AbstractSaslClient instead of OAuthBearerSaslClient.
> [https://www.mail-archive.com/dev@kafka.apache.org/msg120743.html]
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13228) ApiVersionRequest are not correctly handled in kraft mode

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-13228:
---
Fix Version/s: (was: 3.0.1)

> ApiVersionRequest are not correctly handled in kraft mode
> -
>
> Key: KAFKA-13228
> URL: https://issues.apache.org/jira/browse/KAFKA-13228
> Project: Kafka
>  Issue Type: Bug
>Reporter: dengziming
>Assignee: dengziming
>Priority: Major
>
> I'am trying to describe quorum in kraft mode but got 
> `org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support DESCRIBE_QUORUM`.
> This happens because we only concerns `ApiKeys.zkBrokerApis()` when we call 
> `NodeApiVersions.create()`



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13188) Release the memory back into MemoryPool

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-13188:
---
Fix Version/s: (was: 3.0.1)

> Release the memory back into MemoryPool
> ---
>
> Key: KAFKA-13188
> URL: https://issues.apache.org/jira/browse/KAFKA-13188
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Lucas Wang
>Assignee: Alok Nikhil
>Priority: Major
>
> Tushar made a [hotfix change|https://github.com/linkedin/kafka/pull/186] to 
> the linkedin/kafka repo hosting apache kafka 2.4.
> The change is about releasing memory back to the MemoryPool for the kafka 
> consumer, and his benchmark showed significant improvement in terms of the 
> memory graduating from Young Gen and promoted to Old Gen.
> Given the benefit, the change should also be added trunk.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13242) KRaft Controller doesn't handle UpdateFeaturesRequest

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-13242:
---
Fix Version/s: (was: 3.0.1)

> KRaft Controller doesn't handle UpdateFeaturesRequest
> -
>
> Key: KAFKA-13242
> URL: https://issues.apache.org/jira/browse/KAFKA-13242
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: dengziming
>Assignee: dengziming
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13517) Add ConfigurationKeys to ConfigResource class

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison updated KAFKA-13517:
---
Fix Version/s: (was: 2.8.1)

> Add ConfigurationKeys to ConfigResource class
> -
>
> Key: KAFKA-13517
> URL: https://issues.apache.org/jira/browse/KAFKA-13517
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 2.8.1, 3.0.0
>Reporter: Vikas Singh
>Assignee: Vikas Singh
>Priority: Major
>
> A list of {{ConfigResource}} class is passed as argument to 
> {{AdminClient::describeConfigs}} api to indicate configuration of the 
> entities to fetch. The {{ConfigResource}} class is made up of two fields, 
> name and type of entity. Kafka returns *all* configurations for the entities 
> provided to the admin client api.
> This admin api in turn uses {{DescribeConfigsRequest}} kafka api to get the 
> configuration for the entities in question. In addition to name and type of 
> entity whose configuration to get, Kafka {{DescribeConfigsResource}} 
> structure also lets users provide {{ConfigurationKeys}} list, which allows 
> users to fetch only the configurations that are needed.
> However, this field isn't exposed in the {{ConfigResource}} class that is 
> used by AdminClient, so users of AdminClient have no way to ask for specific 
> configuration. The API always returns *all* configurations. Then the user of 
> the {{AdminClient::describeConfigs}} go over the returned list and filter out 
> the config keys that they are interested in.
> This results in boilerplate code for all users of 
> {{AdminClient::describeConfigs}} api, in addition to  being wasteful use of 
> resource. It becomes painful in large cluster case where to fetch one 
> configuration of all topics, we need to fetch all configuration of all 
> topics, which can be huge in size. 
> Creating this Jira to add same field (i.e. {{{}ConfigurationKeys{}}}) to the 
> {{ConfigResource}} structure to bring it to parity to 
> {{DescribeConfigsResource}} Kafka API structure. There should be no backward 
> compatibility issue as the field will be optional and will behave same way if 
> it is not specified (i.e. by passing null to backend kafka api) 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-12468) Initial offsets are copied from source to target cluster

[Guram Savinov (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489716#comment-17489716
 ] 

Guram Savinov commented on KAFKA-12468:
---

[~bdeneuter] there is IdentityReplicationPolicy which can be used to preserve 
topic names, maybe you don't need to implement your CustomReplicationPolicy.

https://issues.apache.org/jira/browse/KAFKA-9726

> Initial offsets are copied from source to target cluster
> 
>
> Key: KAFKA-12468
> URL: https://issues.apache.org/jira/browse/KAFKA-12468
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0
>Reporter: Bart De Neuter
>Priority: Major
>
> We have an active-passive setup where  the 3 connectors from mirror maker 2 
> (heartbeat, checkpoint and source) are running on a dedicated Kafka connect 
> cluster on the target cluster.
> Offset syncing is enabled as specified by KIP-545. But when activated, it 
> seems the offsets from the source cluster are initially copied to the target 
> cluster without translation. This causes a negative lag for all synced 
> consumer groups. Only when we reset the offsets for each topic/partition on 
> the target cluster and produce a record on the topic/partition in the source, 
> the sync starts working correctly. 
> I would expect that the consumer groups are synced but that the current 
> offsets of the source cluster are not copied to the target cluster.
> This is the configuration we are currently using:
> Heartbeat connector
>  
> {code:xml}
> {
>   "name": "mm2-mirror-heartbeat",
>   "config": {
> "name": "mm2-mirror-heartbeat",
> "connector.class": 
> "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
> "source.cluster.alias": "eventador",
> "target.cluster.alias": "msk",
> "source.cluster.bootstrap.servers": "",
> "target.cluster.bootstrap.servers": "",
> "topics": ".*",
> "groups": ".*",
> "tasks.max": "1",
> "replication.policy.class": "CustomReplicationPolicy",
> "sync.group.offsets.enabled": "true",
> "sync.group.offsets.interval.seconds": "5",
> "emit.checkpoints.enabled": "true",
> "emit.checkpoints.interval.seconds": "30",
> "emit.heartbeats.interval.seconds": "30",
> "key.converter": " 
> org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter": 
> "org.apache.kafka.connect.converters.ByteArrayConverter"
>   }
> }
> {code}
> Checkpoint connector:
> {code:xml}
> {
>   "name": "mm2-mirror-checkpoint",
>   "config": {
> "name": "mm2-mirror-checkpoint",
> "connector.class": 
> "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
> "source.cluster.alias": "eventador",
> "target.cluster.alias": "msk",
> "source.cluster.bootstrap.servers": "",
> "target.cluster.bootstrap.servers": "",
> "topics": ".*",
> "groups": ".*",
> "tasks.max": "40",
> "replication.policy.class": "CustomReplicationPolicy",
> "sync.group.offsets.enabled": "true",
> "sync.group.offsets.interval.seconds": "5",
> "emit.checkpoints.enabled": "true",
> "emit.checkpoints.interval.seconds": "30",
> "emit.heartbeats.interval.seconds": "30",
> "key.converter": " 
> org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter": 
> "org.apache.kafka.connect.converters.ByteArrayConverter"
>   }
> }
> {code}
>  Source connector:
> {code:xml}
> {
>   "name": "mm2-mirror-source",
>   "config": {
> "name": "mm2-mirror-source",
> "connector.class": 
> "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> "source.cluster.alias": "eventador",
> "target.cluster.alias": "msk",
> "source.cluster.bootstrap.servers": "",
> "target.cluster.bootstrap.servers": "",
> "topics": ".*",
> "groups": ".*",
> "tasks.max": "40",
> "replication.policy.class": "CustomReplicationPolicy",
> "sync.group.offsets.enabled": "true",
> "sync.group.offsets.interval.seconds": "5",
> "emit.checkpoints.enabled": "true",
> "emit.checkpoints.interval.seconds": "30",
> "emit.heartbeats.interval.seconds": "30",
> "key.converter": " 
> org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter": 
> "org.apache.kafka.connect.converters.ByteArrayConverter"
>   }
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (KAFKA-13661) KRaft uses the wrong permission for adding topic partitions

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-13661:

Summary: KRaft uses the wrong permission for adding topic partitions  (was: 
KRaft uses the wrong permission for creating partitions)

> KRaft uses the wrong permission for adding topic partitions
> ---
>
> Key: KAFKA-13661
> URL: https://issues.apache.org/jira/browse/KAFKA-13661
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> [~cmccabe] caught this as part of KAFKA-13646. KRaft currently checks CREATE 
> on the topic resource. It should be ALTER. This will be fixed in trunk as 
> part of KAFKA-13646, but it would be good to fix for 3.0 and 3.1 as well.
> Note this does not affect zookeeper-based clusters.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13661) KRaft uses the wrong permission for creating partitions

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13661:
---

 Summary: KRaft uses the wrong permission for creating partitions
 Key: KAFKA-13661
 URL: https://issues.apache.org/jira/browse/KAFKA-13661
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.0.0, 3.1.0
Reporter: Jason Gustafson
Assignee: Jason Gustafson


[~cmccabe] caught this as part of KAFKA-13646. KRaft currently checks CREATE on 
the topic resource. It should be ALTER. This will be fixed in trunk as part of 
KAFKA-13646, but it would be good to fix for 3.0 and 3.1 as well.

Note this does not affect zookeeper-based clusters.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)

[Guram Savinov (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489723#comment-17489723
 ] 

Guram Savinov commented on KAFKA-7500:
--

Please update KIP-382 documentation:

LegacyReplicationPolicy -> IdentityReplicationPolicy

https://issues.apache.org/jira/browse/KAFKA-9726

> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect, mirrormaker
>Affects Versions: 2.4.0
>Reporter: Ryanne Dolan
>Assignee: Ryanne Dolan
>Priority: Major
>  Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect 
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka] tombentley commented on a change in pull request #11673: KAFKA-13577: Replace easymock with mockito in kafka:core - part 2

[GitBox]

tombentley commented on a change in pull request #11673:
URL: https://github.com/apache/kafka/pull/11673#discussion_r802922650



##
File path: core/src/test/scala/unit/kafka/server/ReplicaManagerQuotasTest.scala
##
@@ -157,10 +158,9 @@ class ReplicaManagerQuotasTest {
   def shouldIncludeThrottledReplicasForConsumerFetch(): Unit = {
 setUpMocks(fetchInfo)
 
-val quota = mockQuota(100)
-expect(quota.isQuotaExceeded).andReturn(true).once()
-expect(quota.isQuotaExceeded).andReturn(true).once()
-replay(quota)
+val quota = mockQuota()
+when(quota.isQuotaExceeded).thenReturn(true)
+//expect(quota.isQuotaExceeded).andReturn(true).once()

Review comment:
   Can we remove this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] hachikuji merged pull request #11649: KAFKA-13646: Implement KIP-801: KRaft authorizer

[GitBox]

hachikuji merged pull request #11649:
URL: https://github.com/apache/kafka/pull/11649


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-13646) Implement KIP-801: KRaft authorizer

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13646.
-
Fix Version/s: 3.2.0
   Resolution: Fixed

> Implement KIP-801: KRaft authorizer
> ---
>
> Key: KAFKA-13646
> URL: https://issues.apache.org/jira/browse/KAFKA-13646
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin McCabe
>Assignee: Colin McCabe
>Priority: Major
>  Labels: kip-500, kip-801
> Fix For: 3.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-13638) Slow KTable update when forwarding multiple values from transformer

[Bruno Cadonna (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489741#comment-17489741
 ] 

Bruno Cadonna commented on KAFKA-13638:
---

[~Lejon] Could you try to use another state store directory ({{state.dir}} 
config). By default that config points to {{/tmp/kafka-streams}}. Maybe the OS 
makes something weird with the temporary directory. Just an idea! 

> Slow KTable update when forwarding multiple values from transformer
> ---
>
> Key: KAFKA-13638
> URL: https://issues.apache.org/jira/browse/KAFKA-13638
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Ulrik
>Priority: Major
> Attachments: KafkaTest.java
>
>
> I have a topology where I stream messages from an input topic, transform the 
> message to multiple messages (via context.forward), and then store those 
> messages in a KTable.
> Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my 
> tests take significantly longer time to run. 
>  
> I have attached a test class to demonstrate my scenario. When running this 
> test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following 
> numbers:
>  
> *Version 2.8.1*
>  * one input message and one output message: 541 ms
>  * 8 input message and 30 output message per input message (240 output 
> messages in total): 919 ms
>  
> *Version 3.1.0*
>  * one input message and one output message: 908 ms
>  * 8 input message and 30 output message per input message (240 output 
> messages in total): 6 sec 94 ms
>  
> Even when the transformer just transforms and forwards one input message to 
> one output message, the test takes approx. 400 ms longer to run.
> When transforming 8 input messages to 240 output messages it takes approx 5 
> seconds longer.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (KAFKA-13422) Even if the correct username and password are configured, when ClientBroker or KafkaClient tries to establish a SASL connection to ServerBroker, an exception is thrown

[Guozhang Wang (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489744#comment-17489744
 ] 

Guozhang Wang commented on KAFKA-13422:
---

I'm unfortunately less familiar with security.auth module, [~rsivaram] [~ijuma] 
could you please chime in with your thoughts?

> Even if the correct username and password are configured, when ClientBroker 
> or KafkaClient tries to establish a SASL connection to ServerBroker, an 
> exception is thrown: (Authentication failed: Invalid username or password)
> --
>
> Key: KAFKA-13422
> URL: https://issues.apache.org/jira/browse/KAFKA-13422
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 2.7.1, 3.0.0
>Reporter: RivenSun
>Priority: Major
> Attachments: CustomerAuthCallbackHandler.java, 
> LoginContext_login_debug.png, SaslClientCallbackHandler_handle_debug.png
>
>
>  
> h1. Foreword:
> When deploying a Kafka cluster with a higher version (2.7.1), I encountered 
> an exception of communication identity authentication failure between 
> brokers. In the current latest version 3.0.0, this problem can also be 
> reproduced.
> h1. Problem recurring:
> h2. 1）broker Version is 3.0.0
> h3. The content of kafka_server_jaas.conf of each broker is exactly the same, 
> the content is as follows：
>  
>  
> {code:java}
> KafkaServer {
>   org.apache.kafka.common.security.plain.PlainLoginModule required
>   username="admin"
>   password="kJTVDziatPgjXG82sFHc4O1EIuewmlvS"
>   user_admin="kJTVDziatPgjXG82sFHc4O1EIuewmlvS"
>   user_alice="alice";
>   org.apache.kafka.common.security.scram.ScramLoginModule required
>   username="admin_scram"
>   password="admin_scram_password";
>  
> };
> {code}
>  
>  
> h3. broker server.properties：
> One of the broker configuration files is provided, and the content of the 
> configuration files of other brokers is only different from the localPublicIp 
> of advertised.listeners.
>  
> {code:java}
> broker.id=1
> broker.rack=us-east-1a
> advertised.listeners=SASL_PLAINTEXT://localPublicIp:9779,SASL_SSL://localPublicIp:9889,INTERNAL_SSL://:9009,PLAIN_PLUGIN_SSL://localPublicIp:9669
> log.dirs=/asyncmq/kafka/data_1,/asyncmq/kafka/data_2
> zookeeper.connect=***
> listeners=SASL_PLAINTEXT://:9779,SASL_SSL://:9889,INTERNAL_SSL://:9009,PLAIN_PLUGIN_SSL://:9669
> listener.security.protocol.map=INTERNAL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAIN_PLUGIN_SSL:SASL_SSL
> listener.name.plain_plugin_ssl.plain.sasl.server.callback.handler.class=org.apache.kafka.common.security.plain.internals.PlainServerCallbackHandler
> #ssl config
> ssl.keystore.password=***
> ssl.key.password=***
> ssl.truststore.password=***
> ssl.keystore.location=***
> ssl.truststore.location=***
> ssl.client.auth=none
> ssl.endpoint.identification.algorithm=
> #broker communicate config
> #security.inter.broker.protocol=SASL_PLAINTEXT
> inter.broker.listener.name=INTERNAL_SSL
> sasl.mechanism.inter.broker.protocol=PLAIN
> #sasl authentication config
> sasl.kerberos.service.name=kafka
> sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,GSSAPI
> delegation.token.master.key=***
> delegation.token.expiry.time.ms=8640
> delegation.token.max.lifetime.ms=31536
> {code}
>  
>  
> Then start all brokers at the same time. Each broker has actually been 
> started successfully, but when establishing a connection between the 
> controller node and all brokers, the identity authentication has always 
> failed. The connection between brokers cannot be established normally, 
> causing the entire Kafka cluster to be unable to provide external services.
> h3. The server log keeps printing abnormally like crazy：
> The real ip sensitive information of the broker in the log, I use ** 
> instead of here
>  
> {code:java}
> [2021-10-29 14:16:19,831] INFO [SocketServer listenerType=ZK_BROKER, 
> nodeId=3] Started socket server acceptors and processors 
> (kafka.network.SocketServer)
> [2021-10-29 14:16:19,836] INFO Kafka version: 3.0.0 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,836] INFO Kafka commitId: 8cb0a5e9d3441962 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,836] INFO Kafka startTimeMs: 1635516979831 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,837] INFO [KafkaServer id=3] started 
> (kafka.server.KafkaServer)
> [2021-10-29 14:16:20,249] INFO [SocketServer listenerType=ZK_BROKER, 
> nodeId=3] Failed authentication with /** (Authentication failed: Invalid 
> username or password) (org.apache.kaf

[GitHub] [kafka] mimaison commented on a change in pull request #11673: KAFKA-13577: Replace easymock with mockito in kafka:core - part 2

[GitBox]

mimaison commented on a change in pull request #11673:
URL: https://github.com/apache/kafka/pull/11673#discussion_r802994307



##
File path: core/src/test/scala/unit/kafka/server/ReplicaManagerQuotasTest.scala
##
@@ -157,10 +158,9 @@ class ReplicaManagerQuotasTest {
   def shouldIncludeThrottledReplicasForConsumerFetch(): Unit = {
 setUpMocks(fetchInfo)
 
-val quota = mockQuota(100)
-expect(quota.isQuotaExceeded).andReturn(true).once()
-expect(quota.isQuotaExceeded).andReturn(true).once()
-replay(quota)
+val quota = mockQuota()
+when(quota.isQuotaExceeded).thenReturn(true)
+//expect(quota.isQuotaExceeded).andReturn(true).once()

Review comment:
   Oops! I forgot to remove it when rebasing. I've fixed it now




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison commented on a change in pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append

[GitBox]

mimaison commented on a change in pull request #11744:
URL: https://github.com/apache/kafka/pull/11744#discussion_r803007182



##
File path: core/src/main/scala/kafka/log/OffsetIndex.scala
##
@@ -136,7 +136,7 @@ class OffsetIndex(_file: File, baseOffset: Long, 
maxIndexSize: Int = -1, writabl
 
   /**
* Append an entry for the given offset/location pair to the index. This 
entry must have a larger offset than all subsequent entries.
-   * @throws IndexOffsetOverflowException if the offset causes index offset to 
overflow
+   * @throws InvalidOffsetException if the offset causes index offset to 
overflow

Review comment:
   This method does indeed throw `IndexOffsetOverflowException` (from 
`relativeOffset()`) for the reason listed. 
   
   I'm assuming you got confused because it also throws 
`InvalidOffsetException`. It's ok to add another `@throws` tag if you want but 
we don't want to remove the existing one.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] mimaison commented on pull request #11731: KAFKA-13293: Reloading SSL Engine Factory

[GitBox]

mimaison commented on pull request #11731:
URL: https://github.com/apache/kafka/pull/11731#issuecomment-1034113905


   Thanks @teabot for the contribution!
   
   This PR adds new configurations and these are considered public API. So in 
order to accept this change we need a [Kafka Improvement 
Proposal](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals).
   
   Let me know if you have any questions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] lmr3796 closed pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append

[GitBox]

lmr3796 closed pull request #11744:
URL: https://github.com/apache/kafka/pull/11744


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1468: KAFKA-3790: Allow for removal of non specific ACLs

[GitBox]

vvcephei closed pull request #1468:
URL: https://github.com/apache/kafka/pull/1468


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1468: KAFKA-3790: Allow for removal of non specific ACLs

[GitBox]

vvcephei commented on pull request #1468:
URL: https://github.com/apache/kafka/pull/1468#issuecomment-1034117452


   Hi @slaunay ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1415: KAFKA-3737: Change log level for error during produce request

[GitBox]

vvcephei commented on pull request #1415:
URL: https://github.com/apache/kafka/pull/1415#issuecomment-1034117748


   Hi @fhussonnois ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1415: KAFKA-3737: Change log level for error during produce request

[GitBox]

vvcephei closed pull request #1415:
URL: https://github.com/apache/kafka/pull/1415


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1269: KAFKA-3622: Use descriptive error message if port number is missing from url

[GitBox]

vvcephei closed pull request #1269:
URL: https://github.com/apache/kafka/pull/1269


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1269: KAFKA-3622: Use descriptive error message if port number is missing from url

[GitBox]

vvcephei commented on pull request #1269:
URL: https://github.com/apache/kafka/pull/1269#issuecomment-1034118007


   Hi @peterableda ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1244: MINOR: Docs for ACLs over SSL auth and KAFKA_OPTS

[GitBox]

vvcephei commented on pull request #1244:
URL: https://github.com/apache/kafka/pull/1244#issuecomment-1034118238


   Hi @QwertyManiac ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1244: MINOR: Docs for ACLs over SSL auth and KAFKA_OPTS

[GitBox]

vvcephei closed pull request #1244:
URL: https://github.com/apache/kafka/pull/1244


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1150: KAFKA-3474: add metrics to track replica fetcher timeouts

[GitBox]

vvcephei closed pull request #1150:
URL: https://github.com/apache/kafka/pull/1150


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1150: KAFKA-3474: add metrics to track replica fetcher timeouts

[GitBox]

vvcephei commented on pull request #1150:
URL: https://github.com/apache/kafka/pull/1150#issuecomment-1034118461


   Hi @junrao ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] lmr3796 commented on a change in pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append

[GitBox]

lmr3796 commented on a change in pull request #11744:
URL: https://github.com/apache/kafka/pull/11744#discussion_r803018010



##
File path: core/src/main/scala/kafka/log/OffsetIndex.scala
##
@@ -136,7 +136,7 @@ class OffsetIndex(_file: File, baseOffset: Long, 
maxIndexSize: Int = -1, writabl
 
   /**
* Append an entry for the given offset/location pair to the index. This 
entry must have a larger offset than all subsequent entries.
-   * @throws IndexOffsetOverflowException if the offset causes index offset to 
overflow
+   * @throws InvalidOffsetException if the offset causes index offset to 
overflow

Review comment:
   Ah @mimaison you're right.  Let me try to change it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1147: [KAFKA-3472] Allow MirrorMaker to copy selected partitions and choose target topic name

[GitBox]

vvcephei closed pull request #1147:
URL: https://github.com/apache/kafka/pull/1147


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1135: [KAFKA-3458] Selector should throw InterruptException when interrupted.

[GitBox]

vvcephei commented on pull request #1135:
URL: https://github.com/apache/kafka/pull/1135#issuecomment-1034118967


   Hi @sruehl ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1111: KAFKA-3428 Remove metadata sync bottleneck from mirrormaker's producer

[GitBox]

vvcephei closed pull request #:
URL: https://github.com/apache/kafka/pull/


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1147: [KAFKA-3472] Allow MirrorMaker to copy selected partitions and choose target topic name

[GitBox]

vvcephei commented on pull request #1147:
URL: https://github.com/apache/kafka/pull/1147#issuecomment-1034118707


   Hi @ooasis ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1135: [KAFKA-3458] Selector should throw InterruptException when interrupted.

[GitBox]

vvcephei closed pull request #1135:
URL: https://github.com/apache/kafka/pull/1135


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1111: KAFKA-3428 Remove metadata sync bottleneck from mirrormaker's producer

[GitBox]

vvcephei commented on pull request #:
URL: https://github.com/apache/kafka/pull/#issuecomment-1034119203


   Hi @maysamyabandeh ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1078: Fixes for Windows #154

[GitBox]

vvcephei closed pull request #1078:
URL: https://github.com/apache/kafka/pull/1078


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1078: Fixes for Windows #154

[GitBox]

vvcephei commented on pull request #1078:
URL: https://github.com/apache/kafka/pull/1078#issuecomment-1034119371


   Hi @JeffersJi ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #1035: KAFKA-3359 Parallel log-recovery of un-flushed segments on startup

[GitBox]

vvcephei commented on pull request #1035:
URL: https://github.com/apache/kafka/pull/1035#issuecomment-1034119827


   Hello,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #983: KAFKA-3300: Avoid over allocating disk space and memory for index files.

[GitBox]

vvcephei commented on pull request #983:
URL: https://github.com/apache/kafka/pull/983#issuecomment-1034120088


   Hi @becketqin ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #983: KAFKA-3300: Avoid over allocating disk space and memory for index files.

[GitBox]

vvcephei closed pull request #983:
URL: https://github.com/apache/kafka/pull/983


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #1035: KAFKA-3359 Parallel log-recovery of un-flushed segments on startup

[GitBox]

vvcephei closed pull request #1035:
URL: https://github.com/apache/kafka/pull/1035


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #907: KAFKA-3234; Clarify minISR in documentation and auto-generate topic configuration docs

[GitBox]

vvcephei commented on pull request #907:
URL: https://github.com/apache/kafka/pull/907#issuecomment-1034120532


   Hi @jjkoshy ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #907: KAFKA-3234; Clarify minISR in documentation and auto-generate topic configuration docs

[GitBox]

vvcephei closed pull request #907:
URL: https://github.com/apache/kafka/pull/907


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #880: KAFKA-3190 Producer should not fire callback in Send() method

[GitBox]

vvcephei closed pull request #880:
URL: https://github.com/apache/kafka/pull/880


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #880: KAFKA-3190 Producer should not fire callback in Send() method

[GitBox]

vvcephei commented on pull request #880:
URL: https://github.com/apache/kafka/pull/880#issuecomment-1034120832


   Hi @becketqin ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #824: KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used in java.util.Properties

[GitBox]

vvcephei closed pull request #824:
URL: https://github.com/apache/kafka/pull/824


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #824: KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used in java.util.Properties

[GitBox]

vvcephei commented on pull request #824:
URL: https://github.com/apache/kafka/pull/824#issuecomment-1034121012


   Hi @crhyne ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #757: KAFKA-3082: Make LogManager.InitialTaskDelayMs configurable

[GitBox]

vvcephei commented on pull request #757:
URL: https://github.com/apache/kafka/pull/757#issuecomment-1034121306


   Ho @j-nowak ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #757: KAFKA-3082: Make LogManager.InitialTaskDelayMs configurable

[GitBox]

vvcephei closed pull request #757:
URL: https://github.com/apache/kafka/pull/757


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #735: KAFKA-3065: Remove unused topic partitions from RecordAccumulator

[GitBox]

vvcephei closed pull request #735:
URL: https://github.com/apache/kafka/pull/735


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #735: KAFKA-3065: Remove unused topic partitions from RecordAccumulator

[GitBox]

vvcephei commented on pull request #735:
URL: https://github.com/apache/kafka/pull/735#issuecomment-1034121499


   Hi @rajinisivaram ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #719: KAFKA-3049: VerifiableProperties does not respect 'default' properties of underlying java.util.Properties instance

[GitBox]

vvcephei commented on pull request #719:
URL: https://github.com/apache/kafka/pull/719#issuecomment-1034121675


   Hi @jeffreyolchovy ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #719: KAFKA-3049: VerifiableProperties does not respect 'default' properties of underlying java.util.Properties instance

[GitBox]

vvcephei closed pull request #719:
URL: https://github.com/apache/kafka/pull/719


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #62: add support libvirt as provider. KAFKA-2183

[GitBox]

vvcephei commented on pull request #62:
URL: https://github.com/apache/kafka/pull/62#issuecomment-1034121935


   Hi @pronix ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei closed pull request #62: add support libvirt as provider. KAFKA-2183

[GitBox]

vvcephei closed pull request #62:
URL: https://github.com/apache/kafka/pull/62


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] vvcephei commented on pull request #200: KAFKA-2512: Add version check to broker and clients.

[GitBox]

vvcephei commented on pull request #200:
URL: https://github.com/apache/kafka/pull/200#issuecomment-1034122122


   Hi @becketqin ,
   
   It seems like this PR stalled. I'll close it out for now, but if you or 
anyone else want to resume this work, please feel free to re-open it (or start 
a new one)!
   
   Thanks,
   John


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org