[jira] [Updated] (KAFKA-14432) RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread Lucas Brutschy (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Brutschy updated KAFKA-14432: --- Priority: Blocker (was: Major) > RocksDBStore relies on finalizers to not leak memory > ---

[jira] [Updated] (KAFKA-14432) RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread Lucas Brutschy (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Brutschy updated KAFKA-14432: --- Fix Version/s: 3.4.0 > RocksDBStore relies on finalizers to not leak memory > --

[GitHub] [kafka] divijvaidya commented on pull request #12911: KAFKA-14418: Do not allow modifying partition of internal topics from admin commands

2022-12-07 Thread GitBox
divijvaidya commented on PR #12911: URL: https://github.com/apache/kafka/pull/12911#issuecomment-1340820090 Discarding this PR. We will start a KIP for a better way to handle modifications of number of partitions for `__consumer_offsets`. -- This is an automated message from the Apache G

[GitHub] [kafka] divijvaidya closed pull request #12911: KAFKA-14418: Do not allow modifying partition of internal topics from admin commands

2022-12-07 Thread GitBox
divijvaidya closed pull request #12911: KAFKA-14418: Do not allow modifying partition of internal topics from admin commands URL: https://github.com/apache/kafka/pull/12911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [kafka] cadonna commented on a diff in pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
cadonna commented on code in PR #12935: URL: https://github.com/apache/kafka/pull/12935#discussion_r1042067136 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java: ## @@ -108,7 +108,9 @@ public class RocksDBStore implements KeyValueStore, BatchWri

[GitHub] [kafka] divijvaidya commented on pull request #12331: KAFKA-1194: changes needed to run on Windows

2022-12-07 Thread GitBox
divijvaidya commented on PR #12331: URL: https://github.com/apache/kafka/pull/12331#issuecomment-1340831891 @MPeli what is the latest on this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [kafka] rajinisivaram commented on pull request #12954: KAFKA-14352: Rack-aware consumer partition assignment protocol changes (KIP-881)

2022-12-07 Thread GitBox
rajinisivaram commented on PR #12954: URL: https://github.com/apache/kafka/pull/12954#issuecomment-1340845106 @dajac Thanks for the review, merging to trunk. Will also confirm with @ableegoldman if we can merge to 3.4 branch. -- This is an automated message from the Apache Git Service. To

[GitHub] [kafka] rajinisivaram merged pull request #12954: KAFKA-14352: Rack-aware consumer partition assignment protocol changes (KIP-881)

2022-12-07 Thread GitBox
rajinisivaram merged PR #12954: URL: https://github.com/apache/kafka/pull/12954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.a

[GitHub] [kafka] dengziming commented on a diff in pull request #12951: MINOR: Move MetadataQuorumCommand from `core` to `tools`

2022-12-07 Thread GitBox
dengziming commented on code in PR #12951: URL: https://github.com/apache/kafka/pull/12951#discussion_r1042114095 ## bin/kafka-metadata-quorum.sh: ## @@ -14,4 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -exe

[GitHub] [kafka] lucasbru commented on a diff in pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
lucasbru commented on code in PR #12935: URL: https://github.com/apache/kafka/pull/12935#discussion_r1042174486 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java: ## @@ -108,7 +108,9 @@ public class RocksDBStore implements KeyValueStore, BatchWr

[GitHub] [kafka] dengziming commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
dengziming commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042151200 ## core/src/main/scala/kafka/server/KafkaConfig.scala: ## @@ -1665,6 +1665,7 @@ class KafkaConfig private(doLog: Boolean, val props: java.util.Map[_, _], dynami

[GitHub] [kafka] dengziming commented on pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
dengziming commented on PR #12961: URL: https://github.com/apache/kafka/pull/12961#issuecomment-1340958183 We can close #11042 and #11281 after merging this, but we can take a look to see whether we can port some useful code. -- This is an automated message from the Apache Git Service. To

[GitHub] [kafka] dengziming merged pull request #12912: MINOR: Fix some warnings

2022-12-07 Thread GitBox
dengziming merged PR #12912: URL: https://github.com/apache/kafka/pull/12912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apac

[GitHub] [kafka] dengziming commented on a diff in pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
dengziming commented on code in PR #12797: URL: https://github.com/apache/kafka/pull/12797#discussion_r1042208012 ## core/src/main/scala/kafka/admin/BrokerApiVersionsCommand.scala: ## @@ -95,8 +96,6 @@ object BrokerApiVersionsCommand { def checkArgs(): Unit = { Com

[GitHub] [kafka] justinrlee commented on a diff in pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
justinrlee commented on code in PR #12797: URL: https://github.com/apache/kafka/pull/12797#discussion_r1042238071 ## core/src/main/scala/kafka/admin/BrokerApiVersionsCommand.scala: ## @@ -95,8 +96,6 @@ object BrokerApiVersionsCommand { def checkArgs(): Unit = { Com

[GitHub] [kafka] justinrlee commented on a diff in pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
justinrlee commented on code in PR #12797: URL: https://github.com/apache/kafka/pull/12797#discussion_r1042243417 ## core/src/main/scala/kafka/tools/ConsoleConsumer.scala: ## @@ -218,7 +220,7 @@ object ConsoleConsumer extends Logging { .withRequiredArg .describedAs

[GitHub] [kafka] justinrlee commented on pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
justinrlee commented on PR #12797: URL: https://github.com/apache/kafka/pull/12797#issuecomment-1341013617 Hey @dengziming thanks for the review! I can build and test all of this locally and add output to the PR description. Are you also asking that I add automated tests for these, or is th

[GitHub] [kafka] dajac commented on pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
dajac commented on PR #12797: URL: https://github.com/apache/kafka/pull/12797#issuecomment-1341049038 Do we need a KIP for these changes? It seems that we are changing public interfaces here. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [kafka] justinrlee commented on pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
justinrlee commented on PR #12797: URL: https://github.com/apache/kafka/pull/12797#issuecomment-1341058382 > Do we need a KIP for these changes? It seems that we are changing public interfaces here. I was hoping no considering that this is a minor, non-breaking change (all existing u

[GitHub] [kafka] dajac commented on pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
dajac commented on PR #12797: URL: https://github.com/apache/kafka/pull/12797#issuecomment-1341074602 > As an additional quality-of-life change, kafka-console-producer and kafka-console-consumer also now support --command-config (in addition to continuing to support --producer.config and --

[jira] [Assigned] (KAFKA-13878) Connect deadlock in WorkerConnector on desired state change

2022-12-07 Thread Shaik Zakir Hussain (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaik Zakir Hussain reassigned KAFKA-13878: --- Assignee: Shaik Zakir Hussain > Connect deadlock in WorkerConnector on desi

[GitHub] [kafka] C0urante commented on pull request #12955: KAFKA-14443: Close topic creation Admin clients in MM2 connectors

2022-12-07 Thread GitBox
C0urante commented on PR #12955: URL: https://github.com/apache/kafka/pull/12955#issuecomment-1341129084 @ableegoldman Thanks for taking a look--I agree that a more thorough refactoring can wait for a follow-up PR. @gharris1727 I really like the general approach in the new `Backgroun

[jira] [Created] (KAFKA-14448) ZK brokers register with KRaft during migration

2022-12-07 Thread David Arthur (Jira)
David Arthur created KAFKA-14448: Summary: ZK brokers register with KRaft during migration Key: KAFKA-14448 URL: https://issues.apache.org/jira/browse/KAFKA-14448 Project: Kafka Issue Type: S

[jira] [Created] (KAFKA-14447) Controlled shutdown for ZK brokers during migration

2022-12-07 Thread David Arthur (Jira)
David Arthur created KAFKA-14447: Summary: Controlled shutdown for ZK brokers during migration Key: KAFKA-14447 URL: https://issues.apache.org/jira/browse/KAFKA-14447 Project: Kafka Issue Typ

[GitHub] [kafka] mumrah commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
mumrah commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042322800 ## core/src/main/scala/kafka/server/BrokerToControllerChannelManager.scala: ## @@ -37,42 +38,53 @@ import scala.collection.Seq import scala.compat.java8.OptionConverter

[GitHub] [kafka] mumrah commented on pull request #12946: KAFKA-14427 ZK client support for migrations

2022-12-07 Thread GitBox
mumrah commented on PR #12946: URL: https://github.com/apache/kafka/pull/12946#issuecomment-1341224855 @dengziming yes, there will be more PRs coming :). We are trying to get the basics for migration in place for 3.4 as an "early access" feature. If you find issues or have questions after t

[jira] [Created] (KAFKA-14449) Brokers not re-joining the ISR list and stuck at started until all the brokers restart

2022-12-07 Thread Swathi Mocharla (Jira)
Swathi Mocharla created KAFKA-14449: --- Summary: Brokers not re-joining the ISR list and stuck at started until all the brokers restart Key: KAFKA-14449 URL: https://issues.apache.org/jira/browse/KAFKA-14449

[GitHub] [kafka] divijvaidya commented on a diff in pull request #11390: [KAFKA-13369] Follower fetch protocol changes for tiered storage.

2022-12-07 Thread GitBox
divijvaidya commented on code in PR #11390: URL: https://github.com/apache/kafka/pull/11390#discussion_r1042140948 ## core/src/main/scala/kafka/server/ReplicaFetcherThread.scala: ## @@ -192,4 +204,140 @@ class ReplicaFetcherThread(name: String, partition.truncateFullyAndSta

[GitHub] [kafka] cmccabe commented on pull request #12932: KAFKA-14425; The Kafka protocol should support nullable structs

2022-12-07 Thread GitBox
cmccabe commented on PR #12932: URL: https://github.com/apache/kafka/pull/12932#issuecomment-1341277577 Thanks for this, @dajac . Can you add a test that the `Invalid default for struct field` message appears as expected when the default is set to something other than `null`? `MessageDataGe

[GitHub] [kafka] yashmayya commented on a diff in pull request #12947: KAFKA-6586: Refactor ConnectDistributed and ConnectStandalone to re-use shared logic

2022-12-07 Thread GitBox
yashmayya commented on code in PR #12947: URL: https://github.com/apache/kafka/pull/12947#discussion_r1042258502 ## connect/runtime/src/main/java/org/apache/kafka/connect/cli/AbstractConnectCli.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] [kafka] yashmayya commented on a diff in pull request #12947: KAFKA-6586: Refactor ConnectDistributed and ConnectStandalone to re-use shared logic

2022-12-07 Thread GitBox
yashmayya commented on code in PR #12947: URL: https://github.com/apache/kafka/pull/12947#discussion_r1042468179 ## connect/runtime/src/main/java/org/apache/kafka/connect/cli/ConnectStandalone.java: ## @@ -21,41 +21,51 @@ import org.apache.kafka.common.utils.Utils; import org

[GitHub] [kafka] kirktrue closed pull request #12814: KAFKA-14274: Implement fetching logic

2022-12-07 Thread GitBox
kirktrue closed pull request #12814: KAFKA-14274: Implement fetching logic URL: https://github.com/apache/kafka/pull/12814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [kafka] kirktrue closed pull request #12832: KAFKA-14365: Refactor Fetcher to allow different implementations 1/2

2022-12-07 Thread GitBox
kirktrue closed pull request #12832: KAFKA-14365: Refactor Fetcher to allow different implementations 1/2 URL: https://github.com/apache/kafka/pull/12832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [kafka] kirktrue closed pull request #12834: KAFKA-14365: Refactor Fetcher to allow different implementations 2/2

2022-12-07 Thread GitBox
kirktrue closed pull request #12834: KAFKA-14365: Refactor Fetcher to allow different implementations 2/2 URL: https://github.com/apache/kafka/pull/12834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Created] (KAFKA-14450) Rack-aware partition assignment for consumers (KIP-881)

2022-12-07 Thread Rajini Sivaram (Jira)
Rajini Sivaram created KAFKA-14450: -- Summary: Rack-aware partition assignment for consumers (KIP-881) Key: KAFKA-14450 URL: https://issues.apache.org/jira/browse/KAFKA-14450 Project: Kafka I

[jira] [Updated] (KAFKA-14352) Support rack-aware partition assignment for Kafka consumers

2022-12-07 Thread Rajini Sivaram (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-14352: --- Parent: KAFKA-14450 Issue Type: Sub-task (was: New Feature) > Support rack-aware pa

[jira] [Created] (KAFKA-14451) Make range assignor rack-aware if consumer racks are configured

2022-12-07 Thread Rajini Sivaram (Jira)
Rajini Sivaram created KAFKA-14451: -- Summary: Make range assignor rack-aware if consumer racks are configured Key: KAFKA-14451 URL: https://issues.apache.org/jira/browse/KAFKA-14451 Project: Kafka

[jira] [Created] (KAFKA-14452) Make sticky assignors rack-aware if consumer racks are configured.

2022-12-07 Thread Rajini Sivaram (Jira)
Rajini Sivaram created KAFKA-14452: -- Summary: Make sticky assignors rack-aware if consumer racks are configured. Key: KAFKA-14452 URL: https://issues.apache.org/jira/browse/KAFKA-14452 Project: Kafka

[GitHub] [kafka] mumrah commented on pull request #12913: KIP-866 Allow ZK brokers to register in KRaft

2022-12-07 Thread GitBox
mumrah commented on PR #12913: URL: https://github.com/apache/kafka/pull/12913#issuecomment-1341366297 @mimaison thanks for taking a look. This was a PR for the POC I've been working on for KIP-866. Now that the KIP is approved, we've started targeting `trunk`. I'll open a new PR to replace

[GitHub] [kafka] mumrah closed pull request #12913: KIP-866 Allow ZK brokers to register in KRaft

2022-12-07 Thread GitBox
mumrah closed pull request #12913: KIP-866 Allow ZK brokers to register in KRaft URL: https://github.com/apache/kafka/pull/12913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [kafka] mjsax commented on a diff in pull request #12944: [KAFKA-14395] add config to configure client supplier

2022-12-07 Thread GitBox
mjsax commented on code in PR #12944: URL: https://github.com/apache/kafka/pull/12944#discussion_r1042536401 ## streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java: ## @@ -1076,7 +1082,12 @@ public class StreamsConfig extends AbstractConfig { T

[GitHub] [kafka] lihaosky commented on a diff in pull request #12944: [KAFKA-14395] add config to configure client supplier

2022-12-07 Thread GitBox
lihaosky commented on code in PR #12944: URL: https://github.com/apache/kafka/pull/12944#discussion_r1042544360 ## streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java: ## @@ -1076,7 +1082,12 @@ public class StreamsConfig extends AbstractConfig {

[jira] [Commented] (KAFKA-14449) Brokers not re-joining the ISR list and stuck at started until all the brokers restart

2022-12-07 Thread Swathi Mocharla (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644466#comment-17644466 ] Swathi Mocharla commented on KAFKA-14449: - [~jsancio] , could you take a look?

[GitHub] [kafka] cmccabe commented on pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher

2022-12-07 Thread GitBox
cmccabe commented on PR #12958: URL: https://github.com/apache/kafka/pull/12958#issuecomment-1341414560 failing test is unrelated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [kafka] cmccabe merged pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher

2022-12-07 Thread GitBox
cmccabe merged PR #12958: URL: https://github.com/apache/kafka/pull/12958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.

[GitHub] [kafka] junrao commented on pull request #11390: [KAFKA-13369] Follower fetch protocol changes for tiered storage.

2022-12-07 Thread GitBox
junrao commented on PR #11390: URL: https://github.com/apache/kafka/pull/11390#issuecomment-1341457933 @showuon : I plan to take another look at the PR in the next few days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [kafka] akhileshchg commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
akhileshchg commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042536606 ## clients/src/main/java/org/apache/kafka/common/requests/AbstractControlRequest.java: ## @@ -43,6 +49,8 @@ protected AbstractControlRequest(ApiKeys api, short vers

[GitHub] [kafka] dajac merged pull request #12853: KAFKA-14367; Add `ListGroups` to the new `GroupCoordinator` interface

2022-12-07 Thread GitBox
dajac merged PR #12853: URL: https://github.com/apache/kafka/pull/12853 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

[GitHub] [kafka] hachikuji opened a new pull request, #12963: MINOR: More consistent handling of snapshot IDs

2022-12-07 Thread GitBox
hachikuji opened a new pull request, #12963: URL: https://github.com/apache/kafka/pull/12963 The ID we use for snapshots consists of the exclusive committed offset and epoch. That is, the offset in the ID is one more than the offset of the last committed record. This is currently handled a

[GitHub] [kafka] dajac commented on pull request #12932: KAFKA-14425; The Kafka protocol should support nullable structs

2022-12-07 Thread GitBox
dajac commented on PR #12932: URL: https://github.com/apache/kafka/pull/12932#issuecomment-1341640919 > Can you add a test that the Invalid default for struct field message appears as expected when the default is set to something other than null? MessageDataGeneratorTest.java has some examp

[jira] [Created] (KAFKA-14453) Flaky test suite MirrorConnectorsWithCustomForwardingAdminIntegrationTest

2022-12-07 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-14453: - Summary: Flaky test suite MirrorConnectorsWithCustomForwardingAdminIntegrationTest Key: KAFKA-14453 URL: https://issues.apache.org/jira/browse/KAFKA-14453 Project:

[GitHub] [kafka] C0urante commented on pull request #12955: KAFKA-14443: Close topic creation Admin clients in MM2 connectors

2022-12-07 Thread GitBox
C0urante commented on PR #12955: URL: https://github.com/apache/kafka/pull/12955#issuecomment-1341647209 The integration test failures, though worrying, do not appear to be caused by these changes and have been occurring for some time now. I've filed https://issues.apache.org/jira/browse/KA

[GitHub] [kafka] C0urante merged pull request #12955: KAFKA-14443: Close topic creation Admin clients in MM2 connectors

2022-12-07 Thread GitBox
C0urante merged PR #12955: URL: https://github.com/apache/kafka/pull/12955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache

[jira] [Updated] (KAFKA-14443) Mirror Maker Connectors leak admin clients used for topic creation

2022-12-07 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14443: -- Affects Version/s: (was: 3.4.0) > Mirror Maker Connectors leak admin clients used for topi

[jira] [Updated] (KAFKA-14443) Mirror Maker Connectors leak admin clients used for topic creation

2022-12-07 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14443: -- Priority: Blocker (was: Critical) > Mirror Maker Connectors leak admin clients used for topic

[GitHub] [kafka] C0urante commented on pull request #12955: KAFKA-14443: Close topic creation Admin clients in MM2 connectors

2022-12-07 Thread GitBox
C0urante commented on PR #12955: URL: https://github.com/apache/kafka/pull/12955#issuecomment-1341649696 Merged and backported to the 3.4 release branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Commented] (KAFKA-14443) Mirror Maker Connectors leak admin clients used for topic creation

2022-12-07 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644530#comment-17644530 ] Chris Egerton commented on KAFKA-14443: --- Merged and backported the fix to the 3.4

[GitHub] [kafka] cmccabe opened a new pull request, #12964: MINOR: Introduce MetadataProvenance and ImageReWriter

2022-12-07 Thread GitBox
cmccabe opened a new pull request, #12964: URL: https://github.com/apache/kafka/pull/12964 Introduce MetadataProvenance to encapsulate the three-tuple of (offset, epoch, timestamp) that is associated with each MetadataImage, as well as each on-disk snapshot. Remove offset and epoch t

[GitHub] [kafka] jsancio commented on a diff in pull request #12963: MINOR: More consistent handling of snapshot IDs

2022-12-07 Thread GitBox
jsancio commented on code in PR #12963: URL: https://github.com/apache/kafka/pull/12963#discussion_r1042716470 ## core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala: ## @@ -261,7 +261,7 @@ class BrokerMetadataPublisher( if (_firstPublish) { f

[GitHub] [kafka] cmccabe commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
cmccabe commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042759981 ## clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java: ## @@ -66,10 +75,14 @@ public LeaderAndIsrRequest build(short version) {

[GitHub] [kafka] cmccabe commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
cmccabe commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042760995 ## clients/src/main/resources/common/message/EnvelopeRequest.json: ## @@ -16,10 +16,11 @@ { "apiKey": 58, "type": "request", - "listeners": ["controller"], + "

[GitHub] [kafka] cmccabe commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
cmccabe commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042763145 ## core/src/main/scala/kafka/server/BrokerToControllerChannelManager.scala: ## @@ -37,42 +38,55 @@ import scala.collection.Seq import scala.compat.java8.OptionConverte

[GitHub] [kafka] cmccabe commented on a diff in pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
cmccabe commented on code in PR #12961: URL: https://github.com/apache/kafka/pull/12961#discussion_r1042763929 ## core/src/main/scala/kafka/server/BrokerToControllerChannelManager.scala: ## @@ -164,25 +184,63 @@ class BrokerToControllerChannelManagerImpl( private val logConte

[GitHub] [kafka] cmccabe commented on pull request #12961: KAFKA-14446: API forwarding support from zkBrokers to the Controller

2022-12-07 Thread GitBox
cmccabe commented on PR #12961: URL: https://github.com/apache/kafka/pull/12961#issuecomment-1341714694 I think the better approach to take here would be to make the `ControllerNodeProvider` return a tuple of `(isKraft, controller id)` and then have it check the metadata cache to see if it

[GitHub] [kafka] hachikuji commented on a diff in pull request #12862: Consumer refator find coordinator

2022-12-07 Thread GitBox
hachikuji commented on code in PR #12862: URL: https://github.com/apache/kafka/pull/12862#discussion_r1042773463 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java: ## @@ -0,0 +1,303 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [kafka] philipnee commented on a diff in pull request #12862: Consumer refator find coordinator

2022-12-07 Thread GitBox
philipnee commented on code in PR #12862: URL: https://github.com/apache/kafka/pull/12862#discussion_r1042792698 ## clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorResponse.java: ## @@ -50,6 +52,22 @@ public FindCoordinatorResponse(FindCoordinatorResponseDa

[GitHub] [kafka] hachikuji commented on a diff in pull request #12945: KAFKA-14365: Refactor Fetcher to allow different implementations

2022-12-07 Thread GitBox
hachikuji commented on code in PR #12945: URL: https://github.com/apache/kafka/pull/12945#discussion_r1042793295 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetsFinder.java: ## @@ -0,0 +1,805 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [kafka] mumrah opened a new pull request, #12965: KAFKA-14448 Let ZK brokers register with KRaft controller

2022-12-07 Thread GitBox
mumrah opened a new pull request, #12965: URL: https://github.com/apache/kafka/pull/12965 Prior to starting a KIP-866 migration, the ZK brokers must register themselves with the active KRaft controller. The controller waits for all brokers to register in order to verify that all the brokers

[GitHub] [kafka] kirktrue commented on a diff in pull request #12945: KAFKA-14365: Refactor Fetcher to allow different implementations

2022-12-07 Thread GitBox
kirktrue commented on code in PR #12945: URL: https://github.com/apache/kafka/pull/12945#discussion_r1042805066 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetsFinder.java: ## @@ -0,0 +1,805 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [kafka] jsancio commented on a diff in pull request #12963: MINOR: More consistent handling of snapshot IDs

2022-12-07 Thread GitBox
jsancio commented on code in PR #12963: URL: https://github.com/apache/kafka/pull/12963#discussion_r1042809968 ## core/src/test/scala/unit/kafka/server/metadata/BrokerMetadataListenerTest.scala: ## @@ -166,15 +172,15 @@ class BrokerMetadataListenerTest { image = newImage

[GitHub] [kafka] jolshan opened a new pull request, #12966: KAFKA-14417: Producer doesn't handle REQUEST_TIMED_OUT for InitProducerIdRequest, treats as fatal error -- server-side

2022-12-07 Thread GitBox
jolshan opened a new pull request, #12966: URL: https://github.com/apache/kafka/pull/12966 To handle the server side, we will leave it to the client to timeout and retry forever (long max) for now. I've also configured the poll on the array blocking queue to this value. In the future

[GitHub] [kafka] jsancio commented on a diff in pull request #12963: MINOR: More consistent handling of snapshot IDs

2022-12-07 Thread GitBox
jsancio commented on code in PR #12963: URL: https://github.com/apache/kafka/pull/12963#discussion_r1042828301 ## core/src/test/scala/unit/kafka/utils/TestUtils.scala: ## @@ -1230,7 +1230,7 @@ object TestUtils extends Logging { TestUtils.waitUntilTrue( () => {

[GitHub] [kafka] ableegoldman commented on a diff in pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
ableegoldman commented on code in PR #12935: URL: https://github.com/apache/kafka/pull/12935#discussion_r1042855507 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java: ## @@ -526,6 +521,9 @@ public synchronized void close() { fOptions.cl

[GitHub] [kafka] ableegoldman commented on a diff in pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
ableegoldman commented on code in PR #12935: URL: https://github.com/apache/kafka/pull/12935#discussion_r1042856605 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java: ## @@ -526,6 +521,9 @@ public synchronized void close() { fOptions.cl

[GitHub] [kafka] dengziming commented on pull request #12797: MINOR: Remove requirement to specify --bootstrap-server in utility scripts if specified in property file

2022-12-07 Thread GitBox
dengziming commented on PR #12797: URL: https://github.com/apache/kafka/pull/12797#issuecomment-1341885319 > This part definitely needs one because we introduce a new command line argument. Yes, I agree with you, we can't add/delete/change arguments without a KIP. in this PR, I think

[GitHub] [kafka] ableegoldman commented on a diff in pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
ableegoldman commented on code in PR #12935: URL: https://github.com/apache/kafka/pull/12935#discussion_r1042857583 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java: ## @@ -229,40 +230,34 @@ void openDB(final Map configs, final File stateDir)

[GitHub] [kafka] ableegoldman commented on pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
ableegoldman commented on PR #12935: URL: https://github.com/apache/kafka/pull/12935#issuecomment-1341889792 I did look over this and it seems reasonable but I'll refer to @cadonna's evaluation here as the expert on RocksDB metrics/statistics. Since he approved and the build passed (modulo

[jira] [Created] (KAFKA-14454) KTableKTableForeignKeyInnerJoinCustomPartitionerIntegrationTest#shouldThrowIllegalArgumentExceptionWhenCustomPartionerReturnsMultiplePartitions passes when run individua

2022-12-07 Thread Sagar Rao (Jira)
Sagar Rao created KAFKA-14454: - Summary: KTableKTableForeignKeyInnerJoinCustomPartitionerIntegrationTest#shouldThrowIllegalArgumentExceptionWhenCustomPartionerReturnsMultiplePartitions passes when run individually but not when is run as part of the I

[GitHub] [kafka] ableegoldman merged pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
ableegoldman merged PR #12935: URL: https://github.com/apache/kafka/pull/12935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.ap

[GitHub] [kafka] ableegoldman commented on pull request #12935: KAFKA-14432: RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread GitBox
ableegoldman commented on PR #12935: URL: https://github.com/apache/kafka/pull/12935#issuecomment-1341892785 Merged to trunk and cherrypicked to 3.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [kafka] ableegoldman merged pull request #12960: MINOR: Bump trunk to 3.5.0-SNAPSHOT

2022-12-07 Thread GitBox
ableegoldman merged PR #12960: URL: https://github.com/apache/kafka/pull/12960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.ap

[jira] [Resolved] (KAFKA-14432) RocksDBStore relies on finalizers to not leak memory

2022-12-07 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-14432. Resolution: Fixed > RocksDBStore relies on finalizers to not leak memory >

[GitHub] [kafka] ableegoldman commented on pull request #12955: KAFKA-14443: Close topic creation Admin clients in MM2 connectors

2022-12-07 Thread GitBox
ableegoldman commented on PR #12955: URL: https://github.com/apache/kafka/pull/12955#issuecomment-1341916089 Thanks guys, appreciate you splitting out the refactoring even though it's a hassle (and maybe doesn't feel as good) 🙏 -- This is an automated message from the Apache Git Service.

[GitHub] [kafka] philipnee commented on a diff in pull request #12862: Consumer refator find coordinator

2022-12-07 Thread GitBox
philipnee commented on code in PR #12862: URL: https://github.com/apache/kafka/pull/12862#discussion_r1042900099 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java: ## @@ -0,0 +1,303 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [kafka] philipnee commented on a diff in pull request #12862: Consumer refator find coordinator

2022-12-07 Thread GitBox
philipnee commented on code in PR #12862: URL: https://github.com/apache/kafka/pull/12862#discussion_r1042905241 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/DefaultBackgroundThread.java: ## @@ -50,57 +49,83 @@ public class DefaultBackgroundThread extends

[GitHub] [kafka] ableegoldman commented on a diff in pull request #12962: KAFKA-14318: KIP-878, Introduce partition autoscaling configs

2022-12-07 Thread GitBox
ableegoldman commented on code in PR #12962: URL: https://github.com/apache/kafka/pull/12962#discussion_r1042911388 ## streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/AssignmentTestUtils.java: ## @@ -112,6 +113,56 @@ public final class AssignmentTes

[jira] [Commented] (KAFKA-12886) Enable request forwarding by default

2022-12-07 Thread Deng Ziming (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644601#comment-17644601 ] Deng Ziming commented on KAFKA-12886: - KAFKA-14446 will enable forwarding for migrat

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042930128 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042935917 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042936427 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042935917 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042942666 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042943030 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r104291 ## docs/connect.html: ## @@ -593,6 +701,107 @@ Resuming from Previous Off Of course, you might need to read many keys for each of the input streams. The Offset

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042945330 ## docs/connect.html: ## @@ -593,6 +701,107 @@ Resuming from Previous Off Of course, you might need to read many keys for each of the input streams. The Offset

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042946060 ## docs/connect.html: ## @@ -593,6 +701,107 @@ Resuming from Previous Off Of course, you might need to read many keys for each of the input streams. The Offset

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042946621 ## docs/connect.html: ## @@ -593,6 +701,107 @@ Resuming from Previous Off Of course, you might need to read many keys for each of the input streams. The Offset

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042947164 ## docs/connect.html: ## @@ -369,6 +369,114 @@ Error Reportin # Tolerate all errors. errors.tolerance=all +Exactly-once support + +Kafka Connect is capabl

[GitHub] [kafka] tikimims commented on a diff in pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on code in PR #12941: URL: https://github.com/apache/kafka/pull/12941#discussion_r1042949185 ## connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedConfig.java: ## @@ -232,9 +232,12 @@ public String toString() { public

[GitHub] [kafka] tikimims commented on pull request #12941: KAFKA-13709: Add docs for exactly-once support in Connect

2022-12-07 Thread GitBox
tikimims commented on PR #12941: URL: https://github.com/apache/kafka/pull/12941#issuecomment-1342083368 Hi @C0urante I took a first pass. :) Let me know if you'd like me to take another pass after any updates. 👍 -- This is an automated message from the Apache Git Service. To respond to

  1   2   >