[GitHub] [kafka] showuon commented on pull request #10811: KAFKA-12598: remove zookeeper support on configCommand except security config

showuon commented on pull request #10811: URL: https://github.com/apache/kafka/pull/10811#issuecomment-871984330 @ijuma , I've updated the PR: 1. remove the support to update quota related broker configs using ZooKeeper. 2. require `zk-tls-config-file` option when `zookeeper` is provid

[GitHub] [kafka] cmccabe merged pull request #10949: KAFKA-13019: Add MetadataImage and MetadataDelta classes for KRaft Snapshots

cmccabe merged pull request #10949: URL: https://github.com/apache/kafka/pull/10949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr.

[GitHub] [kafka] ayoukhananov commented on a change in pull request #10951: KAFKA-12841: NPE from the provided metadata in client callback in case of ApiException

ayoukhananov commented on a change in pull request #10951: URL: https://github.com/apache/kafka/pull/10951#discussion_r662049082 ## File path: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java ## @@ -982,8 +982,12 @@ private void throwIfProducerClosed(

[GitHub] [kafka] ayoukhananov commented on a change in pull request #10951: KAFKA-12841: NPE from the provided metadata in client callback in case of ApiException

ayoukhananov commented on a change in pull request #10951: URL: https://github.com/apache/kafka/pull/10951#discussion_r662049082 ## File path: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java ## @@ -982,8 +982,12 @@ private void throwIfProducerClosed(

[GitHub] [kafka] showuon commented on a change in pull request #10940: KAFKA-13007: KafkaAdminClient getListOffsetsCalls reuse cluster snapshot

showuon commented on a change in pull request #10940: URL: https://github.com/apache/kafka/pull/10940#discussion_r662073522 ## File path: clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java ## @@ -4200,6 +4200,7 @@ public ListOffsetsResult listOffsets(Ma

[GitHub] [kafka] showuon commented on pull request #10940: KAFKA-13007: KafkaAdminClient getListOffsetsCalls reuse cluster snapshot

showuon commented on pull request #10940: URL: https://github.com/apache/kafka/pull/10940#issuecomment-872027365 > this will reduce the time complexity from O(n^2) to O(n). I think we reduce the time complexity from O(n) to O(1). FYI~ -- This is an automated message from the Apache

[GitHub] [kafka] showuon edited a comment on pull request #10940: KAFKA-13007: KafkaAdminClient getListOffsetsCalls reuse cluster snapshot

showuon edited a comment on pull request #10940: URL: https://github.com/apache/kafka/pull/10940#issuecomment-872027365 > this will reduce the time complexity from O(n^2) to O(n). I think we reduce the time complexity from O(n) to O(1), suppose n is the partition size. FYI~ -- Thi

[GitHub] [kafka] mimaison merged pull request #10652: KAFKA-9726 IdentityReplicationPolicy

mimaison merged pull request #10652: URL: https://github.com/apache/kafka/pull/10652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr

[GitHub] [kafka] showuon commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

showuon commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662082332 ## File path: raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java ## @@ -495,6 +506,10 @@ public int sizeInBytes() { public

[GitHub] [kafka] mimaison commented on pull request #10652: KAFKA-9726 IdentityReplicationPolicy

mimaison commented on pull request #10652: URL: https://github.com/apache/kafka/pull/10652#issuecomment-872037198 Thanks @ryannedolan, @mdedetrich and @ivanyu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [kafka] showuon commented on a change in pull request #10938: Fix StringSerializer could not be found when it not in ContextClassLoader

showuon commented on a change in pull request #10938: URL: https://github.com/apache/kafka/pull/10938#discussion_r662092163 ## File path: clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java ## @@ -725,14 +725,21 @@ else if (value instanceof String)

[jira] [Commented] (KAFKA-13009) Metrics recorder is re-initialised with different task

[ https://issues.apache.org/jira/browse/KAFKA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372587#comment-17372587 ] Bruno Cadonna commented on KAFKA-13009: --- Hi [~VictorvandenHoven]! Thank you for fi

[jira] [Commented] (KAFKA-13002) listOffsets must downgrade immediately for non MAX_TIMESTAMP specs

[ https://issues.apache.org/jira/browse/KAFKA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372615#comment-17372615 ] Bruno Cadonna commented on KAFKA-13002: --- [~dajac] I deployed the soak cluster that

[GitHub] [kafka] OmniaGM commented on pull request #10937: KAFKA-10587 Rename kafka-mirror-maker CLI command line arguments for KIP-629

OmniaGM commented on pull request #10937: URL: https://github.com/apache/kafka/pull/10937#issuecomment-872098303 Hi, @xvrl, @mimaison can you please have a look into this? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[jira] [Commented] (KAFKA-13002) listOffsets must downgrade immediately for non MAX_TIMESTAMP specs

[ https://issues.apache.org/jira/browse/KAFKA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372621#comment-17372621 ] David Jacot commented on KAFKA-13002: - [~cadonna] Sounds good, thanks. > listOffset

[GitHub] [kafka] showuon commented on a change in pull request #10932: KAFKA-12958: add an invariant that notified leaders are never asked to load snapshot

showuon commented on a change in pull request #10932: URL: https://github.com/apache/kafka/pull/10932#discussion_r662158289 ## File path: raft/src/main/java/org/apache/kafka/raft/ReplicatedCounter.java ## @@ -157,13 +160,18 @@ public synchronized void handleSnapshot(SnapshotRe

[GitHub] [kafka] cadonna commented on a change in pull request #10943: Fix verification of version probing

cadonna commented on a change in pull request #10943: URL: https://github.com/apache/kafka/pull/10943#discussion_r662163821 ## File path: tests/kafkatest/tests/streams/streams_upgrade_test.py ## @@ -474,7 +474,7 @@ def do_rolling_bounce(self, processor, counter, current_genera

[GitHub] [kafka] cadonna commented on a change in pull request #10944: MINOR: Loose verification of startup in EOS system tests

cadonna commented on a change in pull request #10944: URL: https://github.com/apache/kafka/pull/10944#discussion_r662160463 ## File path: tests/kafkatest/tests/streams/streams_eos_test.py ## @@ -166,7 +166,6 @@ def abort_streams(self, keep_alive_processor1, keep_alive_processo

[GitHub] [kafka] showuon commented on a change in pull request #10932: KAFKA-12958: add an invariant that notified leaders are never asked to load snapshot

showuon commented on a change in pull request #10932: URL: https://github.com/apache/kafka/pull/10932#discussion_r662155569 ## File path: raft/src/test/java/org/apache/kafka/raft/RaftEventSimulationTest.java ## @@ -1014,6 +1015,26 @@ public void verify() { } } +

[GitHub] [kafka] showuon commented on a change in pull request #10932: KAFKA-12958: add an invariant that notified leaders are never asked to load snapshot

showuon commented on a change in pull request #10932: URL: https://github.com/apache/kafka/pull/10932#discussion_r662157582 ## File path: raft/src/main/java/org/apache/kafka/raft/ReplicatedCounter.java ## @@ -157,13 +160,18 @@ public synchronized void handleSnapshot(SnapshotRe

[jira] [Commented] (KAFKA-13002) listOffsets must downgrade immediately for non MAX_TIMESTAMP specs

[ https://issues.apache.org/jira/browse/KAFKA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372639#comment-17372639 ] Bruno Cadonna commented on KAFKA-13002: --- The soak cluster still looks good and the

[GitHub] [kafka] cadonna edited a comment on pull request #10943: Fix verification of version probing

cadonna edited a comment on pull request #10943: URL: https://github.com/apache/kafka/pull/10943#issuecomment-872127063 @showuon Agreed! All of us stumbled upon this. To be fair, usually we forget to adapt the test, but this time the author remembered to modify it, but missed to change one

[GitHub] [kafka] cadonna edited a comment on pull request #10943: Fix verification of version probing

cadonna edited a comment on pull request #10943: URL: https://github.com/apache/kafka/pull/10943#issuecomment-872127063 @showuon Agreed! All of us stumbled upon this. To be fair, usually we forget to adapt the test, but this time the author remembered to modify it, but missed to change one

[GitHub] [kafka] cadonna commented on pull request #10943: Fix verification of version probing

cadonna commented on pull request #10943: URL: https://github.com/apache/kafka/pull/10943#issuecomment-872127063 @showuon Agreed! All of us stumbled upon this. To be fair, usually we forget to adapt the test, but this time the author remembered to modified it, but missed to change one numb

[GitHub] [kafka] tombentley commented on pull request #9878: KAFKA-6987: Add KafkaFuture.toCompletionStage()

tombentley commented on pull request #9878: URL: https://github.com/apache/kafka/pull/9878#issuecomment-872183699 @dajac @chia7712 (and let me try @rajinisivaram on the off-chance), I'd like to get this into Kafka 3.0, but this hasn't seen any review yet despite repeated requests. Any chan

[GitHub] [kafka] ryannedolan commented on pull request #10805: KAFKA-12436 KIP-720 Deprecate MirrorMaker v1

ryannedolan commented on pull request #10805: URL: https://github.com/apache/kafka/pull/10805#issuecomment-872203006 @ijuma can we merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Commented] (KAFKA-13002) listOffsets must downgrade immediately for non MAX_TIMESTAMP specs

[ https://issues.apache.org/jira/browse/KAFKA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372740#comment-17372740 ] David Jacot commented on KAFKA-13002: - Awesome, thanks! > listOffsets must downgrad

[GitHub] [kafka] OmniaGM commented on pull request #10937: KAFKA-10587 Rename kafka-mirror-maker CLI command line arguments for KIP-629

OmniaGM commented on pull request #10937: URL: https://github.com/apache/kafka/pull/10937#issuecomment-872226653 > MirrorMakerIntegrationTest @showuon Thanks for having a look, the pr already includes `MirrorMakerIntegrationTest.scala`. I did a quick search and from the look of it,

[GitHub] [kafka] OmniaGM edited a comment on pull request #10937: KAFKA-10587 Rename kafka-mirror-maker CLI command line arguments for KIP-629

OmniaGM edited a comment on pull request #10937: URL: https://github.com/apache/kafka/pull/10937#issuecomment-872226653 > Overall LGTM. Could you please also update the test file for MirrorMaker? (i.e. MirrorMakerIntegrationTest.scala) Thanks. @showuon Thanks for having a look, the p

[jira] [Assigned] (KAFKA-10588) Rename kafka-console-consumer CLI command line arguments for KIP-629

[ https://issues.apache.org/jira/browse/KAFKA-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim reassigned KAFKA-10588: - Assignee: Omnia Ibrahim (was: Xavier Léauté) > Rename kafka-console-consumer CLI comma

[jira] [Assigned] (KAFKA-10589) Rename kafka-replica-verification CLI command line arguments for KIP-629

[ https://issues.apache.org/jira/browse/KAFKA-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim reassigned KAFKA-10589: - Assignee: Omnia Ibrahim (was: Xavier Léauté) > Rename kafka-replica-verification CLI c

[GitHub] [kafka] mimaison commented on pull request #10743: KIP-699: Update FindCoordinator to resolve multiple Coordinators at a time

mimaison commented on pull request #10743: URL: https://github.com/apache/kafka/pull/10743#issuecomment-872252096 I've rebased on trunk to resolve conflicts and I've had to make small changes due to updates on trunk. All new changes are in https://github.com/apache/kafka/pull/10743/commits

[jira] [Created] (KAFKA-13023) make "range, cooperative-sticky" as the default assignor in V3.0

Luke Chen created KAFKA-13023: - Summary: make "range, cooperative-sticky" as the default assignor in V3.0 Key: KAFKA-13023 URL: https://issues.apache.org/jira/browse/KAFKA-13023 Project: Kafka I

[GitHub] [kafka] rondagostino commented on pull request #10811: KAFKA-12598: remove zookeeper support on configCommand except security config

rondagostino commented on pull request #10811: URL: https://github.com/apache/kafka/pull/10811#issuecomment-872258415 > require zk-tls-config-file option when zookeeper is provided I think this may be too much of a constraint. While it is true that these operations should be using T

[GitHub] [kafka] rondagostino commented on pull request #10811: KAFKA-12598: remove zookeeper support on configCommand except security config

rondagostino commented on pull request #10811: URL: https://github.com/apache/kafka/pull/10811#issuecomment-872262617 > I think this may be too much of a constraint One counterargument to this is that the bootstrap use case is a pretty unusual one and would very likely be utilized by

[GitHub] [kafka] ijuma commented on a change in pull request #10805: KAFKA-12436 KIP-720 Deprecate MirrorMaker v1

ijuma commented on a change in pull request #10805: URL: https://github.com/apache/kafka/pull/10805#discussion_r662307716 ## File path: core/src/test/scala/integration/kafka/tools/MirrorMakerIntegrationTest.scala ## @@ -35,6 +35,9 @@ import org.junit.jupiter.api.Test import o

[GitHub] [kafka] ijuma commented on pull request #9497: KAFKA-10619: Producer will enable idempotence and acks all by default

ijuma commented on pull request #9497: URL: https://github.com/apache/kafka/pull/9497#issuecomment-872270984 @d8tltanc Merged trunk and ran the tests here https://github.com/apache/kafka/pull/10950, failures are unrelated: > Build / JDK 8 and Scala 2.12 / kafka.server.RaftClusterTes

[GitHub] [kafka] ijuma commented on pull request #9497: KAFKA-10619: Producer will enable idempotence and acks all by default

ijuma commented on pull request #9497: URL: https://github.com/apache/kafka/pull/9497#issuecomment-872271275 @d8tltanc Can you please file a JIRA for the follow-up work? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [kafka] ijuma commented on a change in pull request #9497: KAFKA-10619: Producer will enable idempotence and acks all by default

ijuma commented on a change in pull request #9497: URL: https://github.com/apache/kafka/pull/9497#discussion_r662317902 ## File path: clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java ## @@ -300,7 +300,7 @@ .define(RET

[GitHub] [kafka] ijuma merged pull request #9497: KAFKA-10619: Producer will enable idempotence and acks all by default

ijuma merged pull request #9497: URL: https://github.com/apache/kafka/pull/9497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@k

[GitHub] [kafka] ijuma edited a comment on pull request #9497: KAFKA-10619: Configure producer with idempotence and acks all by default (KIP-679)

ijuma edited a comment on pull request #9497: URL: https://github.com/apache/kafka/pull/9497#issuecomment-872271275 @d8tltanc Can you please file a JIRA for the follow-up work? 1. Adjust config priority 2. Adjust the JUnit tests so we get good coverage of the non default behavior

[GitHub] [kafka] eshizhan commented on a change in pull request #10938: Fix StringSerializer could not be found when it not in ContextClassLoader

eshizhan commented on a change in pull request #10938: URL: https://github.com/apache/kafka/pull/10938#discussion_r662351754 ## File path: clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java ## @@ -725,14 +725,21 @@ else if (value instanceof String)

[GitHub] [kafka] zhaohaidao commented on a change in pull request #10932: KAFKA-12958: add an invariant that notified leaders are never asked to load snapshot

zhaohaidao commented on a change in pull request #10932: URL: https://github.com/apache/kafka/pull/10932#discussion_r662377755 ## File path: raft/src/test/java/org/apache/kafka/raft/RaftEventSimulationTest.java ## @@ -1014,6 +1015,26 @@ public void verify() { } }

[GitHub] [kafka] zhaohaidao commented on a change in pull request #10932: KAFKA-12958: add an invariant that notified leaders are never asked to load snapshot

zhaohaidao commented on a change in pull request #10932: URL: https://github.com/apache/kafka/pull/10932#discussion_r662378094 ## File path: raft/src/main/java/org/apache/kafka/raft/ReplicatedCounter.java ## @@ -157,13 +160,18 @@ public synchronized void handleSnapshot(Snapsho

[GitHub] [kafka] ijuma closed pull request #10950: Test https://github.com/warrenzhu25/kafka/tree/10619

ijuma closed pull request #10950: URL: https://github.com/apache/kafka/pull/10950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...

[GitHub] [kafka] ijuma commented on pull request #10950: Test https://github.com/warrenzhu25/kafka/tree/10619

ijuma commented on pull request #10950: URL: https://github.com/apache/kafka/pull/10950#issuecomment-872329748 The original PR was merged so we can close this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [kafka] ijuma commented on pull request #10935: KAFKA-13003: In kraft mode also advertise configured advertised port instead of socket port

ijuma commented on pull request #10935: URL: https://github.com/apache/kafka/pull/10935#issuecomment-872330571 @ueisele quick heads up that most committers are pretty busy this week due to the 3.0 feature freeze. We should be able to review and merge this next week. -- This is an automat

[GitHub] [kafka] zhaohaidao commented on pull request #10932: KAFKA-12958: add an invariant that notified leaders are never asked to load snapshot

zhaohaidao commented on pull request #10932: URL: https://github.com/apache/kafka/pull/10932#issuecomment-872330694 > @zhaohaidao , thanks for the PR. Left some comments. Thanks. Thanks for your advice. Comments have been addressed. Please continue to review if you have time. -- T

[GitHub] [kafka] xvrl commented on a change in pull request #10937: KAFKA-10587 Rename kafka-mirror-maker CLI command line arguments for KIP-629

xvrl commented on a change in pull request #10937: URL: https://github.com/apache/kafka/pull/10937#discussion_r662425926 ## File path: core/src/main/scala/kafka/tools/MirrorMaker.scala ## @@ -454,8 +454,8 @@ object MirrorMaker extends Logging with KafkaMetricsGroup { .of

[jira] [Resolved] (KAFKA-12952) Metadata Snapshot File Delimiters

[ https://issues.apache.org/jira/browse/KAFKA-12952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio resolved KAFKA-12952. Fix Version/s: 3.0.0 Resolution: Fixed > Metadata Snapshot

[jira] [Resolved] (KAFKA-12863) Configure controller snapshot generation

[ https://issues.apache.org/jira/browse/KAFKA-12863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio resolved KAFKA-12863. Fix Version/s: 3.0.0 Resolution: Fixed > Configure controll

[jira] [Assigned] (KAFKA-12646) Implement snapshot generation on brokers

[ https://issues.apache.org/jira/browse/KAFKA-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio reassigned KAFKA-12646: -- Assignee: Colin McCabe (was: Jose Armando Garcia Sancio) > I

[jira] [Created] (KAFKA-13024) Kafka Streams is dropping messages with null key during repartition

Damien Gasparina created KAFKA-13024: Summary: Kafka Streams is dropping messages with null key during repartition Key: KAFKA-13024 URL: https://issues.apache.org/jira/browse/KAFKA-13024 Project:

[jira] [Updated] (KAFKA-13024) Kafka Streams is dropping messages with null key during repartition

[ https://issues.apache.org/jira/browse/KAFKA-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Gasparina updated KAFKA-13024: - Description: {{KStream.repartition}} is silently filtering messages with null keys. A si

[GitHub] [kafka] vamossagar12 commented on pull request #10798: KAFKA-9168: Adding direct byte buffer support to rocksdb state store

vamossagar12 commented on pull request #10798: URL: https://github.com/apache/kafka/pull/10798#issuecomment-872398600 @guozhangwang , thats the commit i was referring to. Setting a byte order and seems to have an impact on performance. Also, earlier i was calling put first and then calling

[GitHub] [kafka] vamossagar12 commented on pull request #10798: KAFKA-9168: Adding direct byte buffer support to rocksdb state store

vamossagar12 commented on pull request #10798: URL: https://github.com/apache/kafka/pull/10798#issuecomment-872399579 > You were referring to this commit [68a947c](https://github.com/apache/kafka/commit/68a947c0eb6a5cc4bdde24083c83f4638e708edb) as for the tweaks right? > > BTW it's

[jira] [Resolved] (KAFKA-12467) Implement QuorumController metadata snapshots

[ https://issues.apache.org/jira/browse/KAFKA-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-12467. -- Fix Version/s: 3.0.0 Resolution: Fixed > Implement QuorumController metadata snapshots

[GitHub] [kafka] OmniaGM commented on a change in pull request #10937: KAFKA-10587 Rename kafka-mirror-maker CLI command line arguments for KIP-629

OmniaGM commented on a change in pull request #10937: URL: https://github.com/apache/kafka/pull/10937#discussion_r662449984 ## File path: core/src/main/scala/kafka/tools/MirrorMaker.scala ## @@ -454,8 +454,8 @@ object MirrorMaker extends Logging with KafkaMetricsGroup {

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662469119 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -16,34 +16,41 @@ */ package kafka.raft -import java.io.File -import java.ni

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662470041 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -16,34 +16,41 @@ */ package kafka.raft -import java.io.File -import java.ni

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662470041 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -16,34 +16,41 @@ */ package kafka.raft -import java.io.File -import java.ni

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662470041 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -16,34 +16,41 @@ */ package kafka.raft -import java.io.File -import java.ni

[jira] [Commented] (KAFKA-13010) Flaky test org.apache.kafka.streams.integration.TaskMetadataIntegrationTest.shouldReportCorrectCommittedOffsetInformation()

[ https://issues.apache.org/jira/browse/KAFKA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372977#comment-17372977 ] Bruno Cadonna commented on KAFKA-13010: --- Some logs that might be interesting: {co

[jira] [Comment Edited] (KAFKA-13010) Flaky test org.apache.kafka.streams.integration.TaskMetadataIntegrationTest.shouldReportCorrectCommittedOffsetInformation()

[ https://issues.apache.org/jira/browse/KAFKA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372977#comment-17372977 ] Bruno Cadonna edited comment on KAFKA-13010 at 7/1/21, 6:01 PM: --

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662493849 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -312,26 +315,142 @@ final class KafkaMetadataLog private ( } } - over

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662494186 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -312,26 +313,152 @@ final class KafkaMetadataLog private ( } } - over

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662498528 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -312,26 +315,142 @@ final class KafkaMetadataLog private ( } } - over

[jira] [Commented] (KAFKA-12841) NPE from the provided metadata in client callback in case of ApiException

[ https://issues.apache.org/jira/browse/KAFKA-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372984#comment-17372984 ] Kirk True commented on KAFKA-12841: --- There are some conditions inside the {{KafkaProdu

[GitHub] [kafka] mumrah commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

mumrah commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662505337 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -312,26 +315,142 @@ final class KafkaMetadataLog private ( } } - overr

[GitHub] [kafka] mumrah commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

mumrah commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662506190 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -312,26 +315,142 @@ final class KafkaMetadataLog private ( } } - overr

[GitHub] [kafka] jeffkbkim commented on pull request #10940: KAFKA-13007: KafkaAdminClient getListOffsetsCalls reuse cluster snapshot

jeffkbkim commented on pull request #10940: URL: https://github.com/apache/kafka/pull/10940#issuecomment-872460573 i have renamed cluster() to buildCluster(). i will submit a separate PR for the benchmark -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662509722 ## File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java ## @@ -2153,14 +2154,36 @@ private boolean maybeCompleteShutdown(long current

[jira] [Comment Edited] (KAFKA-12841) NPE from the provided metadata in client callback in case of ApiException

[ https://issues.apache.org/jira/browse/KAFKA-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372989#comment-17372989 ] Avi Youkhananov edited comment on KAFKA-12841 at 7/1/21, 6:31 PM:

[jira] [Commented] (KAFKA-12841) NPE from the provided metadata in client callback in case of ApiException

[ https://issues.apache.org/jira/browse/KAFKA-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372989#comment-17372989 ] Avi Youkhananov commented on KAFKA-12841: - [~kirktrue] This is what i did in thi

[GitHub] [kafka] jeffkbkim commented on pull request #10940: KAFKA-13007: KafkaAdminClient getListOffsetsCalls reuse cluster snapshot

jeffkbkim commented on pull request #10940: URL: https://github.com/apache/kafka/pull/10940#issuecomment-872467503 @showuon thanks! on the time complexity, i think you may have missed the for-loop wrapped around. -- This is an automated message from the Apache Git Service. To res

[GitHub] [kafka] mumrah commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

mumrah commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662517293 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -16,34 +16,41 @@ */ package kafka.raft -import java.io.File -import java.nio

[GitHub] [kafka] mumrah commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

mumrah commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662517391 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -16,34 +16,41 @@ */ package kafka.raft -import java.io.File -import java.nio

[GitHub] [kafka] kirktrue commented on pull request #10951: KAFKA-12841: NPE from the provided metadata in client callback in case of ApiException

kirktrue commented on pull request #10951: URL: https://github.com/apache/kafka/pull/10951#issuecomment-872470098 @ayoukhananov - you're absolutely correct that there are conditions wherein `tp` wasn't initialized. This should fix that case too. -- This is an automated message from the A

[jira] [Assigned] (KAFKA-12841) NPE from the provided metadata in client callback in case of ApiException

[ https://issues.apache.org/jira/browse/KAFKA-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True reassigned KAFKA-12841: - Assignee: Kirk True > NPE from the provided metadata in client callback in case of ApiException

[GitHub] [kafka] cmccabe commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

cmccabe commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662522112 ## File path: core/src/main/scala/kafka/raft/KafkaMetadataLog.scala ## @@ -312,26 +315,142 @@ final class KafkaMetadataLog private ( } } - over

[jira] [Commented] (KAFKA-12841) NPE from the provided metadata in client callback in case of ApiException

[ https://issues.apache.org/jira/browse/KAFKA-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373002#comment-17373002 ] Kirk True commented on KAFKA-12841: --- I have submitted a [pull request|https://github.c

[GitHub] [kafka] jolshan opened a new pull request #10952: KAFKA-12257: Consumer mishandles topics deleted and recreated with the same name

jolshan opened a new pull request #10952: URL: https://github.com/apache/kafka/pull/10952 Store topic ID info in consumer metadata. We will always take the topic ID from the latest metadata response and remove any topic IDs from the cache if the metadata response did not return a topic ID

[jira] [Updated] (KAFKA-12952) Metadata Snapshot File Delimiters

[ https://issues.apache.org/jira/browse/KAFKA-12952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niket Goel updated KAFKA-12952: --- Description: Create new Control Records that will serve as the header and footer for a Metadata Sna

[jira] [Created] (KAFKA-13025) Add more information to the Snapshot header and footer

Niket Goel created KAFKA-13025: -- Summary: Add more information to the Snapshot header and footer Key: KAFKA-13025 URL: https://issues.apache.org/jira/browse/KAFKA-13025 Project: Kafka Issue Type

[GitHub] [kafka] garyrussell commented on pull request #7670: KAFKA-7016: Not hide the stack trace for ApiException

garyrussell commented on pull request #7670: URL: https://github.com/apache/kafka/pull/7670#issuecomment-872514693 Can we PLEASE get this merged? It is incredibly difficult to track down problems in code that calls the clients, if we can't tell where the call is made from. -- This is an

[GitHub] [kafka] jsancio commented on a change in pull request #10864: KAFKA-12155 Metadata log and snapshot cleaning

jsancio commented on a change in pull request #10864: URL: https://github.com/apache/kafka/pull/10864#discussion_r662550101 ## File path: core/src/main/scala/kafka/server/KafkaConfig.scala ## @@ -1048,9 +1058,14 @@ object KafkaConfig { .defineInternal(InitialBrokerRegist

[GitHub] [kafka] mimaison merged pull request #10743: KIP-699: Update FindCoordinator to resolve multiple Coordinators at a time

mimaison merged pull request #10743: URL: https://github.com/apache/kafka/pull/10743 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr

[jira] [Created] (KAFKA-13026) Idempotent producer (KAFKA-10619) follow-up

Cheng Tan created KAFKA-13026: - Summary: Idempotent producer (KAFKA-10619) follow-up Key: KAFKA-13026 URL: https://issues.apache.org/jira/browse/KAFKA-13026 Project: Kafka Issue Type: Improvemen

[GitHub] [kafka] jsancio commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

jsancio commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662594464 ## File path: raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java ## @@ -495,6 +506,10 @@ public int sizeInBytes() { public

[GitHub] [kafka] hachikuji merged pull request #10940: KAFKA-13007: KafkaAdminClient getListOffsetsCalls reuse cluster snapshot

hachikuji merged pull request #10940: URL: https://github.com/apache/kafka/pull/10940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubsc

[jira] [Updated] (KAFKA-13025) Add more information to the Snapshot header and footer

[ https://issues.apache.org/jira/browse/KAFKA-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio updated KAFKA-13025: --- Labels: kip-500 (was: ) > Add more information to the Snapshot head

[GitHub] [kafka] cmccabe commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

cmccabe commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662599796 ## File path: metadata/src/test/java/org/apache/kafka/metalog/LocalLogManager.java ## @@ -198,7 +207,12 @@ synchronized void unregisterLogManager(LocalLog

[GitHub] [kafka] junrao merged pull request #10896: KAFKA-12964: Collect and rename snapshot files prior to async deletion.

junrao merged pull request #10896: URL: https://github.com/apache/kafka/pull/10896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr..

[GitHub] [kafka] cmccabe commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

cmccabe commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662600721 ## File path: raft/src/main/java/org/apache/kafka/raft/Batch.java ## @@ -99,38 +116,68 @@ public boolean equals(Object o) { Batch batch = (Batch)

[jira] [Resolved] (KAFKA-12964) Corrupt segment recovery can delete new producer state snapshots

[ https://issues.apache.org/jira/browse/KAFKA-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-12964. - Fix Version/s: 3.0.0 Resolution: Fixed merged the PR to trunk > Corrupt segment recovery can del

[GitHub] [kafka] cmccabe commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

cmccabe commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662601954 ## File path: raft/src/main/java/org/apache/kafka/raft/ReplicatedCounter.java ## @@ -106,7 +108,10 @@ public synchronized void handleCommit(BatchReader r

[GitHub] [kafka] cmccabe commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

cmccabe commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662602197 ## File path: raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java ## @@ -207,6 +208,12 @@ private void completeCurrentBatch() {

[GitHub] [kafka] jsancio commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

jsancio commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662604574 ## File path: raft/src/main/java/org/apache/kafka/raft/ReplicatedCounter.java ## @@ -106,7 +108,10 @@ public synchronized void handleCommit(BatchReader r

[GitHub] [kafka] jsancio commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

jsancio commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662608275 ## File path: raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java ## @@ -207,6 +208,12 @@ private void completeCurrentBatch() {

[GitHub] [kafka] jsancio commented on a change in pull request #10946: KAFKA-12997: Expose the append time for batches from raft

jsancio commented on a change in pull request #10946: URL: https://github.com/apache/kafka/pull/10946#discussion_r662608352 ## File path: raft/src/main/java/org/apache/kafka/raft/Batch.java ## @@ -99,38 +116,68 @@ public boolean equals(Object o) { Batch batch = (Batch)

  1   2   >