[GitHub] [kafka] satishd commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broke

2021-10-05 Thread GitBox
satishd commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722923763 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/CommittedOffsetsFile.java ## @@ -0,0 +1,83 @@ +/* + * Licensed to

[GitHub] [kafka] satishd commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broke

2021-10-05 Thread GitBox
satishd commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722911577 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataSnapshotFile.java ## @@ -0,0 +1,253 @@ +/* + * L

[GitHub] [kafka] satishd commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broke

2021-10-05 Thread GitBox
satishd commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722884223 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/CommittedOffsetsFile.java ## @@ -0,0 +1,83 @@ +/* + * Licensed to

[GitHub] [kafka] mjsax commented on a change in pull request #11370: MINOR: remove unneeded size and add lock coarsening to inMemoryKeyValueStore

2021-10-05 Thread GitBox
mjsax commented on a change in pull request #11370: URL: https://github.com/apache/kafka/pull/11370#discussion_r722840836 ## File path: streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueStore.java ## @@ -98,9 +96,15 @@ public synchronized void put(

[GitHub] [kafka] mjsax commented on pull request #11355: KAFKA-13298: Improve documentation on EOS KStream requirements

2021-10-05 Thread GitBox
mjsax commented on pull request #11355: URL: https://github.com/apache/kafka/pull/11355#issuecomment-935259937 Thanks for the PR @cddr! Merged to `trunk`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [kafka] mjsax merged pull request #11355: KAFKA-13298: Improve documentation on EOS KStream requirements

2021-10-05 Thread GitBox
mjsax merged pull request #11355: URL: https://github.com/apache/kafka/pull/11355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...

[jira] [Updated] (KAFKA-13350) Handle task corrupted exception on a per state store basis

2021-10-05 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13350: Description: When we hit an `OffsetOutOfRangeException` during restore, we close a tasks a

[jira] [Created] (KAFKA-13350) Handle task corrupted exception on a per state store basis

2021-10-05 Thread Matthias J. Sax (Jira)
Matthias J. Sax created KAFKA-13350: --- Summary: Handle task corrupted exception on a per state store basis Key: KAFKA-13350 URL: https://issues.apache.org/jira/browse/KAFKA-13350 Project: Kafka

[GitHub] [kafka] satishd commented on pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broker restarts.

2021-10-05 Thread GitBox
satishd commented on pull request #11058: URL: https://github.com/apache/kafka/pull/11058#issuecomment-935124023 Thanks @junrao for the comments, added in-line replies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [kafka] satishd commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broke

2021-10-05 Thread GitBox
satishd commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722794018 ## File path: storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerRestartTest.java ## @@ -0,0 +1,

[GitHub] [kafka] satishd commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broke

2021-10-05 Thread GitBox
satishd commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722791508 ## File path: storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerRestartTest.java ## @@ -0,0 +1,

[GitHub] [kafka] kowshik commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broke

2021-10-05 Thread GitBox
kowshik commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722772977 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataSnapshotFile.java ## @@ -0,0 +1,253 @@ +/* + * L

[GitHub] [kafka] junrao commented on a change in pull request #11058: KAFKA-12802 Added a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broker

2021-10-05 Thread GitBox
junrao commented on a change in pull request #11058: URL: https://github.com/apache/kafka/pull/11058#discussion_r722776127 ## File path: storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerRestartTest.java ## @@ -0,0 +1,1

[GitHub] [kafka] cmccabe opened a new pull request #11383: KAFKA-13340: Change ZooKeeperTestHarness to QuorumTestHarness

2021-10-05 Thread GitBox
cmccabe opened a new pull request #11383: URL: https://github.com/apache/kafka/pull/11383 Change ZooKeeperTestHarness to QuorumTestHarness so that integration tests which inherit from this class can test both ZK and KRaft mode. Test cases which do this can specify the modes they support

[GitHub] [kafka] OmniaGM commented on a change in pull request #11220: KAFKA-10777: Add additional configuration to control MirrorMaker 2 internal topics naming convention

2021-10-05 Thread GitBox
OmniaGM commented on a change in pull request #11220: URL: https://github.com/apache/kafka/pull/11220#discussion_r722666580 ## File path: connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/ReplicationPolicy.java ## @@ -52,9 +52,39 @@ default String originalTop

[GitHub] [kafka] TheKnowles opened a new pull request #11382: KAFKA-13348: Allow Source Tasks to Handle Producer Exceptions

2021-10-05 Thread GitBox
TheKnowles opened a new pull request #11382: URL: https://github.com/apache/kafka/pull/11382 This change allows Source Tasks the option to receive Producer Exceptions and decide what to do with them. Currently the task is just killed if the producer throws an exception via its callback. Su

[jira] [Assigned] (KAFKA-13349) Allow Iterator.remove on KeyValueIterator

2021-10-05 Thread Jira
[ https://issues.apache.org/jira/browse/KAFKA-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bartłomiej Stachura reassigned KAFKA-13349: --- Assignee: Bartłomiej Stachura > Allow Iterator.remove on KeyValueIterator >

[jira] [Updated] (KAFKA-13349) Allow Iterator.remove on KeyValueIterator

2021-10-05 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13349: Labels: needs-kip newbie++ (was: newbie++) > Allow Iterator.remove on KeyValueIterator >

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13346: Component/s: streams > Kafka Streams fails due to RocksDB Locks Not Available Exception >

[jira] [Created] (KAFKA-13349) Allow Iterator.remove on KeyValueIterator

2021-10-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-13349: - Summary: Allow Iterator.remove on KeyValueIterator Key: KAFKA-13349 URL: https://issues.apache.org/jira/browse/KAFKA-13349 Project: Kafka Issue Type: Impro

[jira] [Commented] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424647#comment-17424647 ] A. Sophie Blee-Goldman commented on KAFKA-13346: +1 to increasing the op

[GitHub] [kafka] ijuma merged pull request #11374: MINOR: TopicIdPartition improvements

2021-10-05 Thread GitBox
ijuma merged pull request #11374: URL: https://github.com/apache/kafka/pull/11374 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...

[GitHub] [kafka] hachikuji commented on pull request #11372: [DO NOT MERGE] MINOR: Do not send data on abortable error too

2021-10-05 Thread GitBox
hachikuji commented on pull request #11372: URL: https://github.com/apache/kafka/pull/11372#issuecomment-934656281 @guozhangwang I've been trying to write a test case for this scenario, but so far I've been unsuccessful. I think in order to hit this case, we have to send to a new partition

[GitHub] [kafka] ableegoldman commented on a change in pull request #11381: KAFKA-12648: allow users to set a StreamsUncaughtExceptionHandler on individual named topologies

2021-10-05 Thread GitBox
ableegoldman commented on a change in pull request #11381: URL: https://github.com/apache/kafka/pull/11381#discussion_r722488122 ## File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java ## @@ -1280,29 +1281,36 @@ int process(final int m

[jira] [Assigned] (KAFKA-10865) Improve trace-logging for Transformations (including Predicates)

2021-10-05 Thread Randall Hauch (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch reassigned KAFKA-10865: - Assignee: Tiago Martins (was: Govinda Sakhare) > Improve trace-logging for Transformat

[jira] [Updated] (KAFKA-10865) Improve trace-logging for Transformations (including Predicates)

2021-10-05 Thread Randall Hauch (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch updated KAFKA-10865: -- [~upsidedownsmile], I've assigned this issue to you as requested. You are now listed as a contrib

[GitHub] [kafka] ableegoldman commented on a change in pull request #11381: KAFKA-12648: allow users to set a StreamsUncaughtExceptionHandler on individual named topologies

2021-10-05 Thread GitBox
ableegoldman commented on a change in pull request #11381: URL: https://github.com/apache/kafka/pull/11381#discussion_r722480165 ## File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java ## @@ -299,6 +301,7 @@ public boolean isRunning()

[jira] [Commented] (KAFKA-13164) State store is attached to wrong node in the Kafka Streams topology

2021-10-05 Thread Hao Li (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424590#comment-17424590 ] Hao Li commented on KAFKA-13164: Hi Ralph,   I actually can see output for both cases

[jira] [Assigned] (KAFKA-13348) Allow Source Tasks to Handle Producer Exceptions

2021-10-05 Thread Knowles Atchison Jr (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Knowles Atchison Jr reassigned KAFKA-13348: --- Assignee: Knowles Atchison Jr > Allow Source Tasks to Handle Producer Excep

[jira] [Created] (KAFKA-13348) Allow Source Tasks to Handle Producer Exceptions

2021-10-05 Thread Knowles Atchison Jr (Jira)
Knowles Atchison Jr created KAFKA-13348: --- Summary: Allow Source Tasks to Handle Producer Exceptions Key: KAFKA-13348 URL: https://issues.apache.org/jira/browse/KAFKA-13348 Project: Kafka

[GitHub] [kafka] wcarlson5 commented on a change in pull request #11381: KAFKA-12648: allow users to set a StreamsUncaughtExceptionHandler on individual named topologies

2021-10-05 Thread GitBox
wcarlson5 commented on a change in pull request #11381: URL: https://github.com/apache/kafka/pull/11381#discussion_r722408690 ## File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java ## @@ -1280,29 +1281,36 @@ int process(final int maxN

[jira] [Created] (KAFKA-13347) kafka-topics.sh required arguments and defaults

2021-10-05 Thread Scott M Messner (Jira)
Scott M Messner created KAFKA-13347: --- Summary: kafka-topics.sh required arguments and defaults Key: KAFKA-13347 URL: https://issues.apache.org/jira/browse/KAFKA-13347 Project: Kafka Issue T

[jira] [Commented] (KAFKA-10865) Improve trace-logging for Transformations (including Predicates)

2021-10-05 Thread Tiago Martins (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424480#comment-17424480 ] Tiago Martins commented on KAFKA-10865: --- Hello, I assume that [~govi20] is no long

[GitHub] [kafka] upsidedownsmile commented on pull request #11375: KAFKA-10865: Log transformed record in WorkerSinkTask

2021-10-05 Thread GitBox
upsidedownsmile commented on pull request #11375: URL: https://github.com/apache/kafka/pull/11375#issuecomment-934430195 @rhauch could you please review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[jira] [Commented] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread Bruno Cadonna (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424424#comment-17424424 ] Bruno Cadonna commented on KAFKA-13346: --- Hi [~amitguptaiitr85], Could you try to

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Gupta updated KAFKA-13346: --- Description: Hello, We are using Kafka Streams and we observe that some times on some of the hosts

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Gupta updated KAFKA-13346: --- Description: Hello, We are using Kafka Streams and we observe that some times on some of the hosts

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Gupta updated KAFKA-13346: --- Description: Hello, We are using Kafka Streams and we observe that some times on some of the hosts

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available exception

2021-10-05 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Gupta updated KAFKA-13346: --- Summary: Kafka Streams fails due to RocksDB Locks Not Available exception (was: Kafka Streams fails

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

2021-10-05 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Gupta updated KAFKA-13346: --- Summary: Kafka Streams fails due to RocksDB Locks Not Available Exception (was: Kafka Streams fails

[jira] [Updated] (KAFKA-13346) Kafka Streams fails due to unexpected exception

2021-10-05 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Gupta updated KAFKA-13346: --- Description: Hello, We are using Kafka Streams and we observe that some times on some of the hosts

[jira] [Created] (KAFKA-13346) Kafka Streams fails due to unexpected exception

2021-10-05 Thread Amit Gupta (Jira)
Amit Gupta created KAFKA-13346: -- Summary: Kafka Streams fails due to unexpected exception Key: KAFKA-13346 URL: https://issues.apache.org/jira/browse/KAFKA-13346 Project: Kafka Issue Type: Bug

[GitHub] [kafka] viktorsomogyi commented on pull request #11373: MINOR: Add shebang to gradlewAll

2021-10-05 Thread GitBox
viktorsomogyi commented on pull request #11373: URL: https://github.com/apache/kafka/pull/11373#issuecomment-934265796 @ijuma would you please quickly review this small pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [kafka] ableegoldman commented on pull request #11381: KAFKA-12648: allow users to set a StreamsUncaughtExceptionHandler on individual named topologies

2021-10-05 Thread GitBox
ableegoldman commented on pull request #11381: URL: https://github.com/apache/kafka/pull/11381#issuecomment-934171853 Haven't finished writing tests for this, but the non-testing code is ready for review -- cc @wcarlson5 @guozhangwang @rodesai -- This is an automated message from the Ap

[GitHub] [kafka] ableegoldman opened a new pull request #11381: KAFKA-12648: allow users to set a StreamsUncaughtExceptionHandler on individual named topologies

2021-10-05 Thread GitBox
ableegoldman opened a new pull request #11381: URL: https://github.com/apache/kafka/pull/11381 In the current exception handler, there's not much granularity and all exceptions from all sources are treated equally. With the introduction of independent named topologies, it would be nice to

[GitHub] [kafka] kowshik edited a comment on pull request #11345: Allow empty last segment to have missing offset index during recovery

2021-10-05 Thread GitBox
kowshik edited a comment on pull request #11345: URL: https://github.com/apache/kafka/pull/11345#issuecomment-934133766 @ccding During a call to `log.flush()`, we remember [here](https://github.com/apache/kafka/blob/1d3b96389b325520648d29b6363941f50e5b6d35/core/src/main/scala/kafka/log/Unif

[GitHub] [kafka] lkokhreidze commented on pull request #10851: KAFKA-6718 / Rack aware standby task assignor

2021-10-05 Thread GitBox
lkokhreidze commented on pull request #10851: URL: https://github.com/apache/kafka/pull/10851#issuecomment-934149747 Thanks for the feedback @cadonna I will review and address your comments this week. -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [kafka] kowshik commented on pull request #11345: Allow empty last segment to have missing offset index during recovery

2021-10-05 Thread GitBox
kowshik commented on pull request #11345: URL: https://github.com/apache/kafka/pull/11345#issuecomment-934133766 @ccding During a call to `log.flush()`, we remember [here](https://github.com/apache/kafka/blob/1d3b96389b325520648d29b6363941f50e5b6d35/core/src/main/scala/kafka/log/UnifiedLog.

[GitHub] [kafka] patrickstuedi commented on a change in pull request #11234: KAFKA-13212: add support infinite query for session store

2021-10-05 Thread GitBox
patrickstuedi commented on a change in pull request #11234: URL: https://github.com/apache/kafka/pull/11234#discussion_r721953611 ## File path: streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredSessionStore.java ## @@ -351,10 +345,8 @@ public V fetchSession