Build failed in Jenkins: kafka-trunk-jdk11 #966

See Changes: [cmccabe] [MINOR] allow additional JVM args in KafkaService (#7297) [jason] MINOR: Small logging fixes in AbstractCoordinator (#7230) [github] KAFKA-9198; Complete purgatory operations on receiving

[jira] [Created] (KAFKA-9210) kafka stream loss data

panpan.liu created KAFKA-9210: - Summary: kafka stream loss data Key: KAFKA-9210 URL: https://issues.apache.org/jira/browse/KAFKA-9210 Project: Kafka Issue Type: Bug Affects Versions: 2.0.1

Build failed in Jenkins: kafka-2.4-jdk8 #78

See Changes: [jason] MINOR: Fetch only from leader should be respected in purgatory (#7650) [jason] KAFKA-9198; Complete purgatory operations on receiving StopReplica -- [..

Re: [VOTE] KIP-518: Allow listing consumer groups per state

+1 (non-binding) I can see this being useful. Thanks for the KIP! Regards, Kevin On Mon, Nov 18, 2019 at 4:09 AM Mickael Maison wrote: > Hi all, > > I'd like to start the vote on KIP-518: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+state >

Build failed in Jenkins: kafka-trunk-jdk8 #4052

See Changes: [jason] MINOR: Small logging fixes in AbstractCoordinator (#7230) [github] KAFKA-9198; Complete purgatory operations on receiving StopReplica [matthias] HOTFIX: Fix unit tests that failed when exec

Re: ClientsMetricsTest.shouldAddCommitIdMetric() failed on RC0 ...

Bruno, I tested using the 2.4.0 release candidate 0 artifacts. These were uploaded as part of seeking the open Kafka community feedback. On Mon, Nov 18, 2019 at 12:24 PM Bruno Cadonna wrote: > Hi Vahid and Eric, > > Thank you for your input. > > I suppose you both used the archive of the releas

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

Hi Jason, Thanks for looking into the KIP. Apologies for my late reply. Increasing replica max lag to 30-45 secs did not help as we observed that a few fetch requests took more than 1-2 minutes. We do not want to increase further as it increases upper bound on commit latency. We have strict SLAs on

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Jun, Thanks for your detailed review and comments. >40. Local segment metadata storage: The KIP makes the assumption that the metadata for the archived log segments are cached locally in every broker and provides a specific implementation for the local storage in the framework. We probably shou

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

hi, I reviewed the current code, the ProduceMetadata maintains an expiry threshold for every topic, every time when we write to a topic we will set the expiry time to -1 to indicate it should be updated, this does work to reduce the size of the topic working set, but the producer will continue fetc

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

General mechanism for fetching metrics via RPC sounds like a good idea. Especially since Kafka has clients in many languages, but support for JMX is not wide-spread outside the Java ecosystem. Command-line script seems like a bad fit for use-cases where you want the clients themselves to use metric

Build failed in Jenkins: kafka-trunk-jdk8 #4051

See Changes: [cmccabe] [MINOR] allow additional JVM args in KafkaService (#7297) -- [...truncated 5.54 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldAll

[jira] [Created] (KAFKA-9209) Avoid sending unnecessary offset updates from consumer after KIP-211

Michael Bingham created KAFKA-9209: -- Summary: Avoid sending unnecessary offset updates from consumer after KIP-211 Key: KAFKA-9209 URL: https://issues.apache.org/jira/browse/KAFKA-9209 Project: Kafka

Re: [kafka-clients] [VOTE] 2.4.0 RC0

Fix via https://github.com/apache/kafka/pull/7707 Thanks for the quick fix Bruno! On 11/18/19 11:02 AM, Eric Lalonde wrote: > This test has been failing when executed from the command line. I have not > run this test in an IDE. > >> On Nov 18, 2019, at 6:16 AM, Bruno Cadonna wrote: >> >> Hi

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

Hi, In general, metrics are a "broker status API" telling you important things about the state of the broker, its performance, etc. etc., right? What argument is there for creating a separate API for this particular metric? If you argue that JMX is not convenient, it seems like that would als

[jira] [Resolved] (KAFKA-9198) StopReplica handler should complete pending purgatory operations

[ https://issues.apache.org/jira/browse/KAFKA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-9198. Fix Version/s: 2.4.0 Resolution: Fixed The PR has been merged. I meant to change the

[jira] [Resolved] (KAFKA-8571) Not complete delayed produce requests when processing StopReplicaRequest causing high produce latency for acks=all

[ https://issues.apache.org/jira/browse/KAFKA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-8571. Resolution: Fixed > Not complete delayed produce requests when processing StopReplicaReques

[jira] [Reopened] (KAFKA-9198) StopReplica handler should complete pending purgatory operations

[ https://issues.apache.org/jira/browse/KAFKA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reopened KAFKA-9198: > StopReplica handler should complete pending purgatory operations > --

ClientsMetricsTest.shouldAddCommitIdMetric() failed on RC0 ...

Hi Vahid and Eric, Thank you for your input. I suppose you both used the archive of the release candidate and did not checkout the tag from the git repository. I found the issue. The archive misses the .git directory that is needed for the unit test to pass. Opened the following PR to fix it: h

Build failed in Jenkins: kafka-trunk-jdk8 #4050

See Changes: [jason] MINOR: Add missing parentheses in docs/streams/tutorial.html (#7696) [github] KAFKA-9180: Introduce BrokerMetadataCheckpointTest (#7700) -- [...trun

[jira] [Created] (KAFKA-9208) Flaky Test SslAdminClientIntegrationTest.testCreatePartitions

Sophie Blee-Goldman created KAFKA-9208: -- Summary: Flaky Test SslAdminClientIntegrationTest.testCreatePartitions Key: KAFKA-9208 URL: https://issues.apache.org/jira/browse/KAFKA-9208 Project: Kafk

[jira] [Created] (KAFKA-9207) Replica Out of Sync as ReplicaFetcher thread is dead

Xue Liu created KAFKA-9207: -- Summary: Replica Out of Sync as ReplicaFetcher thread is dead Key: KAFKA-9207 URL: https://issues.apache.org/jira/browse/KAFKA-9207 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

On Mon, Nov 18, 2019, at 10:05, Brian Byrne wrote: > On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe wrote: > > > Two seconds doesn't seem like a reasonable amount of time to leave for the > > metadata fetch. Fetching halfway through the expiration period seems more > > reasonable. It also doesn't

[DISCUSS] KIP-547: Extend ConsumerInterceptor to allow modification of Consumer Commits

Hi all, I'd like to open discussion on KIP-547: Extend ConsumerInterceptor to allow modification of Consumer Commits This KIP hopes to enable better access to the Metadata included while committing offsets. LINK: https://cwiki.apache.org/confluence/display/KAFKA/KIP-547%3A+Extend+ConsumerInterce

Re: [kafka-clients] [VOTE] 2.4.0 RC0

This test has been failing when executed from the command line. I have not run this test in an IDE. > On Nov 18, 2019, at 6:16 AM, Bruno Cadonna wrote: > > Hi, > > ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed > from an IDE. The test fails because the test expects a f

[jira] [Created] (KAFKA-9206) Consumer should handle `CORRUPT_MESSAGE` error code in fetch response

Jason Gustafson created KAFKA-9206: -- Summary: Consumer should handle `CORRUPT_MESSAGE` error code in fetch response Key: KAFKA-9206 URL: https://issues.apache.org/jira/browse/KAFKA-9206 Project: Kafk

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

Hi Noa, Re; uptime. Sure, it was just a suggestion. However, we should be clear that there are actually two timestamps at play. Your initial proposal suggested using the timestamp from the registration znode. However, this changes each time the broker loses its session. It does not reflect the act

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

Thanks, everyone involved! On Mon, Nov 18, 2019 at 7:51 AM John Roesler wrote: > Thanks to you, also, Navinder! > > Looking forward to getting this feature in. > -John > > On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar > wrote: > > > > Hello all, > > > > With 4 binding +1 votes from Guozhang W

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe wrote: > Two seconds doesn't seem like a reasonable amount of time to leave for the > metadata fetch. Fetching halfway through the expiration period seems more > reasonable. It also doesn't require us to create a new configuration key, > which is nic

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

Here's a summary that can go under the section "What’s new in Kafka broker, producer, and consumer" as an "improvement". Feel free to rephrase as you see fit. When a partition is paused by the user in the consumer the partition is > considered "unfetchable". When the consumer has already fetched

[jira] [Created] (KAFKA-9205) Add an option to enforce rack-aware partition reassignment

Vahid Hashemian created KAFKA-9205: -- Summary: Add an option to enforce rack-aware partition reassignment Key: KAFKA-9205 URL: https://issues.apache.org/jira/browse/KAFKA-9205 Project: Kafka

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

+1 (binding) Thanks for the KIP On Mon, Nov 18, 2019 at 3:54 PM Sean Glover wrote: > > +1 (non-binding) > > Good idea. It will streamline the Streams Scala DSL nicely. It will also > affect 2.11 users of embedded-kafka. Are there any other non-broker > dependencies that could be affected? > >

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

That makes sense to me. Ismael On Mon, Nov 18, 2019 at 8:40 AM Sean Glover wrote: > Hi Manikumar, > > I'm putting together an akka.io blog post regarding [KAFKA-7548] - > KafkaConsumer should not throw away already fetched data for paused > partitions. Since it doesn't change any user-facing A

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

Hi Manikumar, I'm putting together an akka.io blog post regarding [KAFKA-7548] - KafkaConsumer should not throw away already fetched data for paused partitions. Since it doesn't change any user-facing APIs it has no KIP, but it has a significant impact on consumer use cases that frequently pause

Re: [kafka-clients] [VOTE] 2.4.0 RC0

Thanks Bruno, Just to clarify, I ran the tests from the command line: ./gradlew streams:test Regards, --Vahid On Mon, Nov 18, 2019 at 6:16 AM Bruno Cadonna wrote: > Hi, > > ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed > from an IDE. The test fails because the test exp

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

Thanks Chris. will update the blog content. On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton wrote: > Hi Manikumar, > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend Connect > Converter to support headers") but the content appears to correspond to > KIP-481 ("SerDe Improvement

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

+1 (non-binding) Good idea. It will streamline the Streams Scala DSL nicely. It will also affect 2.11 users of embedded-kafka. Are there any other non-broker dependencies that could be affected? Sean On Mon, Nov 18, 2019 at 10:43 AM Ismael Juma wrote: > Yes, everyone is encouraged to vote.

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

Thanks to you, also, Navinder! Looking forward to getting this feature in. -John On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar wrote: > > Hello all, > > With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill Bejeck, > and John Roesler, the vote passes. > Thanks Guozhang, Matthias, B

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

Thanks, Ismael. +1 (binding) John On Mon, Nov 18, 2019 at 9:43 AM Ismael Juma wrote: > > Yes, everyone is encouraged to vote. Committer votes are binding, but we > are interested in what the wider community thinks too. > > Ismael > > On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko > wrote: > >

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

Hi Noa, I understand the desire for batching. However, once you do that, you either need request forwarding or broker metadata propagation. It's worth exploring, but I suspect you will end up with most of the changes needed by the original proposal, in that case. Ismael On Mon, Nov 18, 2019 at 7

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

Yes, everyone is encouraged to vote. Committer votes are binding, but we are interested in what the wider community thinks too. Ismael On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko wrote: > Do I understand correctly, that non-commiters can also vote, despite their > votes don't decide? > > If

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

Hi Jason & Gwen, Sure, this would solve our use case. I do have two questions, however: Moving from start time to uptime in theory would neatly side step the clock skew problem, but in practice I’m not sure it helps that much as the straightforward way of going about implementing this by return

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

+1 (binding) On Mon, Nov 18, 2019 at 9:30 AM Satish Duggana wrote: > +1 (non-binding), Thanks Ismael for the KIP. I believe it is time to > drop Scala 2.11. > > > On Mon, Nov 18, 2019 at 7:10 PM Ivan Yurchenko > wrote: > > > > Do I understand correctly, that non-commiters can also vote, despite

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

+1 (non-binding), Thanks Ismael for the KIP. I believe it is time to drop Scala 2.11. On Mon, Nov 18, 2019 at 7:10 PM Ivan Yurchenko wrote: > > Do I understand correctly, that non-commiters can also vote, despite their > votes don't decide? > > If so, then +1 from me. > > Ivan > > > On Mon, 18 N

Re: [kafka-clients] [VOTE] 2.4.0 RC0

Hi, ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed from an IDE. The test fails because the test expects a file on the class path which is not there when the test is executed from the IDE, but is there when the test is executed from gradle. I will try to fix the test so that

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

Do I understand correctly, that non-commiters can also vote, despite their votes don't decide? If so, then +1 from me. Ivan On Mon, 18 Nov 2019 at 15:19, Ismael Juma wrote: > Hi all, > > People seemed supportive in general, so I'd like to start a vote on > KIP-531: > > > https://cwiki.apache.

[VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

Hi all, People seemed supportive in general, so I'd like to start a vote on KIP-531: https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5 Ismael

[VOTE] KIP-518: Allow listing consumer groups per state

Hi all, I'd like to start the vote on KIP-518: https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+state Thanks

[jira] [Created] (KAFKA-9204) ReplaceField transformation fails when encountering tombstone event

Georgios Kalogiros created KAFKA-9204: - Summary: ReplaceField transformation fails when encountering tombstone event Key: KAFKA-9204 URL: https://issues.apache.org/jira/browse/KAFKA-9204 Project:

Build failed in Jenkins: kafka-2.4-jdk8 #77

See Changes: [ismael] KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704) -- [...truncated 2.69 MB...] org.apache.kafka.streams.test.OutputVerifierTest > sho

[jira] [Created] (KAFKA-9203) kafka-client 2.3.1 fails to consume lz4 compressed topic in kafka 2.1.1

David Watzke created KAFKA-9203: --- Summary: kafka-client 2.3.1 fails to consume lz4 compressed topic in kafka 2.1.1 Key: KAFKA-9203 URL: https://issues.apache.org/jira/browse/KAFKA-9203 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk8 #4049

See Changes: [ismael] KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704) -- [...truncated 2.74 MB...] org.apache.kafka.streams.test.TestRecordTest > tes