Re: metric for total volume of messages per topic over a period of time

Hey there Amitav, thanks for the proposal. Just to be a bit more concrete on the proposed metrics, you would like to be able to see what the total bytes-in amount was for a given time period (e.g 60 seconds)? Do you mind sharing what the use case of such a metric is? This would require opening up

Build failed in Jenkins: kafka-trunk-jdk11 #221

See Changes: [junrao] MINOR: log when controller begins processing logdir failure event -- [...truncated 2.27 MB...] org.apache.kafka.connect.header.ConnectHeadersTest >

Build failed in Jenkins: kafka-trunk-jdk8 #3324

See Changes: [junrao] MINOR: log when controller begins processing logdir failure event -- [...truncated 2.26 MB...] org.apache.kafka.streams.test.OutputVerifierTest > s

Build failed in Jenkins: kafka-trunk-jdk11 #220

See Changes: [junrao] KAFKA-7829; Javadoc should show that AdminClient.alterReplicaLogDirs() -- [...truncated 2.27 MB...] org.apache.kafka.connect.json.JsonConverterTest

Build failed in Jenkins: kafka-trunk-jdk8 #3323

See Changes: [junrao] KAFKA-7829; Javadoc should show that AdminClient.alterReplicaLogDirs() -- [...truncated 2.26 MB...] org.apache.kafka.streams.test.OutputVerifierTest

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Jan 15, 2019, at 2:26 PM, Colin McCabe wrote: > > I think it makes sense to go back to use-cases again. So far, all of the > use-cases we discussed could be handled by pause and resume. So it makes > sense to try to figure out what the issue with those APIs is. Are they not > well-do

[jira] [Created] (KAFKA-7837) maybeShrinkIsr may not reflect OfflinePartitions immediately

Jun Rao created KAFKA-7837: -- Summary: maybeShrinkIsr may not reflect OfflinePartitions immediately Key: KAFKA-7837 URL: https://issues.apache.org/jira/browse/KAFKA-7837 Project: Kafka Issue Type: I

[jira] [Created] (KAFKA-7838) improve logging in Partition.maybeShrinkIsr()

Jun Rao created KAFKA-7838: -- Summary: improve logging in Partition.maybeShrinkIsr() Key: KAFKA-7838 URL: https://issues.apache.org/jira/browse/KAFKA-7838 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-7836) The propagation of log dir failure can be delayed due to slowness in closing the file handles

Jun Rao created KAFKA-7836: -- Summary: The propagation of log dir failure can be delayed due to slowness in closing the file handles Key: KAFKA-7836 URL: https://issues.apache.org/jira/browse/KAFKA-7836 Proje

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hi Stanislav and Boyang. Thanks for your comments. You are both asking how KIP-345 affects this KIP, so, first, I'll address this point. KIP-345 has recently introduced an option that will allow Kafka consumer applications to avoid rebalances due to the departure and subsequently return of a memb

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hey folks, This KIP has start since a while but has never been merged. https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector Would it be possible to restart the vote ? I still think this KIP could be useful to implement connectors (the P

[VOTE] KIP-416: Notify SourceTask of ACK'd offsets, metadata

Hey y'all, please vote for KIP-416 by replying +1 to this thread. Right now, there is no way for a SourceConnector/Task to know: - whether a record was successfully sent to Kafka, vs filtered out or skipped. - the downstream offsets and metadata of sent records KIP-416 proposes adding a recordLo

[jira] [Resolved] (KAFKA-7829) Javadoc should show that AdminClient.alterReplicaLogDirs() is supported in Kafka 1.1.0 or later

[ https://issues.apache.org/jira/browse/KAFKA-7829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-7829. Resolution: Fixed Fix Version/s: 2.2.0 Merged to trunk. > Javadoc should show that AdminClient.alte

Re: [DISCUSS] Notify SourceTask of ACK'd offsets, metadata

I had to change the KIP number (concurrency is hard!) so the link is now: https://cwiki.apache.org/confluence/display/KAFKA/KIP-416%3A+Notify+SourceTask+of+ACK%27d+offsets%2C+metadata Ryanne On Fri, Jan 11, 2019 at 2:43 PM Ryanne Dolan wrote: > Hey y'all, > > Please review the following small

Re: [VOTE] KIP-307: Allow to define custom processor names with KStreams DSL

FWIW, I'm also +1 on the consistency changes to Joined. -John On Thu, Jan 17, 2019 at 5:04 PM Guozhang Wang wrote: > Hello Florian, > > Thanks for the writeup! I made a pass on the wiki page itself and left a > comment on it regarding the newly added APIs. Could you take a look? If > that makes

Re: [VOTE] KIP-307: Allow to define custom processor names with KStreams DSL

Hello Florian, Thanks for the writeup! I made a pass on the wiki page itself and left a comment on it regarding the newly added APIs. Could you take a look? If that makes sense could you update the wiki page to make sure we have all the relevant public API changes. Guozhang On Thu, Jan 17, 2019

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

+1 for me on Guozhang's proposal for changes to Joined. Thanks, Bill On Thu, Jan 17, 2019 at 5:55 PM Matthias J. Sax wrote: > Thanks for all the follow up comments! > > As I mentioned earlier, I am ok with adding overloads instead of using > Materialized to specify the processor name. Seems thi

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Thanks for all the follow up comments! As I mentioned earlier, I am ok with adding overloads instead of using Materialized to specify the processor name. Seems this is what the majority of people prefers. I am also +1 on Guozhang's suggestion to deprecate `static Joined#named()` and replace it wi

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Wow that's a lot of discussions in 6 days! :) Just catching up and sharing my two cents here: 1. Materialized: I'm inclined to not let Materialized extending Named and add the overload as well. All the rationales have been very well summarized before. Just to emphasize on John's points: Materializ

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Sounds good to me. Thanks, Bill On Thu, Jan 17, 2019 at 4:43 PM Florian Hussonnois wrote: > Sorry, I've sent my previous mail to quickly. Unlike the Consumed, Produced > and Grouped classes, the Joined class does have getter methods. So I > propose to keep the name() method only for this class.

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Sorry, I've sent my previous mail to quickly. Unlike the Consumed, Produced and Grouped classes, the Joined class does have getter methods. So I propose to keep the name() method only for this class. For other classes the name will be accessible through XXXInternal classes. Le jeu. 17 janv. 2019 à

Re: [VOTE] KIP-307: Allow to define custom processor names with KStreams DSL

Yes, thanks Florian! +1 (nonbinding) from me. -John On Thu, Jan 17, 2019 at 3:36 PM Bill Bejeck wrote: > Thanks for the KIP Florian, it will be a useful addition. > > +1 on the KIP for me. > > -Bill > > On Thu, Jan 17, 2019 at 4:31 PM Florian Hussonnois > wrote: > > > Hi folks, > > > > I woul

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Just to chime in regarding NamedInternal. That was my bad mental model to blame. It is indeed coercion, not casting. Even more relevant, I'm not a fan of the XInternal pattern, but it is the pattern we have. It would be worse to start carving out exceptions. So I agree that we should have: * `Name

Re: [VOTE] KIP-307: Allow to define custom processor names with KStreams DSL

Thanks for the KIP Florian, it will be a useful addition. +1 on the KIP for me. -Bill On Thu, Jan 17, 2019 at 4:31 PM Florian Hussonnois wrote: > Hi folks, > > I would like to initiate a vote for the following KIP : > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+defi

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Thanks for your feedback. So I will remove the name() method from the NamedOperation interface. After a first look, I will need to introduce a new class JoinedInternal Le jeu. 17 janv. 2019 à 19:09, Bill Bejeck a écrit : > I'm getting caught up with the current state of this KIP. > > I agree tha

[VOTE] KIP-307: Allow to define custom processor names with KStreams DSL

Hi folks, I would like to initiate a vote for the following KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL Note, there is still some minor discussions regarding the implementation. Thanks -- Florian HUSSONNOIS

[DISCUSS] KIP-400 - Improve exit status in case of errors in ConsoleProducer

Hi everyone, This is ready for discussion. *Jira* - https://issues.apache.org/jira/browse/KAFKA-6812 *KIP -* https://cwiki.apache.org/confluence/display/KAFKA/KIP-400:+Improve+exit+status+in+case+of+errors+in+ConsoleProducer Please let me know if there is anything else I am missing to start th

Build failed in Jenkins: kafka-trunk-jdk8 #3322

See Changes: [jason] KAFKA-7738; Track leader epochs in client Metadata (#6045) -- [...truncated 2.26 MB...] org.apache.kafka.streams.test.OutputVerifierTest > shouldPas

Build failed in Jenkins: kafka-trunk-jdk11 #219

See Changes: [jason] KAFKA-7738; Track leader epochs in client Metadata (#6045) -- [...truncated 2.27 MB...] org.apache.kafka.connect.transforms.MaskFieldTest > withSchem

[jira] [Resolved] (KAFKA-7738) Track partition leader epochs in client metadata

[ https://issues.apache.org/jira/browse/KAFKA-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-7738. Resolution: Fixed Fix Version/s: 2.2.0 > Track partition leader epochs in client met

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

I'm getting caught up with the current state of this KIP. I agree that the question on what to do with overloads is a difficult one to answer. Both John and Matthias have laid out their thoughts thoroughly, and the points made by both resonate with me. I've spent some time thinking about this, a

[jira] [Created] (KAFKA-7835) Kafka Connect - Unable to create multiple connectors in a single HTTP (REST) call

Suraj Fale created KAFKA-7835: - Summary: Kafka Connect - Unable to create multiple connectors in a single HTTP (REST) call Key: KAFKA-7835 URL: https://issues.apache.org/jira/browse/KAFKA-7835 Project: Ka

Build failed in Jenkins: kafka-2.1-jdk8 #107

See Changes: [me] MINOR: Start Connect REST server in standalone mode to match distributed -- [...truncated 912.04 KB...] kafka.server.KafkaConfigTest > testInterBrokerList

metric for total volume of messages per topic over a period of time

Hi I have a small proposal for improvement of Kafka that I would like to discuss with you. Kafka currently has rate metrics exposed per topic for the number of bytes ingested. I am interested in a volume metric. I would like to have direct answer to the question how much data is ingested in a top

Re: [VOTE] KIP-389: Introduce a configurable consumer group size limit

Thanks to everybody for the discussions and votes. This vote has passed with 3 binding votes (Ismael, Jason, Gwen) and 2 non-binding (Boyang, Andrew). I have opened the following PR with the changes implemented - https://github.com/apache/kafka/pull/6163 Thanks, Stanislav On Tue, Jan 15, 2019 at

Build failed in Jenkins: kafka-2.0-jdk8 #215

See Changes: [me] MINOR: Start Connect REST server in standalone mode to match distributed -- [...truncated 2.65 MB...] org.apache.kafka.connect.util.TopicAdminTest > should

[jira] [Resolved] (KAFKA-7520) Adding possibility to configure versions in Mirror Maker ducktape test

[ https://issues.apache.org/jira/browse/KAFKA-7520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Katona resolved KAFKA-7520. -- Resolution: Won't Fix > Adding possibility to configure versions in Mirror Maker ducktape test

Build failed in Jenkins: kafka-trunk-jdk11 #218

See Changes: [me] MINOR: Start Connect REST server in standalone mode to match distributed -- [...truncated 2.27 MB...] org.apache.kafka.connect.json.JsonConverterTest >

Build failed in Jenkins: kafka-trunk-jdk8 #3321

See Changes: [me] MINOR: Start Connect REST server in standalone mode to match distributed -- [...truncated 2.26 MB...] org.apache.kafka.streams.test.OutputVerifierTest >