[jira] [Created] (KAFKA-8226) New MirrorMaker option partition.to.partition

Ernestas Vaiciukevičius created KAFKA-8226: -- Summary: New MirrorMaker option partition.to.partition Key: KAFKA-8226 URL: https://issues.apache.org/jira/browse/KAFKA-8226 Project: Kafka

Re: Stream caching

Hello, yes I know it, I read about rocks db. But do we have ability to make some size limitation for this db? Because we have alerts in grafana, that application is using more then 80% of full app memory. I’ll be very grateful for the help. Thanks. Kohut Yaroslav > On Apr 12, 2019, at 4:02 A

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Rajini, I've made a number of changes to the KIP. 1. I've added RequestedTopicState.requestedConfigs(). This is obviously unrelated to supporting alter broker, but I think it goes some way to addressing one of the points Anna made last year. Anna, wdyt? 2. I've added BrokerState, RequestedBro

Re: Stream caching

And I have a lot of warning that describe in this lira https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc Thanks. Kohut Yaroslav > On Apr 12, 2019, at 10:59 AM, Ярослав Когут wrote: > > Hello, yes I know

[DISCUSS] KIP-452: Tool to view cluster status

Dear all, I would like to start discussion of this KIP aimed at providing command-line tool to display basic information about active Kafka brokers. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-452%3A+Tool+to+view+cluster+status JIRA: https://issues.apache.org/jira/browse/KAFKA-6393

Re: Proposal to Auto Close Inactive Tickets

Hi Addison, in general, I am totally in favor of closing unnecessary tickets! However, over the past few weeks, I have spent quite a bit of time looking at old tickets and evaluating whether those should be closed or are still relevant. For a surprising number of these tickets, I've found that the

Re: Stream caching

Hello Kohut, Is your issue about disk usage, or memory? The first email seems to indicate disks (there's no direct way for bounding the disk usage of state stores, though you can set e.g. retention period to indirectly control disk usage via specifying how long do you want to retain windowed store

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

Just a reminder that KIP-411 is open for voting. No votes received yet! On Fri, Apr 5, 2019 at 9:07 AM Paul Davidson wrote: > Hi all, > > Since we seem to have agreement in the discussion I would like to start > the vote on KIP-411. > > See: > https://cwiki.apache.org/confluence/display/KAFKA/KI

Re: Stream caching

Hello Guozhang! I’m talking about disk usage, about /tmp/kafka-streams. My application merges 3 topics, one of them has 120 partitions. As i understood from documentation(https://docs.confluent.io/current/streams/sizing.html), disk usage depends on partition count. And I really want to decreas

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

Hi everyone. Just a reminder that KIP-411 is now open for voting. No votes received yet! Thanks, Paul On Thu, Apr 4, 2019 at 9:09 AM pdavidson wrote: > Thanks Randall. You're absolutely right that Worker creates the clients > before passing them to the tasks, so I'm very happy with your chang

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

On Thu, Apr 11, 2019 at 2:10 PM Sophie Blee-Goldman wrote: > Thanks for the comments Guozhang! I've answered your questions below > > On Tue, Apr 9, 2019 at 4:38 PM Guozhang Wang wrote: > > > Hi Sophie, > > > > Thanks for the proposed KIP. I've made a pass over it and here are some > > thoughts:

Re: [VOTE] KIP-417: Allow JmxTool to connect to a secured RMI port

+1 (binding). Thanks. Li On 2019/01/29 01:37:43, "Fangbin Sun" wrote: > Hi, All:> > I would like to start a vote on KIP-417 which aims at supporting JmxTool to > connect to a secured RMI port.> > > > The KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-417%3A+Allow+JmxToo

Re: [VOTE] KIP-417: Allow JmxTool to connect to a secured RMI port

+1 (binding). Thanks. Li On 2019/02/01 16:20:11, Harsha Chintalapani wrote: > +1 (binding).> > > Thanks,> > Harsha> > On Jan 31, 2019, 8:08 PM -0800, Manikumar , wrote:> > > Hi,> > >> > > +1 (binding). Thanks for the KIP.> > >> > > On Mon, Jan 28, 2019 at 5:37 PM Fangbin Sun wrote:>

[jira] [Created] (KAFKA-8227) Missing Links in Duality of Streams and Tables section

Bill Bejeck created KAFKA-8227: -- Summary: Missing Links in Duality of Streams and Tables section Key: KAFKA-8227 URL: https://issues.apache.org/jira/browse/KAFKA-8227 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

Thanks for the reminder. +1 (non-binding) Ryanne On Fri, Apr 12, 2019, 11:13 AM Paul Davidson wrote: > Hi everyone. Just a reminder that KIP-411 is now open for voting. No votes > received yet! > > Thanks, > > Paul > > On Thu, Apr 4, 2019 at 9:09 AM pdavidson wrote: > > > Thanks Randall. You

Re: [VOTE] KIP-433: Block old clients on brokers

Hi Ismael, Those are just examples. I think the administrators should be able to block certain client libraries for whatever reason. Some other possible reasons include, force users to check pointing in Kafka instead of zookeeper, forbid an old go (sarama) client library which is known to have som

Re: [DISCUSS] KIP-449: Add connector contexts to Connect worker logs

Thanks for the KIP Randall. It might not be obvious right away, but this is a great improvement when running Connect with multiple connectors or when debugging Connect, for instance in integration tests or system tests. KIP looks good to me overall, I have just a few comments below: 1. In the sni

Re: [DISCUSS] KIP-437: Custom replacement for MaskField SMT

Thanks for the proposal and the PR request, Valeria. I've been working with Valeria on this PR, so it's not surprising I'm in favor of this KIP and improvement so that users can easily mask values with specific replacement values with the same primitive type. This proposal has been open for over

Re: [VOTE] KIP-433: Block old clients on brokers

Hi Ying, The actual reasons are important so that people can evaluate the KIP (and vote). :) Thanks for providing a few more: (1) force users to check pointing in Kafka instead of zookeeper (2) forbid an old go (sarama) client library which is known to have some serious bugs (3) force kafka 1.x c

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

Thanks for the KIP Sophie. I have a couple of additional comments. The current proposal only considers stream-time. While I support this, each time we introduce a new operation based on stream-time, invariably users request that operation support wall-clock time as well. Would we want to consid

[VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi All, This KIP is in discussion for more than a month. The feedback is positive without any objection comments. Therefore, I would like to start voting. https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter R

Re: [DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

Matthias, Are you sure the users are aware that with `withLoggingDisabled()`, they might lose data during failover? OK, we maybe do not necessarily need a WARN log. However, I would at least add a comment like in `StoreBuilder`,ie, /** * Disable the changelog for store built by this {@link Store

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi Mateusz, The KIP looks good. Just a few of questions/suggestions: 1. It would be helpful to see an example of the output with everything enabled. 2. What are the default values for the properties (eg what's the default header separator). 3. What is the separator used between key/value and the

Re: [DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

Thanks for the KIP Maarten. I also agree that keeping the `withLoggingDisabled()` and `withLoggingEnabled(Map)` methods is the better option. When it comes to educating the users on the downside of disabling logging, IMHO I think a comment in the JavaDoc should be sufficient. -Bill On Fri, Apr

Re: [DISCUSS] KIP-449: Add connector contexts to Connect worker logs

Thanks for the review and feedback, Konstantine. 1. Great suggestion. I've updated the KIP to hopefully make it more clear that the uncommented line is unchanged from the existing Log4J configuration file. 2. Regarding including a `-` before the task number is acceptable if it makes it easier to,

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

+1 (non binding) Thanks Ryanne On Fri, Apr 12, 2019, 11:11 AM Paul Davidson wrote: > Just a reminder that KIP-411 is open for voting. No votes received yet! > > On Fri, Apr 5, 2019 at 9:07 AM Paul Davidson > wrote: > > > Hi all, > > > > Since we seem to have agreement in the discussion I would

Re: [VOTE] KIP-445: In-memory session store

Hi all, This KIP passes with three +1 (binding) -- thanks Matthias, Bill, and Guozhang! For those interested, feel free to check out and join the ongoing PR review here: https://github.com/apache/kafka/pull/6525 Cheers, Sophie On Wed, Apr 10, 2019 at 1:21 PM Guozhang Wang wrote: > +1 (binding

[DISCUSS] 2.3.0 release

Hi all, I'd like to volunteer to be the release manager for the our next feature release, the 2.3 release. If that sounds good, I'll post a release plan on Monday. cheers, Colin

Re: [DISCUSS] 2.3.0 release

Thanks for volunteering Colin, +1. Ismael On Sat, Apr 13, 2019 at 2:05 AM Colin McCabe wrote: > Hi all, > > I'd like to volunteer to be the release manager for the our next feature > release, the 2.3 release. > > If that sounds good, I'll post a release plan on Monday. > > cheers, > Colin >

Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams

Hi Guozhang, Responses below: 2. The interface's default implementation will just be > `onPartitionRevoked`, so for user's instantiation if they do not make any > code changes they should be able to recompile the code and continue. Ack, makes sense. 4. Hmm.. not sure if it will work. The main

Re: [VOTE] KIP-433: Block old clients on brokers

Hi, "Relying on min.version seems like a pretty clunky way to achieve the above > list. The challenge is that it's pretty difficult to do it in a way that > works for clients across languages. They each add support for new protocol > versions independently (it could even happen in a bug fix

Re: [VOTE] KIP-433: Block old clients on brokers

Hi Harsha, There is no such thing as 1.1 protocol. I encourage you to describe an example config that achieves what you are suggesting here. It's pretty complicated because the versions are per API and each client evolves independently. Ismael On Sat, Apr 13, 2019 at 4:09 AM Harsha wrote: > Hi

Re: [VOTE] KIP-433: Block old clients on brokers

Hi Ismael, I meant to say blocking clients based on their API version https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48 But If I understand what you are saying, since each client release can support different versions for each of fetch, pr