Re: [DISCUSS] KIP-133: List and Alter Configs Admin APIs

> On May 6, 2017, at 11:27 AM, Ismael Juma wrote: > > Hi James, > > Yes, that's right, it will return all config values. For topic configs, > that means falling back to the respective broker config value, which could > also be a default. If we fallback to the broker config (whether it's a > def

[GitHub] kafka pull request #2938: KAFKA-5096: Log invalid user configs and use defau...

Github user johnma14 closed the pull request at: https://github.com/apache/kafka/pull/2938 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[jira] [Commented] (KAFKA-5096) Only log invalid user configs and overwrite with correct one

[ https://issues.apache.org/jira/browse/KAFKA-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999731#comment-15999731 ] ASF GitHub Bot commented on KAFKA-5096: --- Github user johnma14 closed the pull reques

[GitHub] kafka pull request #2990: KAFKA-5096: Log invalid user configs and use defau...

GitHub user johnma14 opened a pull request: https://github.com/apache/kafka/pull/2990 KAFKA-5096: Log invalid user configs and use defaults Kafka Streams does not allow users to modify some consumer configurations. Currently, it does not allow modifying the value of 'enable.auto.

[jira] [Commented] (KAFKA-5096) Only log invalid user configs and overwrite with correct one

[ https://issues.apache.org/jira/browse/KAFKA-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999734#comment-15999734 ] ASF GitHub Bot commented on KAFKA-5096: --- GitHub user johnma14 opened a pull request:

Re: [VOTE] KIP-138: Change punctuate semantics

+1 (non binding) Thanks Eno > On May 6, 2017, at 11:01 PM, Bill Bejeck wrote: > > +1 > > Thanks, > Bill > > On Sat, May 6, 2017 at 5:58 PM, Matthias J. Sax > wrote: > >> +1 >> >> Thanks a lot for this KIP! >> >> -Matthias >> >> On 5/6/17 10:18 AM, Michal Borowiecki wrote: >>> Hi all, >>>

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

I like this KIP in general and I agree it’s needed. Perhaps Damian can comment on the session store issue? Thanks Eno > On May 6, 2017, at 10:32 PM, Michal Borowiecki > wrote: > > Hi Matthias, > > Agreed. I tried your proposal and indeed it would work. > > However, I think to maintain full b

Kafka Connect Parquet Support?

With the Kafka Connect S3 sink, I can choose Avro or JSON output format. Is there any chance that Parquet will be supported? For record at a time processing, Parquet isn't a good fit. But for reading/writing batches of records, which is what the Kafka Connect Sink is writing, Parquet is generally

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Hi Kyle, Thanks for the KIP again. A couple of comments: - minor: could you add an exact example (similar to what Jay’s example is, or like your Spark/Pig pointers had) to make this super concrete? - my main concern is that we’re exposing this optimization to the DSL. In an ideal world, an opt

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Do I understanding correctly, that with the proposed pattern one could not pass a lambda expression and access the context from within it? TBH, I was hoping that for simple things this would be possible: myStream.map( (key, value, ctx) -> new KeyValue<>(ctx.partition(), value) ) or (more to th

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Hi Michal, Thanks for your comments. We proposed the similar solution to yours in KIP (please look at rejected alternatives). However, after the discussion in mailing list I extended it to rich functions. Maybe we should keep them both: simple and rich versions. Cheers, Jeyhun On Sun, May 7, 201

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Hi Jeyhun, Thanks for your quick reply. Indeed, I understand the existing ValueMapper/Joiner etc. have to remain as-is for backwards compatibility. I was just expressing my surprise that their proposed richer equivalents weren't functional interfaces too. Thanks, Michał On 07/05/17 12:3

Re: [DISCUSS] KIP-133: List and Alter Configs Admin APIs

Hi James, On Sun, May 7, 2017 at 8:36 AM, James Cheng wrote: > Makes sense, and I don't have any concerns about it. Actually, I'm excited > for the capability. That means you can completely tell how a topic is > configured, without needing to have any knowledge of how the broker is > configured.

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

*- minor: could you add an exact example (similar to what Jay’s example is, or like your Spark/Pig pointers had) to make this super concrete?* I have added a more concrete example to the KIP. *- my main concern is that we’re exposing this optimization to the DSL. In an ideal world, an optimizer wo

[GitHub] kafka pull request #2991: KAFKA-4208: Add Record Headers

GitHub user michaelandrepearce opened a pull request: https://github.com/apache/kafka/pull/2991 KAFKA-4208: Add Record Headers Update upgrade.html Raising this now, as KIP-118 is pulled from release as such submitting this without java 8 changes. As per remaining r

[jira] [Commented] (KAFKA-4208) Add Record Headers

[ https://issues.apache.org/jira/browse/KAFKA-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1530#comment-1530 ] ASF GitHub Bot commented on KAFKA-4208: --- GitHub user michaelandrepearce opened a pul

[jira] [Created] (KAFKA-5185) Adding the RecordMetadata that is returned by the producer to the commitRecord method for SourceTask

George Smith created KAFKA-5185: --- Summary: Adding the RecordMetadata that is returned by the producer to the commitRecord method for SourceTask Key: KAFKA-5185 URL: https://issues.apache.org/jira/browse/KAFKA-5185

[GitHub] kafka pull request #2992: KIP-147 WIP for discussion

GitHub user mihbor opened a pull request: https://github.com/apache/kafka/pull/2992 KIP-147 WIP for discussion This is WIP to help discussion, do not merge. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mihbor/kafka KIP-147-wip

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

To visualise this better, I've created a WIP PR: https://github.com/apache/kafka/pull/2992 tentatively having named the new Stores class TypedStores. Thanks, Michał On 07/05/17 10:16, Eno Thereska wrote: I like this KIP in general and I agree it’s needed. Perhaps Damian can comment on the se

Re: [DISCUSS] KIP-133: List and Alter Configs Admin APIs

On Fri, May 5, 2017, at 19:24, Ismael Juma wrote: > Hi Colin, > > Thanks for the review. Comments inline. > > On Fri, May 5, 2017 at 9:45 PM, Colin McCabe wrote: > > > As Jun commented, it's probably more consistent to use > > Collection, unless this really needs to be an ordered > > list. Tha

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Michal, thanks a lot for this comment. I did not consider lambdas when proposing RichFunctions. I thinks it very important to preserve the ability to use lambdas! @Jeyhun: I did not put any thought into this, but can we have a design that allows for both? Also, with regard to lambdas, it might ma

[jira] [Commented] (KAFKA-5185) Adding the RecordMetadata that is returned by the producer to the commitRecord method for SourceTask

[ https://issues.apache.org/jira/browse/KAFKA-5185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1688#comment-1688 ] Konstantine Karantasis commented on KAFKA-5185: --- Thank you [~GeoSmith] for t

[jira] [Created] (KAFKA-5186) Avoid expensive initialization of producer state when upgrading

Jason Gustafson created KAFKA-5186: -- Summary: Avoid expensive initialization of producer state when upgrading Key: KAFKA-5186 URL: https://issues.apache.org/jira/browse/KAFKA-5186 Project: Kafka

[jira] [Created] (KAFKA-5187) producer.close() times out at 30 seconds

Somnath Choudhuri created KAFKA-5187: Summary: producer.close() times out at 30 seconds Key: KAFKA-5187 URL: https://issues.apache.org/jira/browse/KAFKA-5187 Project: Kafka Issue Type: Bu

[jira] [Commented] (KAFKA-5185) Adding the RecordMetadata that is returned by the producer to the commitRecord method for SourceTask

[ https://issues.apache.org/jira/browse/KAFKA-5185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000103#comment-16000103 ] George Smith commented on KAFKA-5185: - [~kkonstantine] Thanks, will do. > Adding the

[jira] [Created] (KAFKA-5188) Add Integration tests for transactional producer

Jason Gustafson created KAFKA-5188: -- Summary: Add Integration tests for transactional producer Key: KAFKA-5188 URL: https://issues.apache.org/jira/browse/KAFKA-5188 Project: Kafka Issue Type

Re: [DISCUSS] KIP 151 - Expose Connector type in REST API

Nice. Thanks! -Konstantine On Sat, May 6, 2017 at 10:43 PM, dan wrote: > thanks for the feedback, it all sounds good. i have made the changes to the > pr and the kip. > > dan > > On Fri, May 5, 2017 at 9:29 AM, Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > Thank you for the

[jira] [Updated] (KAFKA-4291) TopicCommand --describe shows topics marked for deletion as under-replicated and unavailable

[ https://issues.apache.org/jira/browse/KAFKA-4291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4291: --- Fix Version/s: 0.11.0.0 > TopicCommand --describe shows topics marked for deletion as under-replicated

[jira] [Updated] (KAFKA-4291) TopicCommand --describe shows topics marked for deletion as under-replicated and unavailable

[ https://issues.apache.org/jira/browse/KAFKA-4291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4291: --- Status: Patch Available (was: Open) > TopicCommand --describe shows topics marked for deletion as und

[jira] [Commented] (KAFKA-4795) Confusion around topic deletion

[ https://issues.apache.org/jira/browse/KAFKA-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000140#comment-16000140 ] Ismael Juma commented on KAFKA-4795: [~vahid], supposedly you may still see this if th

KIP-Template access

The KIP-Template doesn’t show up when I click to create a new KIP page My username is GeoSmith Thanks, George

[jira] [Commented] (KAFKA-3806) Adjust default values of log.retention.hours and offsets.retention.minutes

[ https://issues.apache.org/jira/browse/KAFKA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000185#comment-16000185 ] Di Shang commented on KAFKA-3806: - This is really problematic for mirrormaker since it man

Re: [DISCUSS] KIP-153 : Include only client traffic in BytesOutPerSec metric

Hi, Xavier, I think the per topic metric is more useful for measuring the traffic from the clients. For internal replication, perhaps the aggregate value is enough. Thanks, Jun On Fri, May 5, 2017 at 6:11 PM, Xavier Léauté wrote: > Any reason we are not keeping the per-topic breakdown for int

[VOTE] KIP-153 : Include only client traffic in BytesOutPerSec metric

Hi, Everyone, I would like to start the voting process for KIP-153 : Include only client traffic in BytesOutPerSec metric. https://cwiki.apache.org/confluence/display/KAFKA/KIP- 153+%3A+Include+only+client+traffic+in+BytesOutPerSec+metric The vote will run for a minimum of 72 hours. Thanks, Ju

[jira] [Created] (KAFKA-5189) trunk unstable - DescribeConsumerGroupTest.testDescribeGroupWithNewConsumerWithShortInitializationTimeout fails 90% of times

radai rosenblatt created KAFKA-5189: --- Summary: trunk unstable - DescribeConsumerGroupTest.testDescribeGroupWithNewConsumerWithShortInitializationTimeout fails 90% of times Key: KAFKA-5189 URL: https://issues.ap

[jira] [Commented] (KAFKA-5189) trunk unstable - DescribeConsumerGroupTest.testDescribeGroupWithNewConsumerWithShortInitializationTimeout fails 90% of times

[ https://issues.apache.org/jira/browse/KAFKA-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000197#comment-16000197 ] Ismael Juma commented on KAFKA-5189: Possibly related to KAFKA-4948. > trunk unstable

[VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

Hi, Everyone, Since this is a relatively simple change, I would like to start the voting process for KIP-153 : Include only client traffic in BytesOutPerSec metric. https://cwiki.apache.org/confluence/display/KAFKA/KIP-153+% 3A+Include+only+client+traffic+in+BytesOutPerSec+metric The vote will r

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

Thanks for the KIP, Jun. +1 from me. Ismael On Mon, May 8, 2017 at 3:40 AM, Jun Rao wrote: > Hi, Everyone, > > Since this is a relatively simple change, I would like to start the voting > process for KIP-153 : Include only client traffic in BytesOutPerSec metric. > > https://cwiki.apache.org/co

Re: [DISCUSS] KIP-133: List and Alter Configs Admin APIs

Thanks for the feedback Colin. Comments inline. On Sun, May 7, 2017 at 9:29 PM, Colin McCabe wrote: > > Hmm. What's the behavior if I try to list the configuration for a topic > that doesn't exist? It seems like in this case, the whole request has > to return an error, and nothing gets listed.

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

+1 (non-binding) James > On May 7, 2017, at 10:59 PM, Ismael Juma wrote: > > Thanks for the KIP, Jun. +1 from me. > > Ismael > > On Mon, May 8, 2017 at 3:40 AM, Jun Rao wrote: > >> Hi, Everyone, >> >> Since this is a relatively simple change, I would like to start the voting >> process for

[jira] [Updated] (KAFKA-3878) Exponential backoff for broker reconnect attempts

[ https://issues.apache.org/jira/browse/KAFKA-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3878: --- Fix Version/s: 0.11.0.0 > Exponential backoff for broker reconnect attempts >

[jira] [Updated] (KAFKA-3878) Exponential backoff for broker reconnect attempts

[ https://issues.apache.org/jira/browse/KAFKA-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3878: --- Assignee: Dana Powers (was: Ismael Juma) Status: Patch Available (was: Open) > Exponential bac

[jira] [Commented] (KAFKA-4982) Add listener tag to socket-server-metrics.connection-... metrics

[ https://issues.apache.org/jira/browse/KAFKA-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000223#comment-16000223 ] Ismael Juma commented on KAFKA-4982: [~ecomar], are you planning to submit a PR for th

[VOTE] KIP-133: List and Alter Configs Admin APIs

Hi everyone, I believe I addressed the comments in the discussion thread and given the impending KIP freeze, I would like to start the voting process for KIP-133: List and Alter Configs Admin APIs: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 133%3A+List+and+Alter+Configs+Admin+APIs As

[VOTE] KIP-133: List and Alter Configs Admin APIs (second attempt)

[Seems like the original message ended up in the discuss thread in GMail, so trying again] Hi everyone, I believe I addressed the comments in the discussion thread and given the impending KIP freeze, I would like to start the voting process for KIP-133: List and Alter Configs Admin APIs: https:/

[jira] [Created] (KAFKA-5190) builds with low parallelism exhaust system open files and crash

radai rosenblatt created KAFKA-5190: --- Summary: builds with low parallelism exhaust system open files and crash Key: KAFKA-5190 URL: https://issues.apache.org/jira/browse/KAFKA-5190 Project: Kafka

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

+1 > On May 7, 2017, at 7:40 PM, Jun Rao wrote: > > Hi, Everyone, > > Since this is a relatively simple change, I would like to start the voting > process for KIP-153 : Include only client traffic in BytesOutPerSec metric. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-153+% > 3A+In

Re: [VOTE] KIP-133: List and Alter Configs Admin APIs (second attempt)

+1 > On May 7, 2017, at 9:01 PM, Ismael Juma wrote: > > [Seems like the original message ended up in the discuss thread in GMail, > so trying again] > > Hi everyone, > > I believe I addressed the comments in the discussion thread and given the > impending KIP freeze, I would like to start the

Re: [DISCUSS] KIP-133: List and Alter Configs Admin APIs

The KIP talks about allowing the user to provide a AlterConfigsPolicy. I assume that even if the user does not provide a custom policy, that there are some basic validation that the broker will do by default? The reason I'm asking is I'm thinking ahead to https://issues.apache.org/jira/browse/K

[jira] [Commented] (KAFKA-3480) Autogenerate metrics documentation

[ https://issues.apache.org/jira/browse/KAFKA-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000256#comment-16000256 ] ASF GitHub Bot commented on KAFKA-3480: --- Github user wushujames closed the pull requ

[GitHub] kafka pull request #1202: KAFKA-3480: Autogenerate metrics documentation

Github user wushujames closed the pull request at: https://github.com/apache/kafka/pull/1202 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

James Cheng created KAFKA-5191: -- Summary: Autogenerate Consumer Fetcher metrics Key: KAFKA-5191 URL: https://issues.apache.org/jira/browse/KAFKA-5191 Project: Kafka Issue Type: Sub-task

[jira] [Assigned] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

[ https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Cheng reassigned KAFKA-5191: -- Assignee: James Cheng > Autogenerate Consumer Fetcher metrics > ---

[jira] [Work started] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

[ https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-5191 started by James Cheng. -- > Autogenerate Consumer Fetcher metrics > - > >

[jira] [Commented] (KAFKA-5099) Replica Deletion Regression from KIP-101

[ https://issues.apache.org/jira/browse/KAFKA-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000261#comment-16000261 ] Onur Karaman commented on KAFKA-5099: - By the way, I reran the file descriptor experim

[jira] [Resolved] (KAFKA-5172) CachingSessionStore doesn't fetchPrevious correctly.

[ https://issues.apache.org/jira/browse/KAFKA-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5172. -- Resolution: Fixed Fix Version/s: 0.11.0.0 Issue resolved by pull request 2972 [https://gi

[GitHub] kafka pull request #2972: KAFKA-5172: Fix fetchPrevious to find the correct ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2972 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-5172) CachingSessionStore doesn't fetchPrevious correctly.

[ https://issues.apache.org/jira/browse/KAFKA-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000278#comment-16000278 ] ASF GitHub Bot commented on KAFKA-5172: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #2993: KAFKA-5191: Autogenerate Consumer Fetcher metrics

GitHub user wushujames opened a pull request: https://github.com/apache/kafka/pull/2993 KAFKA-5191: Autogenerate Consumer Fetcher metrics Autogenerate docs for the Consumer Fetcher's metrics. This is a smaller subset of the original PR https://github.com/apache/kafka/pull/1202.

[jira] [Commented] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

[ https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000279#comment-16000279 ] ASF GitHub Bot commented on KAFKA-5191: --- GitHub user wushujames opened a pull reques

[jira] [Updated] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

[ https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Cheng updated KAFKA-5191: --- Reviewer: Ismael Juma Status: Patch Available (was: In Progress) > Autogenerate Consumer Fetche

[jira] [Updated] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

[ https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Cheng updated KAFKA-5191: --- Attachment: generated_fetcher_docs.png > Autogenerate Consumer Fetcher metrics > -

Build failed in Jenkins: kafka-trunk-jdk8 #1490

See Changes: [wangguoz] KAFKA-5172: Fix fetchPrevious to find the correct session -- [...truncated 857.71 KB...] kafka.integration.SaslPlaintextTopicMetadataTest > testAl