[GitHub] kafka pull request #2559: KAFKA-4775. Fix findbugs warnings in kafka-tools

2017-02-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2559 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4775) Fix findbugs warnings in kafka-tools

2017-02-17 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871548#comment-15871548 ] ASF GitHub Bot commented on KAFKA-4775: --- Github user asfgit closed the pull request

[jira] [Resolved] (KAFKA-4775) Fix findbugs warnings in kafka-tools

2017-02-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4775. Resolution: Fixed Fix Version/s: 0.10.3.0 Issue resolved by pull request 2559 [https://github

[jira] [Commented] (KAFKA-4676) Kafka consumers gets stuck for some partitions

2017-02-17 Thread Narendra Kumar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871619#comment-15871619 ] Narendra Kumar commented on KAFKA-4676: --- We are also facing the same issue. [~onur

[GitHub] kafka pull request #2544: KFKA-4340: follow up patch for KAFAK-4340

2017-02-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2544 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4753) KafkaConsumer susceptible to FetchResponse starvation

2017-02-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871679#comment-15871679 ] Ismael Juma commented on KAFKA-4753: [~onurkaraman], are you intending to submit a PR

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
On the point re: headers in the message protocol being a byte array and not a count of elements followed by the elements. Again this was discussed/argued previously. It was agreed on for a few reasons some of which you have obviously picked up on: Broker is able to pass it through opaquely The

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
On the point of varInts Would you be proposing in KIP-98 to convert the other message int’s (key length, value length) also to varint to keep it uniform. Also I assume there will be a static or helper method made to write/read these in the client and server. Cheers Mike On 17/02/2017, 11:22,

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

2017-02-17 Thread Michael Pearce
+0 I think need some unified agreement on the VarInts. Would this also change in all other area’s of the protocol, e.g. value and key length in message protocol, to keep this uniform across all protocols going forwards? On 17/02/2017, 00:23, "Apurva Mehta" wrote: Hi Jun, Thanks fo

[jira] [Commented] (KAFKA-4767) KafkaProducer is not joining its IO thread properly

2017-02-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871783#comment-15871783 ] Ismael Juma commented on KAFKA-4767: [~bgedik], `this.sender.forceClose();` is "some c

Kafka Connect / Access to OffsetStorageReader from SourceConnector

2017-02-17 Thread Florian Hussonnois
Hi Kafka Team, I'm developping a connector which need to monitor the progress of its tasks in order to be able to request a tasks reconfiguration in some situations. Our connector is pretty simple. It's used to stream a thousands of files into Kafka. The connector scans directories then schedules

Build failed in Jenkins: kafka-trunk-jdk8 #1281

2017-02-17 Thread Apache Jenkins Server
See Changes: [ismael] KAFKA-4775; Fix findbugs warnings in kafka-tools -- [...truncated 33781 lines...] kafka.security.auth.SimpleAclAuthorizerTest > testHighConcurrencyModificationOfResourceAc

Build failed in Jenkins: kafka-trunk-jdk7 #1947

2017-02-17 Thread Apache Jenkins Server
See Changes: [ismael] KAFKA-4340; Follow-up fixing system test failures and handling non -- [...truncated 15968 lines...] org.apache.kafka.clients.producer.internals.RecordAccumulatorTest > test

Build failed in Jenkins: kafka-trunk-jdk8 #1282

2017-02-17 Thread Apache Jenkins Server
See Changes: [ismael] KAFKA-4340; Follow-up fixing system test failures and handling non -- [...truncated 4136 lines...] kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED kaf

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

2017-02-17 Thread Jorge Esteban Quilcate Otoya
Hi, according to the feedback, I've updated the KIP: - We have added and ordered the scenarios, scopes and executions of the Reset Offset tool. - Consider it as an extension to the current `ConsumerGroupCommand` tool - Execution will be possible without generating JSON files. https://cwiki.apach

Re: documentation feedback

2017-02-17 Thread Colin McCabe
Hi Brian, Have you created a pull request for this? best, Colin On Thu, Feb 9, 2017, at 15:21, Brian Cornally wrote: > @apachekafka suggested addition to > http://kafka.apache.org/documentation/#security - section 5 - Examples > using console-producer and console-consumer: > > echo "ssl" | /u

[jira] [Commented] (KAFKA-4754) Correctly parse '=' characters in command line overrides

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872116#comment-15872116 ] Colin P. McCabe commented on KAFKA-4754: bq. Its worth noting, it was also possibl

Would kafka support delay queue in the future

2017-02-17 Thread Steven,Zhang(vip.com)
Hello, Some MQ products could support sending the message with ttl property; after ttl timeout, MQ could publish the message to the specified queue. I want to know whether kafka would support the feature in the future. Best Regards, Steven.zhang vipshop 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨

[jira] [Comment Edited] (KAFKA-4754) Correctly parse '=' characters in command line overrides

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872116#comment-15872116 ] Colin P. McCabe edited comment on KAFKA-4754 at 2/17/17 5:02 PM: ---

[DISCUSS] KIP-124: Request rate quotas

2017-02-17 Thread Rajini Sivaram
Hi all, I have just created KIP-124 to introduce request rate quotas to Kafka: https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas The proposal is for a simple percentage request handling time quota that can be allocated to **, ** or **. There are a few other suggesti

[jira] [Commented] (KAFKA-4754) Correctly parse '=' characters in command line overrides

2017-02-17 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872127#comment-15872127 ] Grant Henke commented on KAFKA-4754: {quote} Hmm. It is not a good practice to pass pa

[jira] [Commented] (KAFKA-4754) Correctly parse '=' characters in command line overrides

2017-02-17 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872139#comment-15872139 ] Grant Henke commented on KAFKA-4754: {quote} This could expose the password to anyone

[jira] [Updated] (KAFKA-4757) NetworkClient should log request details at trace level when a request is cancelled because of disconnection

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4757: --- Summary: NetworkClient should log request details at trace level when a request is cancelled b

[jira] [Assigned] (KAFKA-4757) NetworkClient should log request details at trace level when a request is cancelled because of disconnection

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe reassigned KAFKA-4757: -- Assignee: Colin P. McCabe > NetworkClient should log request details at trace level whe

[jira] [Commented] (KAFKA-4757) NetworkClient should log request details at trace level when a request is cancelled because of disconnection

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872176#comment-15872176 ] Colin P. McCabe commented on KAFKA-4757: bq. 1. The [request cancelled] log at one

Re: [DISCUSS] KIP-124: Request rate quotas

2017-02-17 Thread Dong Lin
Hey Rajini, Thanks for the KIP. I have some questions: - I am wondering why throttling based on request rate is listed as a rejected alternative. Can you provide more specific reason why it is difficult for administrators to decide request rates to allocate? It seems to me that determination of r

[jira] [Commented] (KAFKA-4757) NetworkClient should log request details at trace level when a request is cancelled because of disconnection

2017-02-17 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872194#comment-15872194 ] Onur Karaman commented on KAFKA-4757: - Thanks [~cmccabe]. Regarding the second issue,

Re: [DISCUSS] KIP-124: Request rate quotas

2017-02-17 Thread Dong Lin
To correct the typo above: It seems to me that determination of request rate is not any more difficult than determination of *byte* rate as both metrics are commonly used to measure performance and provide guarantee to user. On Fri, Feb 17, 2017 at 9:40 AM, Dong Lin wrote: > Hey Rajini, > > Than

[jira] [Commented] (KAFKA-4676) Kafka consumers gets stuck for some partitions

2017-02-17 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872199#comment-15872199 ] Onur Karaman commented on KAFKA-4676: - [~Narendra Kumar] if you're repeatedly seeing t

[jira] [Commented] (KAFKA-4676) Kafka consumers gets stuck for some partitions

2017-02-17 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872213#comment-15872213 ] Onur Karaman commented on KAFKA-4676: - This ticket never really found a root cause so

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Jason Gustafson
> > Would you be proposing in KIP-98 to convert the other message int’s (key > length, value length) also to varint to keep it uniform. > Also I assume there will be a static or helper method made to write/read > these in the client and server. Yes, that is what we are proposing, so using varints

[jira] [Commented] (KAFKA-4715) Consumer/Producer config does not work with related enums

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872219#comment-15872219 ] Colin P. McCabe commented on KAFKA-4715: Hmm. It seems better to override the {{n

Re: get function

2017-02-17 Thread Colin McCabe
Hi Samy, If I understand your question correctly, you're looking for a way to retrieve a value based on a corresponding key. That would require you to build an index. Kafka can look up records based on the record time, but by default will not build any kind of index based on the record key. In

Re: [DISCUSS] KIP-124: Request rate quotas

2017-02-17 Thread Onur Karaman
Overall a big fan of the KIP. I'd have to agree with Dong. I'm not sure about the decision of using the percentage over the window as opposed to request rate. It's pretty hard to reason about. I just spoke to one of our SRE's and he agrees. Also I may have missed it, but I couldn't find informati

[jira] [Commented] (KAFKA-4754) Correctly parse '=' characters in command line overrides

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872228#comment-15872228 ] Colin P. McCabe commented on KAFKA-4754: Sorry, I guess I was unclear... I definit

[jira] [Comment Edited] (KAFKA-4754) Correctly parse '=' characters in command line overrides

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872228#comment-15872228 ] Colin P. McCabe edited comment on KAFKA-4754 at 2/17/17 6:15 PM: ---

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
Re headers available on the record va interceptors only One of the key use cases requires access on consume at per event/message level at the point that message is being processed, as such the batch interceptors and batch consume api isn't suitable. It needs to be at the record level. This any

Request Wiki edit permission for cshannon

2017-02-17 Thread Christopher Shannon
Hi, I would like to contribute a KIP. Can someone give me access to the wiki to create pages? Thanks, Chris

[jira] [Created] (KAFKA-4777) Kafka client Heartbeat thread use all the cpu.

2017-02-17 Thread Allen Xiang (JIRA)
Allen Xiang created KAFKA-4777: -- Summary: Kafka client Heartbeat thread use all the cpu. Key: KAFKA-4777 URL: https://issues.apache.org/jira/browse/KAFKA-4777 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Jason Gustafson
> > Re headers as byte array and future use by broker. This doesn't take away > from that at all. Nor makes it difficult at all in my opinion. Yeah, I didn't say it was difficult, only awkward. You wouldn't write the schema that way if you were planning to use it on the brokers from the beginning

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Jason Gustafson
Sorry, my mistake. The consumer interceptor is per batch, though I'm not sure that's an actual limitation since you still have access to the individual records. -Jason On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson wrote: > Re headers as byte array and future use by broker. This doesn't take

[jira] [Commented] (KAFKA-4757) NetworkClient should log request details at trace level when a request is cancelled because of disconnection

2017-02-17 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872284#comment-15872284 ] Onur Karaman commented on KAFKA-4757: - [~cmccabe] I double checked and the 2nd logging

Re: Request to add to contributor list

2017-02-17 Thread Guozhang Wang
Done, cheers. Guozhang On Thu, Feb 16, 2017 at 11:01 PM, Shun Takebayashi wrote: > Hi dev team, > > I want to be added as a contributor in Kafka JIRA. > My JIRA ID is takebayashi > > Thanks, > > Shun > -- -- Guozhang

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Magnus Edenhill
Big +1 on VarInts. CPUs are fast, memory is slow. I agree with Jason that we'll want to continue verifying messages, including their headers, so while I appreciate the idea of the opaque header blob it won't be useful in practice. /Magnus 2017-02-17 10:41 GMT-08:00 Jason Gustafson : > Sorry, my

[GitHub] kafka pull request #2563: Kafka 4757: NetworkClient should log request detai...

2017-02-17 Thread cmccabe
GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/2563 Kafka 4757: NetworkClient should log request details at trace level when a request is cancelled because of disconnection You can merge this pull request into a Git repository by running: $ gi

[jira] [Commented] (KAFKA-4757) NetworkClient should log request details at trace level when a request is cancelled because of disconnection

2017-02-17 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872390#comment-15872390 ] Onur Karaman commented on KAFKA-4757: - Ah. Might've found it: Here's what ApiVersionsR

Re: Request Wiki edit permission for cshannon

2017-02-17 Thread Guozhang Wang
Done. Guozhang On Fri, Feb 17, 2017 at 10:20 AM, Christopher Shannon < christopher.l.shan...@gmail.com> wrote: > Hi, > > I would like to contribute a KIP. Can someone give me access to the wiki > to create pages? > > Thanks, > Chris > -- -- Guozhang

[jira] [Updated] (KAFKA-4757) Improve NetworkClient trace logging of request details

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4757: --- Summary: Improve NetworkClient trace logging of request details (was: NetworkClient should lo

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
Yes exactly we have access to the records thus why the header should be accessible via it and not hidden for only interceptors to access. Sent using OWA for iPhone From: Magnus Edenhill Sent: Friday, February 17, 2017 7:34:49 PM To: dev@kafka.apache.org S

Re: Request Wiki edit permission for cshannon

2017-02-17 Thread Christopher Shannon
Awesome, thanks. On Fri, Feb 17, 2017 at 2:30 PM, Guozhang Wang wrote: > Done. > > Guozhang > > On Fri, Feb 17, 2017 at 10:20 AM, Christopher Shannon < > christopher.l.shan...@gmail.com> wrote: > > > Hi, > > > > I would like to contribute a KIP. Can someone give me access to the wiki > > to cre

[jira] [Commented] (KAFKA-4757) Improve NetworkClient trace logging of request details

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872406#comment-15872406 ] Colin P. McCabe commented on KAFKA-4757: Thanks. That's a good find. I just post

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
I am happy to move the definition of the header into the message body, but would cause us not to lazy initialise/parse the headers, as obviously, we would have to traverse these reading the message. This was actually one of Jay’s requests: “2. I think we should think about creating the Map

[GitHub] kafka pull request #2564: KAFKA-4777 fix client heartbeat non-stop retry iss...

2017-02-17 Thread allenxiang
GitHub user allenxiang opened a pull request: https://github.com/apache/kafka/pull/2564 KAFKA-4777 fix client heartbeat non-stop retry issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/allenxiang/kafka client-heartbeat-fix

[jira] [Commented] (KAFKA-4777) Kafka client Heartbeat thread use all the cpu.

2017-02-17 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872435#comment-15872435 ] ASF GitHub Bot commented on KAFKA-4777: --- GitHub user allenxiang opened a pull reques

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Jason Gustafson
The point about creation of maps seems orthogonal. We can still represent the headers as a slice of bytes until the time it is accessed. > Yes exactly we have access to the records thus why the header should be > accessible via it and not hidden for only interceptors to access. As explained abo

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

2017-02-17 Thread Mayuresh Gharat
@Manikumar, Can you give an example how you are planning to use PrincipalBuilder? @Jun Yes, that is right. To give a brief overview, we just extract the cert and hand it over to a third party library for creating a Principal. So we cannot create a Principal from just a string. The main motive beh

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
What’s the issue with exposing a method getHeaders on the producer/consumer record? It doesn’t break anything. We don’t need any special version. Current batch consumer model and consumer interceptors don’t work where headers need to be acted on at per message level at time of processing, very c

[jira] [Commented] (KAFKA-4715) Consumer/Producer config does not work with related enums

2017-02-17 Thread Mathias Kub (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872493#comment-15872493 ] Mathias Kub commented on KAFKA-4715: Overriding any method coming from Enum seems bad

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
Re: “The point about creation of maps seems orthogonal. We can still represent the headers as a slice of bytes until the time it is accessed.” That’s exactly what we’re doing the headers are a slice of bytes, which then gets parsed later if needed, or can be parsed right away, the header

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Jason Gustafson
> > That’s exactly what we’re doing the headers are a slice of bytes, which > then gets parsed later if needed, or can be parsed right away, the headers > is part of the protocol, so can still be validated if wanted. > If you had a header count then you would have to go through each header > key an

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Michael Pearce
For APM single event tracing, need access to the header at the point of processing on the processing thread. As such interceptors will not work/be suitable for these, due to the fact they act on the ConsumerRecords as a batch, before the handling thread can split out and process per message whi

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

2017-02-17 Thread Jun Rao
Hi, Mani, Good point on using PrincipalBuilder for SASL. It seems that PrincipalBuilder already has access to Authenticator. So, we could just enable that in SaslChannelBuilder. We probably could do that in a separate KIP? Hi, Mayuresh, If you don't think there is a concrete use case for using P

[jira] [Updated] (KAFKA-4759) Add support for subnet masks in SimpleACLAuthorizer

2017-02-17 Thread Shun Takebayashi (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shun Takebayashi updated KAFKA-4759: Assignee: Shun Takebayashi Status: Patch Available (was: Open) > Add support for subn

[jira] [Commented] (KAFKA-4757) Improve NetworkClient trace logging of request details

2017-02-17 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872616#comment-15872616 ] Onur Karaman commented on KAFKA-4757: - Left some comments on the PR. Speaking of whic

[GitHub] kafka pull request #2565: MINOR: Use scale of 3 in streams benchmark by defa...

2017-02-17 Thread guozhangwang
GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/2565 MINOR: Use scale of 3 in streams benchmark by default [DO NOT MERGE] You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka KM

[jira] [Commented] (KAFKA-4757) Improve NetworkClient trace logging of request details

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872643#comment-15872643 ] Colin P. McCabe commented on KAFKA-4757: https://github.com/apache/kafka/pull/2563

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

2017-02-17 Thread Stevo Slavic (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872645#comment-15872645 ] Stevo Slavic commented on KAFKA-4385: - [~Jun Yao] [~ewencp] it would open up more opti

[jira] [Commented] (KAFKA-4757) Improve NetworkClient trace logging of request details

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872650#comment-15872650 ] Colin P. McCabe commented on KAFKA-4757: Thanks, [~onurkaraman]. I usually do nam

Re: Should InvalidTopicException be called IllegalTopicException

2017-02-17 Thread Colin McCabe
Hmm. I'm not sure that "illegal" is that different from "invalid" semantically. If an argument is illegal for a given function, you could also say that argument is invalid for that function. Also, changing exception names could be frustrating to users from a compatibility perspective. Perhaps s

[GitHub] kafka pull request #2513: MINOR: Remove Struct from Request/Response classes

2017-02-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2513 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4715) Consumer/Producer config does not work with related enums

2017-02-17 Thread Colin P. McCabe (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872664#comment-15872664 ] Colin P. McCabe commented on KAFKA-4715: bq. Beside that the name() method is fina

[jira] [Commented] (KAFKA-4317) RocksDB checkpoint files lost on kill -9

2017-02-17 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872676#comment-15872676 ] ASF GitHub Bot commented on KAFKA-4317: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #2471: KAFKA-4317: Regularly checkpoint StateStore change...

2017-02-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2471 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Updated] (KAFKA-4317) RocksDB checkpoint files lost on kill -9

2017-02-17 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4317: - Resolution: Fixed Fix Version/s: 0.10.3.0 Status: Resolved (was: Patch Available

[jira] [Created] (KAFKA-4778) OOM on kafka-streams instances with high numbers of unreaped Record classes

2017-02-17 Thread Dave Thomas (JIRA)
Dave Thomas created KAFKA-4778: -- Summary: OOM on kafka-streams instances with high numbers of unreaped Record classes Key: KAFKA-4778 URL: https://issues.apache.org/jira/browse/KAFKA-4778 Project: Kafka

[jira] [Updated] (KAFKA-4778) OOM on kafka-streams instances with high numbers of unreaped Record classes

2017-02-17 Thread Dave Thomas (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thomas updated KAFKA-4778: --- Description: We have a stream processing app with ~8 source/sink stages operating roughly at the rate

[jira] [Updated] (KAFKA-4778) OOM on kafka-streams instances with high numbers of unreaped Record classes

2017-02-17 Thread Dave Thomas (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thomas updated KAFKA-4778: --- Attachment: oom-killer.txt OOM-killer kernel message is attached. > OOM on kafka-streams instances wi

[GitHub] kafka pull request #2566: HOTFIX: ClassCastException in requrest logging

2017-02-17 Thread hachikuji
GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/2566 HOTFIX: ClassCastException in requrest logging You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka hotfix-request-logging Alterna

[GitHub] kafka pull request #2566: HOTFIX: ClassCastException in request logging

2017-02-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2566 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

2017-02-17 Thread Mayuresh Gharat
Hi Jun, Sure sounds good to me. Thanks, Mayuresh On Fri, Feb 17, 2017 at 1:54 PM, Jun Rao wrote: > Hi, Mani, > > Good point on using PrincipalBuilder for SASL. It seems that > PrincipalBuilder already has access to Authenticator. So, we could just > enable that in SaslChannelBuilder. We proba

Jenkins build is back to normal : kafka-trunk-jdk7 #1949

2017-02-17 Thread Apache Jenkins Server
See

[GitHub] kafka pull request #2567: MINOR: Increase consumer init timeout in throttlin...

2017-02-17 Thread apurvam
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2567 MINOR: Increase consumer init timeout in throttling test The throttling system test sometimes fail because it takes longer than the current 10 second time out for partitions to get assigned to the c

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

2017-02-17 Thread Mayuresh Gharat
Hi Jun, I have updated the KIP. Would you mind taking another look? Thanks, Mayuresh On Fri, Feb 17, 2017 at 4:42 PM, Mayuresh Gharat wrote: > Hi Jun, > > Sure sounds good to me. > > Thanks, > > Mayuresh > > On Fri, Feb 17, 2017 at 1:54 PM, Jun Rao wrote: > >> Hi, Mani, >> >> Good point on u

[jira] [Created] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

2017-02-17 Thread Apurva Mehta (JIRA)
Apurva Mehta created KAFKA-4779: --- Summary: Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py Key: KAFKA-4779 URL: https://issues.apache.org/jira/browse/KAFKA-4779 Project: Kaf

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

2017-02-17 Thread Apurva Mehta (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872862#comment-15872862 ] Apurva Mehta commented on KAFKA-4779: - [~rsivaram] do you have an idea about what coul

Build failed in Jenkins: kafka-trunk-jdk8 #1284

2017-02-17 Thread Apache Jenkins Server
See Changes: [jason] HOTFIX: ClassCastException in request logging -- [...truncated 8436 lines...] kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange STARTED kafka.server.

[jira] [Commented] (KAFKA-2319) After controlled shutdown: IllegalStateException: Kafka scheduler has not been started

2017-02-17 Thread JIRA
[ https://issues.apache.org/jira/browse/KAFKA-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872967#comment-15872967 ] Michał Woś commented on KAFKA-2319: --- I've got the same problem after SIGTERM. Moreover:

[jira] [Comment Edited] (KAFKA-2319) After controlled shutdown: IllegalStateException: Kafka scheduler has not been started

2017-02-17 Thread JIRA
[ https://issues.apache.org/jira/browse/KAFKA-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872967#comment-15872967 ] Michał Woś edited comment on KAFKA-2319 at 2/18/17 3:40 AM: I'

[jira] [Comment Edited] (KAFKA-2319) After controlled shutdown: IllegalStateException: Kafka scheduler has not been started

2017-02-17 Thread JIRA
[ https://issues.apache.org/jira/browse/KAFKA-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872967#comment-15872967 ] Michał Woś edited comment on KAFKA-2319 at 2/18/17 3:41 AM: I'

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

2017-02-17 Thread Jason Gustafson
Hey Florian, Can you explain a bit more how having access to the offset storage from the connector helps in your use case? I guess you are planning to use offsets to be able to tell when a task has finished a file? Thanks, Jason On Fri, Feb 17, 2017 at 4:45 AM, Florian Hussonnois wrote: > Hi K