Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

2016-02-13 Thread Ismael Juma
Hi Bill, Thanks for the proposal, it makes sense. A minor comment: `log.retention.mintimestamp` should maybe be called `log.retention.min.timestamp` to be consistent with other properties. Also, it would be good to update the KIP to mention the LogAppendTime point. Ismael On Sat, Feb 13, 2016 at

Re: New Consumer Poll bug?

2016-02-13 Thread Ismael Juma
Thanks for checking Damian. The underlying problem of KAFKA-2978 is that the fetched position was getting out of sync with the consumed position (and fetches were only sent if the fetched position matched the consumed position). The reporter was seeing this problem after rebalancing, but perhaps th

[jira] [Comment Edited] (KAFKA-1464) Add a throttling option to the Kafka replication tool

2016-02-13 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146250#comment-15146250 ] Neha Narkhede edited comment on KAFKA-1464 at 2/13/16 11:36 PM:

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

2016-02-13 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146250#comment-15146250 ] Neha Narkhede commented on KAFKA-1464: -- The most useful resource to throttle for is n

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

2016-02-13 Thread Bill Warshaw
Hello, That is a good catch, thanks for pointing it out. If this KIP is accepted, we'd need to document this and make the log cleaner not run timestamp-based deletion unless message.timestamp.type=LogAppendTime. On Sat, Feb 13, 2016 at 5:38 AM, Andrew Schofield < andrew_schofield_j...@outlook.co

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

2016-02-13 Thread Bill Warshaw
We initially looked at exposing an API through AdminUtils that would manually delete everything in a given partition before a specified offset. We ran into several difficulties in figuring out how to efficiently communicate this "truncate partition" command to all brokers in Zookeeper, without addi

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

2016-02-13 Thread Enrico Olivelli
Hi, I'm currently using kafka 0.9 as a commit log. I would find more useful to set an offset for every partition (actually I am using only one partition per topic) instead of a global timestamp, has this option been already considered? Thanks -- Enrico Il giorno Sab 13 Feb 2016 11:38 Andrew Sc

Re: New Consumer Poll bug?

2016-02-13 Thread Damian Guy
I tried the 0.9.0.1 client and was able to successfully consume the partition without any problems. Thanks, Damian On 13 February 2016 at 14:49, Damian Guy wrote: > Forgot to reply-all… > > > On 13 Feb 2016, at 14:49, Damian Guy wrote: > > Hi Ismael, > > In this case it wasn’t a rebalance as I

[jira] [Closed] (KAFKA-2673) Log JmxTool output to logger

2016-02-13 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eno Thereska closed KAFKA-2673. --- > Log JmxTool output to logger > > > Key: KAFKA-2673 >

[jira] [Resolved] (KAFKA-2673) Log JmxTool output to logger

2016-02-13 Thread chen zhu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chen zhu resolved KAFKA-2673. - Resolution: Not A Problem Not needed. > Log JmxTool output to logger > > >

[jira] [Comment Edited] (KAFKA-2673) Log JmxTool output to logger

2016-02-13 Thread chen zhu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146101#comment-15146101 ] chen zhu edited comment on KAFKA-2673 at 2/13/16 6:03 PM: -- [~enot

[jira] [Commented] (KAFKA-2673) Log JmxTool output to logger

2016-02-13 Thread chen zhu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146101#comment-15146101 ] chen zhu commented on KAFKA-2673: - No problem. Issue closed. > Log JmxTool output to log

Re: New Consumer Poll bug?

2016-02-13 Thread Damian Guy
Forgot to reply-all… > On 13 Feb 2016, at 14:49, Damian Guy wrote: > > Hi Ismael, > > In this case it wasn’t a rebalance as I was using consumer.assign(..) I’ll > give it a try with the 0.9.0.1 client to see if i can reproduce. > I’ve tried the 0.9.0.1 client with consumer groups and it seem

[jira] [Commented] (KAFKA-1449) Extend wire protocol to allow CRC32C

2016-02-13 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145978#comment-15145978 ] Ismael Juma commented on KAFKA-1449: A CRC32C class with appropriate intrinsics was ad

Re: New Consumer Poll bug?

2016-02-13 Thread Ismael Juma
Hi Damian, KAFKA-2978, which would cause consumption to stop would happen after a consumer group rebalance. Was this the case for you? It would be great if you could upgrade the client to 0.9.0.1 RC1 in order to check if the problem still happens. There were other bugs fixed in the 0.9.0 branch a

[jira] [Updated] (KAFKA-2938) Socket server may throw IllegalStateException

2016-02-13 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2938: --- Fix Version/s: (was: 0.9.0.1) > Socket server may throw IllegalStateException > --

RE: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

2016-02-13 Thread Andrew Schofield
This KIP is related to KIP-32, but I strikes me that it only makes sense with one of the two proposed message timestamp types. If I understand correctly, message timestamps are only certain to be monotonically increasing in the log if message.timestamp.type=LogAppendTime. Does timestamp-based

New Consumer Poll bug?

2016-02-13 Thread Damian Guy
I've been having some issues with the New Consumer. I'm aware there is a bug that has been fixed for 0.9.0.1, but is this the same thing? I'm using manual partition assignment due to latency issues making it near impossible to work with the group management features. So, my consumer was going alon